feyninc/tokie

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Fast Tokens

exact tokenizers ids, as fast as the hardware allows

A high-performance tokenizer (BPE, WordPiece, SentencePiece) built with Rust with Python bindings, focused on speed, safety, and resource optimization.

Language model tokenization at GB/s

Fast tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.

Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.

🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer

Datamodels for hugging face tokenizers