georg-jung/FastBertTokenizer

🍡 30x faster tokenization for every HuggingFace model

Fast Tokens

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Fast tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.

exact tokenizers ids, as fast as the hardware allows

A high-performance tokenizer (BPE, WordPiece, SentencePiece) built with Rust with Python bindings, focused on speed, safety, and resource optimization.

Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.

Language model tokenization at GB/s

🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer