ml-rust/splintr

🍡 30x faster tokenization for every HuggingFace model

Fast tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.

Fast Tokens

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

exact tokenizers ids, as fast as the hardware allows

🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer

Language model tokenization at GB/s

Code for the paper "BPE stays on SCRIPT", "Which Pieces Does Unigram Tokenization Really Need?" and MinGram

Fast and memory-efficient library for WordPiece tokenization as it is used by BERT.