Thai natural language processing in Python
Syllable-aware BPE tokenizer for the Amharic language (አማርኛ) – fast, accurate, trainable.
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
🎤 vibrato: Viterbi-based accelerated tokenizer
A multilingual command line sentence tokenizer in Golang
🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer
High-performance .NET BPE tokenizer — up to 618 MiB/s, competitive with Rust. Zero-allocation counting, multilingual cache, o200k/cl100k/r50k/p50k encodings + HuggingFace tokenizer.json support.
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
