sefineh-ai/Amharic-Tokenizer

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

🏷 བོད་ཏོག [pʰøtɔk̚] Tibetan word tokenizer in Python

🎤 vibrato: Viterbi-based accelerated tokenizer

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer

Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.

Unsupervised text tokenizer for Neural Network-based text generation.

Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.

High-performance .NET BPE tokenizer — up to 618 MiB/s, competitive with Rust. Zero-allocation counting, multilingual cache, o200k/cl100k/r50k/p50k encodings + HuggingFace tokenizer.json support.