tiktoken is a fast BPE tokeniser for use with OpenAI's models.
Like openai/tiktoken, 0 likes 0 Bookmark openai/tiktoken, 0 bookmarks 0 Ready-made tokenizer library for working with GPT and tiktoken
Like zurawiki/tiktoken-rs, 0 likes 0 Bookmark zurawiki/tiktoken-rs, 0 bookmarks 0 High-performance .NET BPE tokenizer — up to 618 MiB/s, competitive with Rust. Zero-allocation counting, multilingual cache, o200k/cl100k/r50k/p50k encodings + HuggingFace tokenizer.json support.
Like tryAGI/Tiktoken, 0 likes 0 Bookmark tryAGI/Tiktoken, 0 bookmarks 0 Unofficial ruby binding for tiktoken by way of rust
Like IAPark/tiktoken_ruby, 0 likes 0 Bookmark IAPark/tiktoken_ruby, 0 bookmarks 0 A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.
Like coder/ai-tokenizer, 0 likes 0 Bookmark coder/ai-tokenizer, 0 bookmarks 0 The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). Port of OpenAI's tiktoken with additional features.
Like niieani/gpt-tokenizer, 0 likes 0 Bookmark niieani/gpt-tokenizer, 0 bookmarks 0 Count and truncate text based on tokens
Like simonw/ttok, 0 likes 0 Bookmark simonw/ttok, 0 bookmarks 0 Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.
Like microsoft/Tokenizer, 0 likes 0 Bookmark microsoft/Tokenizer, 0 bookmarks 0 Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.
Like karpathy/minbpe, 0 likes 0 Bookmark karpathy/minbpe, 0 bookmarks 0 
