coder/ai-tokenizer

Ready-made tokenizer library for working with GPT and tiktoken

VSCode Extension that shows token count of the selected text

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). Port of OpenAI's tiktoken with additional features.

📐 Fast token estimation with 95%+ average accuracy in a 2kB bundle

Now Part of Price Per Token. Web tool to count LLM tokens (GPT, Claude, Llama, ...)

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

High-performance .NET BPE tokenizer — up to 618 MiB/s, competitive with Rust. Zero-allocation counting, multilingual cache, o200k/cl100k/r50k/p50k encodings + HuggingFace tokenizer.json support.

Claude's Tokenizer, Offline, Kinda

Pure Go implementation of OpenAI's tiktoken tokenizer