simonw/ttok

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

Ready-made tokenizer library for working with GPT and tiktoken

Split text into semantic chunks, up to a desired chunk size. Supports calculating length by characters and tokens, and is callable from Rust and Python.

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). Port of OpenAI's tiktoken with additional features.

Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.

🚀 The Fastest Chunker in the West 🇺🇸 Upto 1TB/s "semantic" chunking, quick and easy!

Language model tokenization at GB/s

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Claude Usage Tracker