daac-tools/vaporetto

🎤 vibrato: Viterbi-based accelerated tokenizer

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Language model tokenization at GB/s

Unsupervised text tokenizer for Neural Network-based text generation.

Fast state-of-the-art tokenizers for Ruby

Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.

🌿 An easy-to-use Japanese Text Processing tool, which makes it possible to switch tokenizers with small changes of code.

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). Port of OpenAI's tiktoken with additional features.