google/sentencepiece

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Code for the paper "BPE stays on SCRIPT", "Which Pieces Does Unigram Tokenization Really Need?" and MinGram

Fast tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.

Language model tokenization at GB/s

Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.

🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer

🍡 30x faster tokenization for every HuggingFace model

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

Making text a first-class citizen in TensorFlow.