actual-computer/toks

🍡 30x faster tokenization for every HuggingFace model

Fast Tokens

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Language model tokenization at GB/s

Fast tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.

🤗 Tokenizers.js: A pure JS/TS implementation of today's most used tokenizers

Datamodels for hugging face tokenizers

A high-performance tokenizer (BPE, WordPiece, SentencePiece) built with Rust with Python bindings, focused on speed, safety, and resource optimization.

🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer