cimeister/tokenizer-intrinsic-evals

Simple-to-use scoring function for arbitrarily tokenized texts.

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

VSCode Extension that shows token count of the selected text

📐 Fast token estimation with 95%+ average accuracy in a 2kB bundle

A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.

Count and truncate text based on tokens

High-performance .NET BPE tokenizer — up to 618 MiB/s, competitive with Rust. Zero-allocation counting, multilingual cache, o200k/cl100k/r50k/p50k encodings + HuggingFace tokenizer.json support.

Code for the paper "BPE stays on SCRIPT", "Which Pieces Does Unigram Tokenization Really Need?" and MinGram

Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.