If you're curious about how fast Gigatoken will run on your data and with your tokenizer, you can run it without installing Gigatoken using uv: uvx gigatoken bench <tokenizer-name> <dataset>

Introducing the world's fastest tokenizer implementation, Gigatoken! Gigatoken is ~500-1000x faster than HuggingFace, and ~100x faster than OpenAI's tiktoken for most tokenizer definitions on most…