benbrandt/text-splitter

A fast, lightweight and easy-to-use Python library for splitting text into semantically meaningful chunks.

Toolkit to segment text into sentences or other semantic units in a robust, efficient and adaptable way.

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

A multilingual command line sentence tokenizer in Golang

Improved file parsing for LLM’s

Semantic search and document parsing tools for the command line

Language model tokenization at GB/s

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.