feyninc/chunk

Split text into semantic chunks, up to a desired chunk size. Supports calculating length by characters and tokens, and is callable from Rust and Python.

A fast, lightweight and easy-to-use Python library for splitting text into semantically meaningful chunks.

🦛 CHONK your texts with Chonkie ✨ Type-friendly, light-weight, fast and super-simple chunking library

Toolkit to segment text into sentences or other semantic units in a robust, efficient and adaptable way.

Count and truncate text based on tokens

A multilingual command line sentence tokenizer in Golang

Improved file parsing for LLM’s

Adaptive Chunking: automatically select the best chunking method per document for RAG. Accepted at LREC 2026.