feyninc/chonkiejs

🚀 The Fastest Chunker in the West 🇺🇸 Upto 1TB/s "semantic" chunking, quick and easy!

A fast, lightweight and easy-to-use Python library for splitting text into semantically meaningful chunks.

🦛 CHONK docs with Chonkie ✨ — The lightweight ingestion library for fast, efficient and robust RAG pipelines

Split text into semantic chunks, up to a desired chunk size. Supports calculating length by characters and tokens, and is callable from Rust and Python.

Adaptive Chunking: automatically select the best chunking method per document for RAG. Accepted at LREC 2026.

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Improved file parsing for LLM’s

Language model tokenization at GB/s

modest natural-language processing