isaacus-dev/semchunk

Split text into semantic chunks, up to a desired chunk size. Supports calculating length by characters and tokens, and is callable from Rust and Python.

Toolkit to segment text into sentences or other semantic units in a robust, efficient and adaptable way.

🦛 CHONK docs with Chonkie ✨ — The lightweight ingestion library for fast, efficient and robust RAG pipelines

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Language model tokenization at GB/s

A minimal Python package for storing and retrieving text using chunking, embeddings, and vector search.

Improved file parsing for LLM’s

Unsupervised text tokenizer for Neural Network-based text generation.

tiktoken is a fast BPE tokeniser for use with OpenAI's models.