aurelio-labs/semantic-chunkers

🚀 The Fastest Chunker in the West 🇺🇸 Upto 1TB/s "semantic" chunking, quick and easy!

A fast, lightweight and easy-to-use Python library for splitting text into semantically meaningful chunks.

🍱 Semantically create chunks from large document for passing to LLM workflows

🦛 CHONK your texts with Chonkie ✨ Type-friendly, light-weight, fast and super-simple chunking library

One library to split them all: Sentence, Code, Docs. Chunk smarter, not harder; built for LLMs, RAG pipelines, and beyond.

A library for semantically coherent text chunking

Split text into semantic chunks, up to a desired chunk size. Supports calculating length by characters and tokens, and is callable from Rust and Python.

This repository presents the original implementation of LumberChunker: Long-Form Narrative Document Segmentation by André V. Duarte, João Marques, Miguel Graça, Miguel Freire, Lei Li and Arlindo L. Oliveira (accepted at EMNLP 2024 Findings)

Toolkit to segment text into sentences or other semantic units in a robust, efficient and adaptable way.