hplt-project/sacremoses

Making text a first-class citizen in TensorFlow.

Library used by Meilisearch to tokenize queries and documents

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

NeMo text processing for ASR and TTS

Unsupervised text tokenizer for Neural Network-based text generation.

Language model tokenization at GB/s

Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages

NLTK Source

🎤 vibrato: Viterbi-based accelerated tokenizer