huggingface/tokenizers

Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

Unsupervised text tokenizer for Neural Network-based text generation.

NLTK Source

💫 Industrial-strength Natural Language Processing (NLP) in Python

modest natural-language processing

🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages

TensorFlow code and pre-trained models for BERT