tensorflow/text

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Code for the paper "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer"

Unsupervised text tokenizer for Neural Network-based text generation.

Language model tokenization at GB/s

TensorFlow code and pre-trained models for BERT

NLTK Source

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.

NLP tools for Turkish.