ankane/tokenizers-ruby

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Unofficial ruby binding for tiktoken by way of rust

Efficient text classification and representation learning for Ruby

🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer

State-of-the-art transformers for Ruby

🎤 vibrato: Viterbi-based accelerated tokenizer

Language model tokenization at GB/s

Unsupervised text tokenizer for Neural Network-based text generation.

🌿 An easy-to-use Japanese Text Processing tool, which makes it possible to switch tokenizers with small changes of code.