marcelroed/gigatoken

Language model tokenization at GB/s

Rust

4.1k

257

1 contributors

MIT License

FeaturedAdvertise to 180K+ monthly visitors

Enterprise AI Gateway Built For Scale: Route, Govern, and Secure All Your AI Traffic

Control what AI can do before it acts

Give Al Agents a Deeper Understanding of Your Codebase with a Code Graph

Momentum

How fast marcelroed/gigatoken is gaining stars, and where it lands on Trendshift and GitHub Trending

Activity and trending

Stars, forks and merged pull requests each day or month, lined up with the repository's rank on Trendshift Trending

Aug 22 – Oct 21, 2026

Stars +91 in 61 days

6

Forks +38 in 61 days

8

Merged PRs 0 in 61 days

Trendshift Trendingwhere the repository ranked each day it made the top 25

Not in the Trendshift top 25 in this period

Aug 22Aug 30Sep 7Sep 15Sep 23Oct 1Oct 9Oct 17

Stars, forks, merged pull requests and Trendshift Trending rank, by day
Day	Stars gained	Forks gained	Merged PRs
Sat, Aug 22, 2026	+0	+0	0
Sun, Aug 23, 2026	+2	+0	0
Mon, Aug 24, 2026	+6	+1	0
Tue, Aug 25, 2026	+4	+0	0
Wed, Aug 26, 2026	+2	+1	0
Thu, Aug 27, 2026	+1	+0	0
Fri, Aug 28, 2026	+2	+0	0
Sat, Aug 29, 2026	+4	+2	0
Sun, Aug 30, 2026	+1	+0	0
Mon, Aug 31, 2026	+4	+0	0
Tue, Sep 1, 2026	+2	+1	0
Wed, Sep 2, 2026	+0	+0	0
Thu, Sep 3, 2026	+0	+0	0
Fri, Sep 4, 2026	+2	+0	0
Sat, Sep 5, 2026	+1	+0	0
Sun, Sep 6, 2026	+0	+0	0
Mon, Sep 7, 2026	+2	+0	0
Tue, Sep 8, 2026	+2	+0	0
Wed, Sep 9, 2026	+3	+0	0
Thu, Sep 10, 2026	+2	+0	0
Fri, Sep 11, 2026	+4	+0	0
Sat, Sep 12, 2026	+2	+1	0
Sun, Sep 13, 2026	+2	+2	0
Mon, Sep 14, 2026	+1	+0	0
Tue, Sep 15, 2026	+3	+0	0
Wed, Sep 16, 2026	+2	+0	0
Thu, Sep 17, 2026	+1	+0	0
Fri, Sep 18, 2026	+0	+1	0
Sat, Sep 19, 2026	+3	+0	0
Sun, Sep 20, 2026	+1	+0	0
Mon, Sep 21, 2026	+4	+1	0
Tue, Sep 22, 2026	+1	+0	0
Wed, Sep 23, 2026	+0	+0	0
Thu, Sep 24, 2026	+0	+0	0
Fri, Sep 25, 2026	+0	+0	0
Sat, Sep 26, 2026	+2	+0	0
Sun, Sep 27, 2026	+2	+0	0
Mon, Sep 28, 2026	+2	+0	0
Tue, Sep 29, 2026	+0	+0	0
Wed, Sep 30, 2026	+1	+0	0
Thu, Oct 1, 2026	+2	+0	0
Fri, Oct 2, 2026	+0	+0	0
Sat, Oct 3, 2026	+1	+0	0
Sun, Oct 4, 2026	+0	+0	0
Mon, Oct 5, 2026	+0	+0	0
Tue, Oct 6, 2026	+0	+0	0
Wed, Oct 7, 2026	+0	+0	0
Thu, Oct 8, 2026	+0	+0	0
Fri, Oct 9, 2026	+1	+0	0
Sat, Oct 10, 2026	+1	+0	0
Sun, Oct 11, 2026	+0	+0	0
Mon, Oct 12, 2026	+2	+1	0
Tue, Oct 13, 2026	+1	+0	0
Wed, Oct 14, 2026	+3	+3	0
Thu, Oct 15, 2026	+2	+1	0
Fri, Oct 16, 2026	+1	+2	0
Sat, Oct 17, 2026	+0	+8	0
Sun, Oct 18, 2026	+1	+5	0
Mon, Oct 19, 2026	+4	+7	0
Tue, Oct 20, 2026	+1	+1	0
Wed, Oct 21, 2026	+0	+0	0
Total	+91	+38	0

#1 on the list
Rank that day
Not trending

Mentions and achievements

Where people are talking about marcelroed/gigatoken, and every badge it has earned

Jul 301 post
- BTW, the whole typing / streaming thing that most LLMs do and we now assume as the default is a hack to help us perceive response is faster.
  Read full on X, posted by @abhinav
  @abhinav · x.com
Jul 291 post
- While this won't make your training run 1000x faster it will help massively on the data side: things like corpus prep, mid-training data experiments, tokenizer ablations. A pass over a dataset would…
  Read full on X, posted by @joinaicircle
  @joinaicircle · x.com
Jul 233 posts6 posts
- Someone rebuilt LLM tokenization and their benchmarks look silly: 8.79 GB/s on an M4 Max where HuggingFace's tokenizers do 6.9 MB/s. The wild part: HF's baseline is already multithreaded Rust. pip…
  Read full on X, posted by @NewsOfLinux
  @NewsOfLinux · x.com
- Meet Gigatoken: A Rust BPE Tokenizer that Encodes Text at 24.53 GB/s, up to 989x Faster than HuggingFace Tokenizers
  Read full on Reddit
  r/AIDeveloperNews
- Meet Gigatoken: A Rust BPE Tokenizer that Encodes Text at 24.53 GB/s, up to 989x Faster than HuggingFace Tokenizers
  Read full on Reddit
  r/OpenSourceeAI
- Meet Gigatoken: A Rust BPE Tokenizer that Encodes Text at 24.53 GB/s, up to 989x Faster than HuggingFace Tokenizers
  Read full on Reddit
  r/machinelearningnews
- This guy built a tokenizer 1000× faster 🔥 It’s optimized for CPU - good for local LLMs ps: faster tokenzier won’t speed up the entire LLM responses, but it makes the some steps much quicker.
  Read full on X, posted by @hieuSSR
  @hieuSSR · x.com
- HN 今天在刷 GigaToken... Rust tokenizer, 號稱比 HF tokenizers 快到 ~1000x. 大核上 GPT-2 大概 24 GB/s vs HF 的 MB/s 等級, pip install 就能 drop-in HF / tiktoken. 資料前處理卡在 tokenize 的人應該會香
  Read full on X, posted by @shooding
  @shooding · x.com
Jul 222 posts
- The reported 989x GPT-2 tokenizer speedup comes from replacing regex with vectorized text splitting, caching repeated token mappings, and reading files without bouncing through Python. At 24.5 GB/s,…
  Read full on X, posted by @sakurayukiai
  @sakurayukiai · x.com
- GigaToken: ~1000x faster Language model tokenization
  Read full on Reddit
  r/hackernews