Footer

Your go-to destination for discovering trending open-source projects and uncovering the insights that matter

Legal

  • Terms and conditions
  • Privacy policy

More

  • Pricing
  • Advertise
  • onedump
  • gosnakego
  • Contact us

Social

  • X (Twitter)
  • GitHub
Trendshift owl
What they build speaks louder than words
© 2026 Trendshift, created by Julian Li with ❤️
SignalAdvertise
Trendshift
SignalAdvertise
Sign in
Trendshift owl

Discover trends that matter

DailyWeeklyMonthlyYearly
Live mentionsTopics
RepositoriesDevelopers
InsightsStats
Bifrost
Bifrost

Systemcluster/kitoken

Fast tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.

Visit GitHub
Rust
62
2
2 contributors
BSD 2-Clause "Simplified" License
website
FeaturedAdvertise to 180K+ monthly visitors
Bifrost

Fastest Enterprise AI Gateway for Routing, Governing, and Securing all your AI Traffic

Momentum

How fast Systemcluster/kitoken is gaining stars, and where it lands on Trendshift and GitHub Trending

Activity and trending

Stars, forks and merged pull requests each day or month, lined up with the repository's rank on Trendshift Trending

Sep 1 – Oct 27, 2026

Stars +4 in 57 days

01
1

Forks +0 in 57 days

01

Merged PRs 0 in 57 days

01

Trendshift Trendingwhere the repository ranked each day it made the top 25

Not in the Trendshift top 25 in this period

Sep 1Sep 9Sep 17Sep 25Oct 3Oct 11Oct 19Oct 27
The left and right arrow keys step from day to day; Home and End go to the first and the last

Stars, forks, merged pull requests and Trendshift Trending rank, by day
DayStars gainedForks gainedMerged PRs
Tue, Sep 1, 2026+1+00
Wed, Sep 2, 2026+0+00
Thu, Sep 3, 2026+0+00
Fri, Sep 4, 2026+0+00
Sat, Sep 5, 2026+0+00
Sun, Sep 6, 2026+0+00
Mon, Sep 7, 2026+0+00
Tue, Sep 8, 2026+0+00
Wed, Sep 9, 2026+0+00
Thu, Sep 10, 2026+0+00
Fri, Sep 11, 2026+0+00
Sat, Sep 12, 2026+0+00
Sun, Sep 13, 2026+0+00
Mon, Sep 14, 2026+0+00
Tue, Sep 15, 2026+0+00
Wed, Sep 16, 2026+0+00
Thu, Sep 17, 2026+0+00
Fri, Sep 18, 2026+0+00
Sat, Sep 19, 2026+0+00
Sun, Sep 20, 2026+0+00
Mon, Sep 21, 2026+1+00
Tue, Sep 22, 2026+1+00
Wed, Sep 23, 2026+0+00
Thu, Sep 24, 2026+1+00
Fri, Sep 25, 2026+0+00
Sat, Sep 26, 2026+0+00
Sun, Sep 27, 2026+0+00
Mon, Sep 28, 2026+0+00
Tue, Sep 29, 2026+0+00
Wed, Sep 30, 2026+0+00
Thu, Oct 1, 2026+0+00
Fri, Oct 2, 2026+0+00
Sat, Oct 3, 2026+0+00
Sun, Oct 4, 2026+0+00
Mon, Oct 5, 2026+0+00
Tue, Oct 6, 2026+0+00
Wed, Oct 7, 2026+0+00
Thu, Oct 8, 2026+0+00
Fri, Oct 9, 2026+0+00
Sat, Oct 10, 2026+0+00
Sun, Oct 11, 2026+0+00
Mon, Oct 12, 2026+0+00
Tue, Oct 13, 2026+0+00
Wed, Oct 14, 2026+0+00
Thu, Oct 15, 2026+0+00
Fri, Oct 16, 2026+0+00
Sat, Oct 17, 2026+0+00
Sun, Oct 18, 2026+0+00
Mon, Oct 19, 2026+0+00
Tue, Oct 20, 2026+0+00
Wed, Oct 21, 2026+0+00
Thu, Oct 22, 2026+0+00
Fri, Oct 23, 2026+0+00
Sat, Oct 24, 2026+0+00
Sun, Oct 25, 2026+0+00
Mon, Oct 26, 2026+0+00
Tue, Oct 27, 2026+0+00
Total+4+00
  • 1#1 on the list
  • 7Rank that day
  • Not trending

You might also like

Repositories close to Systemcluster/kitoken
FeaturedAdvertise to 180K+ monthly visitors
Bifrost

Fastest Enterprise AI Gateway for Routing, Governing, and Securing all your AI Traffic

huggingface/tokenizers

Rust

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

11.2k1.2k

feyninc/tokie

Rust

🍡 30x faster tokenization for every HuggingFace model

555

ml-rust/splintr

Rust

A high-performance tokenizer (BPE, WordPiece, SentencePiece) built with Rust with Python bindings, focused on speed, safety, and resource optimization.

616

sanderland/script_tok

Python

Code for the paper "BPE stays on SCRIPT", "Which Pieces Does Unigram Tokenization Really Need?" and MinGram

225

crusoecloud/fastokens

Rust

Fast Tokens

15528

huggingface/tokenizers.js

TypeScript

🤗 Tokenizers.js: A pure JS/TS implementation of today's most used tokenizers

5818

google/sentencepiece

C++

Unsupervised text tokenizer for Neural Network-based text generation.

12.2k1.4k

marcelroed/gigatoken

Rust

Language model tokenization at GB/s

4.1k259

daac-tools/vaporetto

Rust

🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer

29812