Footer

Your go-to destination for discovering trending open-source projects and uncovering the insights that matter

Legal

  • Terms and conditions
  • Privacy policy

More

  • Pricing
  • Advertise
  • onedump
  • gosnakego
  • Contact us

Social

  • X (Twitter)
  • GitHub
Trendshift owl
What they build speaks louder than words
© 2026 Trendshift, created by Julian Li with ❤️
SignalAdvertise
Trendshift
SignalAdvertise
Sign in
Trendshift owl

Discover trends that matter

DailyWeeklyMonthlyYearly
Live mentionsTopics
RepositoriesDevelopers
InsightsStats
Bifrost
Bifrost

zouharvi/tokenization-scorer

Simple-to-use scoring function for arbitrarily tokenized texts.

Visit GitHub
Python
52
6
4 contributors
MIT License
website
FeaturedAdvertise to 180K+ monthly visitors
Bifrost

Fastest Enterprise AI Gateway for Routing, Governing, and Securing all your AI Traffic

Momentum

How fast zouharvi/tokenization-scorer is gaining stars, and where it lands on Trendshift and GitHub Trending

Activity and trending

Stars, forks and merged pull requests each day or month, lined up with the repository's rank on Trendshift Trending

Oct 17 – Oct 27, 2026

Stars +0 in 11 days

01

Forks +0 in 11 days

01

Merged PRs 0 in 11 days

01

Trendshift Trendingwhere the repository ranked each day it made the top 25

Not in the Trendshift top 25 in this period

Oct 17Oct 19Oct 21Oct 23Oct 25Oct 27
The left and right arrow keys step from day to day; Home and End go to the first and the last

Stars, forks, merged pull requests and Trendshift Trending rank, by day
DayStars gainedForks gainedMerged PRs
Sat, Oct 17, 2026+0+00
Sun, Oct 18, 2026+0+00
Mon, Oct 19, 2026+0+00
Tue, Oct 20, 2026+0+00
Wed, Oct 21, 2026+0+00
Thu, Oct 22, 2026+0+00
Fri, Oct 23, 2026+0+00
Sat, Oct 24, 2026+0+00
Sun, Oct 25, 2026+0+00
Mon, Oct 26, 2026+0+00
Tue, Oct 27, 2026+0+00
Total+0+00
  • 1#1 on the list
  • 7Rank that day
  • Not trending

You might also like

Repositories close to zouharvi/tokenization-scorer
FeaturedAdvertise to 180K+ monthly visitors
Bifrost

Fastest Enterprise AI Gateway for Routing, Governing, and Securing all your AI Traffic

cimeister/tokenizer-intrinsic-evals

Python

TokEval: intrinsic quality metrics for tokenizers across natural language, code, and math

587

openai/tiktoken

Python

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

19.5k1.6k

johannschopplich/tokenx

TypeScript

📐 Fast token estimation with 95%+ average accuracy in a 2kB bundle

1869

simonw/ttok

Python

Count and truncate text based on tokens

40822

tryAGI/Tiktoken

C#

High-performance .NET BPE tokenizer — up to 618 MiB/s, competitive with Rust. Zero-allocation counting, multilingual cache, o200k/cl100k/r50k/p50k encodings + HuggingFace tokenizer.json support.

926

sanderland/script_tok

Python

Code for the paper "BPE stays on SCRIPT", "Which Pieces Does Unigram Tokenization Really Need?" and MinGram

225

sanderland/ctok

Python

Claude's Tokenizer, Offline, Kinda

977

BedirT/LLM-Token-Counter-VSCode

JavaScript

VSCode Extension that shows token count of the selected text

7311

huggingface/tokenizers

Rust

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

11.2k1.2k