Footer

Your go-to destination for discovering trending open-source projects and uncovering the insights that matter

Legal

  • Terms and conditions
  • Privacy policy

More

  • Pricing
  • Advertise
  • onedump
  • gosnakego
  • Contact us

Social

  • X (Twitter)
  • GitHub
Trendshift owl
What they build speaks louder than words
© 2026 Trendshift, created by Julian Li with ❤️
SignalAdvertise
Trendshift
SignalAdvertise
Sign in
Trendshift owl

Discover trends that matter

SignalAdvertise
DailyWeeklyMonthlyYearly
Live mentionsTopics
RepositoriesDevelopers
InsightsStats
Bifrost
Bifrost

openai/tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

Visit GitHub
Python
19.6k
1.6k
24 contributors
MIT License
AchievementsView all →
FeaturedAdvertise to 180K+ monthly visitors
Bifrost

Fastest Enterprise AI Gateway for Routing, Governing, and Securing all your AI Traffic

Momentum

How fast openai/tiktoken is gaining stars, and where it lands on Trendshift and GitHub Trending

Activity and trending

Stars, forks and merged pull requests each day or month, lined up with the repository's rank on Trendshift Trending

Sep 5 – Nov 4, 2026

Stars +432 in 61 days

051015
15

Forks +88 in 61 days

0246
6

Merged PRs 0 in 61 days

01

Trendshift Trendingwhere the repository ranked each day it made the top 25

Not in the Trendshift top 25 in this period

Sep 5Sep 13Sep 21Sep 29Oct 7Oct 15Oct 23Oct 31
The left and right arrow keys step from day to day; Home and End go to the first and the last

Stars, forks, merged pull requests and Trendshift Trending rank, by day
DayStars gainedForks gainedMerged PRs
Sat, Sep 5, 2026+15+00
Sun, Sep 6, 2026+7+00
Mon, Sep 7, 2026+9+20
Tue, Sep 8, 2026+11+30
Wed, Sep 9, 2026+2+00
Thu, Sep 10, 2026+7+20
Fri, Sep 11, 2026+4+00
Sat, Sep 12, 2026+8+20
Sun, Sep 13, 2026+6+10
Mon, Sep 14, 2026+8+20
Tue, Sep 15, 2026+8+20
Wed, Sep 16, 2026+12+00
Thu, Sep 17, 2026+11+10
Fri, Sep 18, 2026+4+10
Sat, Sep 19, 2026+2+00
Sun, Sep 20, 2026+7+10
Mon, Sep 21, 2026+9+20
Tue, Sep 22, 2026+9+00
Wed, Sep 23, 2026+3+20
Thu, Sep 24, 2026+9+10
Fri, Sep 25, 2026+5+10
Sat, Sep 26, 2026+7+00
Sun, Sep 27, 2026+4+20
Mon, Sep 28, 2026+5+20
Tue, Sep 29, 2026+4+00
Wed, Sep 30, 2026+10+40
Thu, Oct 1, 2026+8+10
Fri, Oct 2, 2026+7+10
Sat, Oct 3, 2026+7+00
Sun, Oct 4, 2026+4+20
Mon, Oct 5, 2026+10+20
Tue, Oct 6, 2026+5+10
Wed, Oct 7, 2026+12+60
Thu, Oct 8, 2026+6+20
Fri, Oct 9, 2026+10+10
Sat, Oct 10, 2026+10+30
Sun, Oct 11, 2026+7+10
Mon, Oct 12, 2026+2+30
Tue, Oct 13, 2026+8+20
Wed, Oct 14, 2026+6+10
Thu, Oct 15, 2026+5+10
Fri, Oct 16, 2026+5+10
Sat, Oct 17, 2026+11+30
Sun, Oct 18, 2026+11+20
Mon, Oct 19, 2026+1+00
Tue, Oct 20, 2026+5+30
Wed, Oct 21, 2026+5+10
Thu, Oct 22, 2026+6+10
Fri, Oct 23, 2026+6+20
Sat, Oct 24, 2026+5+20
Sun, Oct 25, 2026+8+10
Mon, Oct 26, 2026+4+00
Tue, Oct 27, 2026+1+10
Wed, Oct 28, 2026+11+30
Thu, Oct 29, 2026+13+20
Fri, Oct 30, 2026+5+00
Sat, Oct 31, 2026+7+00
Sun, Nov 1, 2026+10+10
Mon, Nov 2, 2026+8+20
Tue, Nov 3, 2026+8+50
Wed, Nov 4, 2026+9+00
Total+432+880
  • 1#1 on the list
  • 7Rank that day
  • Not trending
3

    • GitHub trending history

    Where the repository ranked on GitHub Trending, one dot for each day it was on a list

    May 15, 2024Best #22

    #1#5#10#15#20#25
    Top 10
    Jun 2024AugOctDecFebAprJunAug
    The left and right arrow keys step from day to day; Home and End go to the first and the last

    Mentions and achievements

    Where people are talking about openai/tiktoken, and every badge it has earned
    1. Nov 41 post

      • Your pretty-printed JSON is costing you almost 2x the tokens. I ran the same 5 rows of data through tiktoken tonight in 5 formats. GPT-4o tokenizer. Here's the damage: 1. Pretty JSON: 137 tokens 2.…

        Read full on X, posted by @ReignSwords

        @ReignSwords · x.com

    2. Nov 31 post

      • The whole pipeline in one picture. JSON, code, numbers and non-English text split into more pieces, so they cost more and fill context faster. Tokenizers differ too: that Hindi line is 25 tokens on…

        Read full on X, posted by @akashsairaja

        @akashsairaja · x.com

    3. 3 months earlier
    4. Aug 161 post

      • everyone measures the tokenizer in bytes. your invoice is not in bytes. in tokens: hindi is 2.12x english, not 3.5x. japanese is 2.00x, not 1.25x. and opus 4.6 to 4.7 is the same $5 per million for…

        Read full on X, posted by @SAAANJI1337

        @SAAANJI1337 · x.com

    5. 2 months earlier
    6. Jun 121 post

      • Middle ground: start with each byte as a token, and build up a vocab by merging most frequently co-occurring tokens, until the vocab budget (e.g. ~200k for GPT4o) is exhausted -- called Byte Pair…

        Read full on X, posted by @jasraj_singh11

        @jasraj_singh11 · x.com

    You might also like

    Repositories close to openai/tiktoken
    FeaturedAdvertise to 180K+ monthly visitors
    Bifrost

    Fastest Enterprise AI Gateway for Routing, Governing, and Securing all your AI Traffic

    tryAGI/Tiktoken

    C#

    High-performance .NET BPE tokenizer — up to 618 MiB/s, competitive with Rust. Zero-allocation counting, multilingual cache, o200k/cl100k/r50k/p50k encodings + HuggingFace tokenizer.json support.

    936

    zurawiki/tiktoken-rs

    Rust

    Ready-made tokenizer library for working with GPT and tiktoken

    40971

    tiktoken-go/tokenizer

    Go

    Pure Go implementation of OpenAI's tiktoken tokenizer

    46239

    karpathy/minbpe

    Python

    Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.

    11k1.2k

    simonw/ttok

    Python

    Count and truncate text based on tokens

    40922

    IAPark/tiktoken_ruby

    Ruby

    Unofficial ruby binding for tiktoken by way of rust

    17031

    coder/ai-tokenizer

    TypeScript

    A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.

    636

    microsoft/Tokenizer

    C#

    Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.

    21334

    niieani/gpt-tokenizer

    TypeScript

    The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). Port of OpenAI's tiktoken with additional features.

    85767