Footer

Your go-to destination for discovering trending open-source projects and uncovering the insights that matter

Legal

  • Terms and conditions
  • Privacy policy

More

  • Pricing
  • Advertise
  • onedump
  • gosnakego
  • Contact us

Social

  • X (Twitter)
  • GitHub
Trendshift owl
What they build speaks louder than words
© 2026 Trendshift, created by Julian Li with ❤️
SignalAdvertise
Trendshift
SignalAdvertise
Sign in
Trendshift owl

Discover trends that matter

SignalAdvertise
DailyWeeklyMonthlyYearly
Live mentionsTopics
RepositoriesDevelopers
InsightsStats
Bifrost
Bifrost

huggingface/datatrove

Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.

Visit GitHub
Python
3.4k
326
65 contributors
Apache License 2.0
AchievementsView all →
FeaturedAdvertise to 180K+ monthly visitors
Bifrost

Fastest Enterprise AI Gateway for Routing, Governing, and Securing all your AI Traffic

Momentum

How fast huggingface/datatrove is gaining stars, and where it lands on Trendshift and GitHub Trending

Activity and trending

Stars, forks and merged pull requests each day or month, lined up with the repository's rank on Trendshift Trending

Sep 5 – Nov 4, 2026

Stars +97 in 61 days

024
4

Forks +32 in 61 days

0123
3

Merged PRs 21 in 61 days

0246
6

Trendshift Trendingwhere the repository ranked each day it made the top 25

Not in the Trendshift top 25 in this period

Sep 5Sep 13Sep 21Sep 29Oct 7Oct 15Oct 23Oct 31
The left and right arrow keys step from day to day; Home and End go to the first and the last

Stars, forks, merged pull requests and Trendshift Trending rank, by day
DayStars gainedForks gainedMerged PRs
Sat, Sep 5, 2026+2+00
Sun, Sep 6, 2026+1+20
Mon, Sep 7, 2026+2+00
Tue, Sep 8, 2026+1+00
Wed, Sep 9, 2026+0+00
Thu, Sep 10, 2026+1+00
Fri, Sep 11, 2026+0+10
Sat, Sep 12, 2026+1+00
Sun, Sep 13, 2026+3+00
Mon, Sep 14, 2026+2+20
Tue, Sep 15, 2026+2+00
Wed, Sep 16, 2026+2+00
Thu, Sep 17, 2026+3+13
Fri, Sep 18, 2026+1+00
Sat, Sep 19, 2026+0+00
Sun, Sep 20, 2026+1+00
Mon, Sep 21, 2026+2+10
Tue, Sep 22, 2026+4+10
Wed, Sep 23, 2026+3+10
Thu, Sep 24, 2026+1+01
Fri, Sep 25, 2026+3+20
Sat, Sep 26, 2026+2+10
Sun, Sep 27, 2026+2+00
Mon, Sep 28, 2026+2+04
Tue, Sep 29, 2026+3+26
Wed, Sep 30, 2026+2+03
Thu, Oct 1, 2026+0+00
Fri, Oct 2, 2026+2+00
Sat, Oct 3, 2026+0+00
Sun, Oct 4, 2026+1+00
Mon, Oct 5, 2026+3+00
Tue, Oct 6, 2026+1+00
Wed, Oct 7, 2026+0+00
Thu, Oct 8, 2026+3+00
Fri, Oct 9, 2026+2+00
Sat, Oct 10, 2026+1+10
Sun, Oct 11, 2026+0+00
Mon, Oct 12, 2026+0+20
Tue, Oct 13, 2026+3+10
Wed, Oct 14, 2026+4+10
Thu, Oct 15, 2026+3+00
Fri, Oct 16, 2026+1+10
Sat, Oct 17, 2026+0+10
Sun, Oct 18, 2026+2+10
Mon, Oct 19, 2026+1+10
Tue, Oct 20, 2026+1+00
Wed, Oct 21, 2026+3+20
Thu, Oct 22, 2026+1+00
Fri, Oct 23, 2026+2+32
Sat, Oct 24, 2026+2+00
Sun, Oct 25, 2026+2+00
Mon, Oct 26, 2026+2+10
Tue, Oct 27, 2026+2+02
Wed, Oct 28, 2026+1+00
Thu, Oct 29, 2026+0+10
Fri, Oct 30, 2026+2+00
Sat, Oct 31, 2026+2+20
Sun, Nov 1, 2026+1+00
Mon, Nov 2, 2026+0+00
Tue, Nov 3, 2026+0+00
Wed, Nov 4, 2026+3+00
Total+97+3221
  • 1#1 on the list
  • 7Rank that day
  • Not trending
15

    • GitHub trending history

    Where the repository ranked on GitHub Trending, one dot for each day it was on a list

    Python

    Jan 22 – Jun 3, 2024Best #15

    #1#5#10#15#20#25
    Top 10
    Feb 2024MarAprMayJun
    The left and right arrow keys step from day to day; Home and End go to the first and the last

    Mentions and achievements

    Where people are talking about huggingface/datatrove, and every badge it has earned
    1. Sep 131 post

      • • Runs locally or on Slurm, Ray, and Jobs executors • Includes prebuilt blocks for extraction, filtering, tokenization, and dedup • Supports fsspec file systems, including local, remote, and S3

        Read full on X, posted by @the_osps

        @the_osps · x.com

    2. 2 months earlier
    3. Jul 151 post

      • I processed 1.2 million @CommonCrawl pages for less than $1. No Slurm cluster. DataTrove pipelines can now run on @huggingface Jobs: → same pipeline code, different executor → tasks fanned out across…

        Read full on X, posted by @vanstriendaniel

        @vanstriendaniel · x.com

    4. 2 months earlier
    5. May 131 post

      • Garbage In, Garbage Out – Fix Your Inputs Before They Ruin Your RAG or LLM Pipeline

        Read full on Reddit

        r/WebAfterAI

    You might also like

    Repositories close to huggingface/datatrove
    FeaturedAdvertise to 180K+ monthly visitors
    Bifrost

    Fastest Enterprise AI Gateway for Routing, Governing, and Securing all your AI Traffic

    lotus-data/lotus

    Python

    Optimized Agentic and LLM Bulk Processing Over Your Data

    1.7k180

    HojiChar/HojiChar

    Python

    The robust text processing pipeline framework enabling customizable, efficient, and metric-logged text preprocessing.

    12714

    NVIDIA-NeMo/Curator

    Python

    Scalable data pre processing and curation toolkit for LLMs

    1.9k366

    togethercomputer/RedPajama-Data

    Python

    The RedPajama-Data repository contains code for preparing large datasets for training large language models.

    5k381

    anishathalye/semlib

    Python

    Build data processing and data analysis pipelines that leverage the power of LLMs 🧠

    2639

    allenai/dolma

    Python

    Data and tools for generating and inspecting OLMo pre-training data.

    1.6k209

    marcelroed/gigatoken

    Rust

    Language model tokenization at GB/s

    4.2k261

    google-research/deduplicate-text-datasets

    Rust
    1.3k130

    datajuicer/data-juicer

    Python

    Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷

    7.2k460