Footer

Your go-to destination for discovering trending open-source projects and uncovering the insights that matter

Legal

  • Terms and conditions
  • Privacy policy

More

  • Pricing
  • Advertise
  • onedump
  • gosnakego
  • Contact us

Social

  • X (Twitter)
  • GitHub
Trendshift owl
What they build speaks louder than words
© 2026 Trendshift, created by Julian Li with ❤️
SignalAdvertise
Trendshift
SignalAdvertise
Sign in
Trendshift owl

Discover trends that matter

DailyWeeklyMonthlyYearly
Live mentionsTopics
RepositoriesDevelopers
InsightsStats
Bifrost
Bifrost

chahatsagarmain/distributed-web-crawler

a distributed web crawler

#Web scraping
Visit GitHub
Go
21
1 contributors
Apache License 2.0
FeaturedAdvertise to 180K+ monthly visitors
Bifrost

Enterprise AI Gateway Built For Scale: Route, Govern, and Secure All Your AI Traffic

Momentum

How fast chahatsagarmain/distributed-web-crawler is gaining stars, and where it lands on Trendshift and GitHub Trending

Activity and trending

Stars, forks and merged pull requests each day or month, lined up with the repository's rank on Trendshift Trending

Jun 2026 – Oct 2026

Stars +21 in 5 months

01020
18

Forks +0 in 5 months

01

Merged PRs 0 in 5 months

01

Trendshift Trendingthe best rank each month, on any day it made the top 25

Not in the Trendshift top 25 in this period

Jun 2026Jul 2026Aug 2026Sep 2026Oct 2026
The left and right arrow keys step from month to month; Home and End go to the first and the last

Stars, forks, merged pull requests and best Trendshift Trending rank, by month
MonthStars gainedForks gainedMerged PRs
June 2026+18+00
July 2026+3+00
August 2026+0+00
September 2026+0+00
October 2026+0+00
Total+21+00
  • 1#1 on the list
  • 7Best rank that month
  • Not trending

Social mentions

Recent discussions about this repository across the web
  1. Jun 143 posts

    • Built a Distributed Web Crawler with Go, Kubernetes, RabbitMQ, Redis & MongoDB

      Read full on Reddit

      r/developersIndia

    • Built a Distributed Web Crawler with Go, Kubernetes, RabbitMQ, Redis & MongoDB

      Read full on Reddit

      r/SideProject

    • Wrote a piece on designing and implementing a distributed web crawler that can scale. I had this project in mind for a few years and finally brought it to life this summer. Have a read here:

      Read full on X, posted by @chahatsgr

      @chahatsgr · x.com

You might also like

Repositories close to chahatsagarmain/distributed-web-crawler
FeaturedAdvertise to 180K+ monthly visitors
Bifrost

Enterprise AI Gateway Built For Scale: Route, Govern, and Secure All Your AI Traffic

crawlab-team/crawlab

Go

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

12.3k1.9k

andeya/pholcus

Go

Pholcus is a distributed high-concurrency crawler software written in pure golang

7.6k1.6k

bitmakerla/estela

TypeScript

estela, an elastic web scraping cluster 🕸

20218

crawler-commons/url-frontier

Java

API definition, resources and reference implementation of URL Frontiers

6716

apache/nutch

Java

Apache Nutch is an extensible and scalable web crawler

3.3k1.3k

EasyPi/docker-scrapyd

Dockerfile

🕷️ Scrapyd is an application for deploying and running Scrapy spiders.

8723

dotnetcore/DotnetSpider

C#

DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

4.1k1.1k

groktopus/groktocrawl

Python

Self-hosted, API-compatible Firecrawl alternative with Agent endpoint. MIT license. One docker compose up.

8916

rmax/scrapy-redis

Python

Redis-based components for Scrapy.

5.6k1.5k