Footer

Your go-to destination for discovering trending open-source projects and uncovering the insights that matter

Legal

  • Terms and conditions
  • Privacy policy

More

  • Pricing
  • Advertise
  • onedump
  • gosnakego
  • Contact us

Social

  • X (Twitter)
  • GitHub
Trendshift owl
What they build speaks louder than words
© 2026 Trendshift, created by Julian Li with ❤️
SignalAdvertise
Trendshift
SignalAdvertise
Sign in
Trendshift owl

Discover trends that matter

DailyWeeklyMonthlyYearly
Live mentionsTopics
RepositoriesDevelopers
InsightsStats
Bifrost
Bifrost
Busbar
Busbar
AxiomCode
AxiomCode

spatie/robots-txt

Determine if a page may be crawled from robots.txt, robots meta tags and robot headers

Visit GitHub
PHP
258
45
34 contributors
MIT License
website
FeaturedAdvertise to 180K+ monthly visitors
Bifrost

Enterprise AI Gateway Built For Scale: Route, Govern, and Secure All Your AI Traffic

Busbar

Control what AI can do before it acts

AxiomCode

Give Al Agents a Deeper Understanding of Your Codebase with a Code Graph

Momentum

How fast spatie/robots-txt is gaining stars, and where it lands on Trendshift and GitHub Trending

Activity and trending

Stars, forks and merged pull requests each day or month, lined up with the repository's rank on Trendshift Trending

Oct 14 – Oct 19, 2026

Stars +0 in 6 days

01

Forks +1 in 6 days

01
1

Merged PRs 0 in 6 days

01

Trendshift Trendingwhere the repository ranked each day it made the top 25

Not in the Trendshift top 25 in this period

Oct 14Oct 15Oct 16Oct 17Oct 18Oct 19
The left and right arrow keys step from day to day; Home and End go to the first and the last

Stars, forks, merged pull requests and Trendshift Trending rank, by day
DayStars gainedForks gainedMerged PRs
Wed, Oct 14, 2026+0+10
Thu, Oct 15, 2026+0+00
Fri, Oct 16, 2026+0+00
Sat, Oct 17, 2026+0+00
Sun, Oct 18, 2026+0+00
Mon, Oct 19, 2026+0+00
Total+0+10
  • 1#1 on the list
  • 7Rank that day
  • Not trending

You might also like

Repositories close to spatie/robots-txt
FeaturedAdvertise to 180K+ monthly visitors
AxiomCode

Give Al Agents a Deeper Understanding of Your Codebase with a Code Graph

Bifrost

Enterprise AI Gateway Built For Scale: Route, Govern, and Secure All Your AI Traffic

Busbar

Control what AI can do before it acts

samclarke/robots-parser

JavaScript

NodeJS robots.txt parser with support for wildcard (*) matching.

16819

JayBizzle/Crawler-Detect

PHP

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

2.4k277

omrilotan/isbot

TypeScript

🤖/👨‍🦰 Detect bots/crawlers/spiders using the user agent string

1.2k85

monperrus/crawler-user-agents

Go

Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders. pull-request welcome ⭐

1.4k284

ai-robots-txt/ai.robots.txt

Python

A list of AI agents and robots to block.

4.2k198

duckduckgo/tracker-radar-collector

JavaScript

🕸 Modular, multithreaded, puppeteer-based crawler

16969

dmi3kno/polite

R

Be nice on the web

33512

nuxt-modules/robots

TypeScript

Tame the robots crawling and indexing your Nuxt site.

52854

scrapfly/Antibot-Detector

JavaScript

Real-time detection of anti-bot systems, captchas & fingerprinting techniques. Identifies Cloudflare, Akamai, DataDome, reCAPTCHA, hCaptcha, Shape Security & more with confidence scoring and advanced capture tools.

61171