80.8% of the GitHub Stars Given to AI Bot Blocklist Projects Went to a Single Community List

Summary

Picture a small publisher who checks her server bill and sees traffic doubled, but sales did not. The extra visits came from AI crawlers. She asks a developer friend how to block them, and the friend points her to a list on GitHub that names every known AI bot. That list is where most of the blocking movement lives.

AI companies send crawlers to read websites for training data and live answers. Site owners who want to opt out need an up-to-date list of those crawlers for their robots.txt file or firewall. Developers have published hundreds of such lists and tools on GitHub.

The movement is real but thin. Of 470 repos we found, one shared list collected 80.8% of all stars. Most of the rest are one-off copies that never got a single star.

What we measured

We used GH Archive, the public log of GitHub events, in Google BigQuery. We kept repos whose names pair a blocking word with an AI crawler, bot or scraper word, or that describe AI-focused robots.txt rules. We removed probable forks (repos with no forks and at most two stars that share a name with an earlier forked repo) and repos about chatbots or blockchain that matched by accident. A repo's first recorded event stands in for its creation date.

For each repo we counted stars, forks, code pushes, pull request events, issue events and releases. For the leading list we tracked those counts by quarter. To compare years we divided its stars by all stars GH Archive recorded that year.

One shared list carries the whole movement

We grouped the 470 repos by the year they first appeared.

Every year more people publish their own blocklist, but almost nobody stars or reuses them. The leading list also took 136 of the group's 216 forks (63.0%).

The leading list peaked in 2025

Quarterly activity on the ai.robots.txt repository, as recorded by GH Archive.

Stars peaked in early 2025 at 792 in one quarter. Measured against all GitHub stars, the list drew 17.0 per million in 2024, 27.9 in 2025 and 8.7 in 2026. Recorded activity in 2026 is low partly because GH Archive captured fewer events that year, so the decline may be overstated.

Updates come in bursts as new AI bots appear

Code pushes show how often the list itself changes.

The list received pushes in every month of 2025. Pull request events nearly doubled from 2024 to 2025, which may mean more outside contributors were submitting new bot names. Small one-off lists cannot keep pace with that, which may help explain why users gather around one shared source.

What this means

If you decide to block AI crawlers, rely on a maintained list, not a copy you found once and never updated. New AI bots appear several times a year, and a stale robots.txt lets them through. Also know the trade-off. Blocking every AI crawler can keep your business out of AI answers that customers now use to choose a provider. If AI brand visibility matters to you, block the training bots and keep the answer bots.

Many businesses want to block training crawlers but still allow the bots that power AI search answers. That takes a deliberate robots.txt, not a blanket ban. Our free AI visibility audit checks which AI crawlers can reach your site and whether your current rules help or hurt your LLM SEO.

Limits

We matched repo names only. Blocking rules published inside larger projects, or under names without our keywords, are missed. Some name matches may still be unrelated.

GH Archive recorded far fewer star, fork and pull request events in 2026 and had capture gaps in October 2025 and August to October 2026. 2026 counts understate real activity.

Stars measure interest, not how many websites use a list. Many sites copy a list directly without starring it.

Repos are counted by first recorded event, and fork removal is a name-based heuristic.

Frequently asked questions

Related studies

• The open-source tooling around AI searchGEO and AI SEO Repos on GitHub Hit 322 a Month in 2026
• The open-source tooling around AI searchSEO Data MCP Servers on GitHub More Than Doubled in 2026
• The open-source tooling around AI searchllms.txt GitHub Projects: 83.1% Never Got a Star
• The open-source tooling around AI searchAI Browser Agent Repos on GitHub Grew 22-Fold Since 2024