45.0% of GitHub Issues About AI Crawlers Raise Traffic, Server Load or Cost Problems

Summary

Picture the owner of a small online craft supply store who gets an email from her host: the site went over its bandwidth limit again. Nothing changed in her sales. The extra load came from crawlers that collect pages for AI systems. Open-source maintainers face the same problem, and they write about it in public.

AI companies and many smaller players run crawlers that fetch web pages to train models or to answer questions live. Unlike search engine bots, some hit sites hard and often. When that happens to an open-source project's website, docs or code host, the maintainers open an issue to track it.

We found 551 such issues in 459 projects over the past year. 45.0% of them describe real operating pain: traffic spikes, server load, rate limits or hosting bills. And the rate of these issues roughly quadrupled from late 2025 to late 2026.

What we measured

We used GH Archive, the public log of GitHub events, in Google BigQuery. We extracted every newly opened issue from October 2025 to October 20, 2026 whose title or body mentions AI crawlers, AI scrapers, LLM bots, known AI crawler user agents or AI bot traffic. To keep automated issue streams from skewing results, we dropped repositories with more than 10 matching issues.

We then checked each issue's text for signs of harm (traffic, load, bandwidth, cost, rate limits, outages) and for fixes under discussion. Because GH Archive captured fewer events in some months, we also report the count per million issue events recorded that month. Per study rules we report AI crawlers as a group and name no bots, projects or people.

AI crawler issues are getting more common

New AI crawler issues per month, and the same count per million issue events GH Archive recorded.

Raw counts dip in June to August 2026 because GH Archive captured far fewer events then. Measured per million issue events, the rate climbed steadily: 15.2 in Q4 2025, 19.4 in Q1 2026, 27.4 in Q2 and 60.9 in Q3.

Nearly half describe real operating pain

What the 551 issues talk about. One issue can fall into several groups.

Not every issue is a complaint. Some ask how to let AI crawlers in or how to add an llms.txt file. But cost and load dominate. 15.4% mention money directly, through bandwidth overages, hosting bills or extra compute.

robots.txt is the first fix people reach for

Fixes mentioned in the issues.

132 issues (24.0%) pair a harm with at least one concrete fix. robots.txt only works for crawlers that respect it, which is why heavier tools such as rate limits, firewalls and challenge pages also come up.

What this means

If your site slows down or your hosting bill jumps for no clear reason, check your server logs for AI crawlers. You are not alone: hundreds of technical teams logged the same problem this year. A few rules in robots.txt and a sensible rate limit can fix much of it without shutting out the AI tools your customers use. Setting those rules is routine work in AI search optimization services, and it rarely takes long.

Be careful not to block everything. Some AI crawlers power the answers that send you customers. Our free AI visibility audit checks which AI crawlers reach your site, so you can keep the ones that matter for LLM optimization and limit the rest.

Limits

Issue text in GH Archive comes from public repositories only. Problems handled privately, on other platforms or by hosting providers are not counted.

We matched keywords in titles and the first 3,000 characters of issue bodies. Some matches are feature requests from teams building crawlers, and some complaints use words we did not search for.

GH Archive captured far fewer events in May to August 2026 and had gaps in October 2025 and August to October 2026. We use per-million rates to compare months, but small monthly counts are noisy.

We excluded repositories with more than 10 matching issues to remove automated issue streams, which may also drop a few heavily affected projects.

Frequently asked questions

Related studies

• The open-source tooling around AI searchGEO and AI SEO Repos on GitHub Hit 322 a Month in 2026
• The open-source tooling around AI searchAI Article Generator Repos on GitHub Grew 485% Since 2023
• The open-source tooling around AI search16.1% of New SEO Repos on GitHub Now Mention AI
• The open-source tooling around AI searchAI Browser Agent Repos on GitHub Grew 22-Fold Since 2024