Data study · Developer conversation on Hacker News

Hacker News Comments Mentioning Robots.txt Doubled to 23.4 per 100,000 After GPTBot Launched in August 2023

KGWritten by Kriszta GrenyoPublished October 20264 min read

From January 2015 to July 2023, robots.txt came up in 11.5 of every 100,000 Hacker News comments. Since August 2023 it is 23.4, and 39.1% of those comments mention AI.

Analysis of 5,415 Hacker News comments mentioning robots.txt, January 2015 to September 2026, out of 35.6 million live comments · Hacker News public dataset on Google BigQuery, October 2026

Key findings

1
Robots.txt came up in 23.4 of every 100,000 Hacker News comments from August 2023 to September 2026 (2,627 comments), double the 11.5 per 100,000 from January 2015 to July 2023 (2,788).
2
2025 was the peak year at 33.5 per 100,000 (1,140 comments), 3.8 times the 2022 rate of 8.9.
3
August 2025 was the single busiest month: 233 robots.txt comments, or 78.4 per 100,000.
4
39.1% of robots.txt comments since August 2023 mentioned AI (1,026), against 5.3% before (147).
5
29.8% of robots.txt comments since August 2023 said crawlers ignore it, bypass it or do not respect it (783), up from 15.8% before (440).
6
In a sample of 100 AI-related robots.txt comments, 39 said AI crawlers ignore it or that it is not enough, 10 said it works or is respected, 11 debated whether AI agents acting for a user should follow it and 40 took no position.

Summary

For most of its 30-year life, robots.txt was a dull file. A web developer set it once, told search engines what to skip, and forgot about it. Then AI companies began crawling the web to train and power their models, and site owners went back to the file to try to keep them out.

We counted every Hacker News comment that mentions robots.txt since 2015. Before August 2023, when OpenAI published its GPTBot crawler and how to block it, the file came up in about 11.5 of every 100,000 comments. Since then it has been 23.4.

The tone changed too. Before, robots.txt was mostly a technical detail. Now it is a fight: 39.1% of mentions involve AI, and almost 3 in 10 say crawlers simply ignore it.

What we measured

We used the public Hacker News table on Google BigQuery and counted live comments containing robots.txt from January 2015 to September 2026. Rates divide by all live comments in the same month or year. We flagged comments that also mention AI, and comments that say crawlers ignore, bypass or do not respect the file.

To gauge opinion, our team read a random sample of 100 robots.txt comments from August 2023 onward that also mention AI, and coded whether each said robots.txt is ignored or insufficient, works or is respected, or debated whether AI tools acting for a single user should follow it. With 100 comments, each share has a margin of error of up to about 10 points.

Robots.txt talk doubled after GPTBot

2.0xrobots.txt comments per 100,000 after August 2023 (23.4) versus before (11.5)

Robots.txt had steady background interest for years, rising briefly whenever a crawler or archive made news. AI crawlers turned that background into a steady debate.

Hacker News comments mentioning robots.txt per 100,000 comments

Per 100,000 comments

Year	Per 100,000	Robots.txt comments	All live comments
2,015	14.6	233	1,600,372
2,016	9.9	205	2,060,892
2,017	15.3	355	2,317,106
2,018	10.4	244	2,342,297
2,019	12.5	338	2,700,467
2,020	8.2	260	3,184,862
2,021	8.7	322	3,691,995
2,022	8.9	356	3,983,506
2,023	17.5	722	4,128,014
2,024	20.3	660	3,254,657
2,025	33.5	1,140	3,399,696
2026 (Jan to Sep)	19.9	580	2,911,855

Source: aisearchoptimization.net analysis of 5,415 Hacker News comments mentioning robots.txt, January 2015 to September 2026, out of 35.6 million live comments · Hacker News public dataset on Google BigQuery, October 2026

The rate sat between 8.2 and 15.3 per 100,000 from 2015 to 2022. It reached 17.5 in 2023, 20.3 in 2024 and 33.5 in 2025, then eased to 19.9 in 2026. The busiest months were August 2025 (78.4 per 100,000), March 2025 (50.1) and January 2025 (48.8), months that coincided with widely shared reports about AI crawler behavior.

AI is now part of 4 in 10 robots.txt conversations

39.1%of robots.txt comments since August 2023 mention AI (1,026 of 2,627)

Before 2023 almost no robots.txt comment mentioned AI. Now it is close to the main topic.

Robots.txt comments mentioning AI, and saying crawlers ignore it

Year	Robots.txt comments	Mention AI	Say crawlers ignore or do not respect it
2,015	233	0.0% (0)	12.4% (29)
2,016	205	0.0% (0)	10.7% (22)
2,017	355	0.0% (0)	16.6% (59)
2,018	244	0.8% (2)	12.7% (31)
2,019	338	0.3% (1)	14.5% (49)
2,020	260	1.5% (4)	15.0% (39)
2,021	322	1.9% (6)	17.7% (57)
2,022	356	2.8% (10)	15.7% (56)
2,023	722	26.0% (188)	20.9% (151)
2,024	660	34.2% (226)	27.3% (180)
2,025	1,140	43.1% (491)	33.4% (381)
2026 (Jan to Sep)	580	42.2% (245)	29.1% (169)

The share mentioning AI rose from 2.8% in 2022 to 26.0% in 2023 and 43.1% in 2025. Complaints that crawlers ignore the file roughly doubled as a share over the same period, from 15.7% in 2022 to 33.4% in 2025.

Most developers with a view doubt it stops AI crawlers

39 of 100sampled AI-related robots.txt comments said AI crawlers ignore it or that it is not enough

We read 100 random robots.txt comments from August 2023 onward that also mention AI.

View of robots.txt in AI-related comments (sample of 100)

AI crawlers ignore it or it is not enough

It works or is respected

Debate: should AI agents acting for a user follow it?

No clear position

Comments

View	Comments
AI crawlers ignore it or it is not enough	39
It works or is respected	10
Debate: should AI agents acting for a user follow it?	11
No clear position	40

Skeptics described server logs full of AI crawler traffic and blocked paths being fetched anyway, and many said they moved to firewall rules or proof-of-work checks instead. The smaller group said well-known crawlers do honor the file and that blocking specific user agents worked for them. A newer argument is whether an AI assistant fetching one page for one user counts as a robot at all.

What this means

Robots.txt is still the first and cheapest tool for telling AI crawlers what they may read, and the large, well-known crawlers say they follow it. But site owners on Hacker News report that plenty of bots do not, so treat it as a request, not a lock. If you block AI crawlers, also decide on purpose which ones you want to allow, because blocking the crawlers behind AI answers can keep your business out of them.

Robots.txt rules are among the first things a free AI visibility audit reviews, since answer engine optimization services cannot help pages the bots never reach.

Limits

This is discussion volume on Hacker News, not data about robots.txt files or crawler behavior.

The ignore flag and AI flag are keyword-based; the ignore flag catches some comments that discuss ignoring without claiming it happens.

The opinion sample has 100 comments, so each share has a margin of error of up to about 10 points, and coding involves judgment.

The rise coincides with GPTBot's launch and later AI crawler news; the data does not show those events caused it.

Dead and deleted comments carry no text in the dataset. 2026 covers January to September.

Embed this research

Add the charts from this study to your page. The embed links back here.

<iframe src="https://aisearchoptimization.net/embed/data-studies/robots-txt-discussion-since-gptbot" title="Developers Talk About Robots.txt 2x as Often Since GPTBot" width="100%" height="720" style="border:0" loading="lazy"></iframe>

Cite this study

APA

Grenyo, K. (2026). Developers Talk About Robots.txt 2x as Often Since GPTBot. AI Search Optimization. https://aisearchoptimization.net/resources/data-studies/robots-txt-discussion-since-gptbot

Plain link

Developers Talk About Robots.txt 2x as Often Since GPTBot - AI Search Optimization - https://aisearchoptimization.net/resources/data-studies/robots-txt-discussion-since-gptbot

Download the data (CSV)

Frequently asked questions

Did talk about robots.txt increase after GPTBot launched?

Yes. It rose from 11.5 comments per 100,000 between January 2015 and July 2023 to 23.4 per 100,000 from August 2023 to September 2026.

How much of the robots.txt conversation is about AI now?

Since August 2023, 39.1% of robots.txt comments mention AI (1,026 of 2,627), against 5.3% before.

Do developers think robots.txt stops AI crawlers?

Mostly not. In a sample of 100 AI-related robots.txt comments, 39 said AI crawlers ignore it or it is not enough, while 10 said it works or is respected.

When was robots.txt discussed most?

In 2025, at 33.5 comments per 100,000. The busiest single month was August 2025, with 233 comments or 78.4 per 100,000.

See where AI search leaves you out

Run the free AI visibility audit. If you want the fixes done for you, the plan is a flat $299/month.

Run the free audit See the $299/mo plan