Hacker News Stories About Blocking AI Crawlers Average 51% More Points and 70% More Comments Than Other Stories

Summary

Picture the owner of a small online shop who checks their hosting bill and finds traffic has tripled, but sales have not moved. The extra visitors are bots collecting pages for AI models. Plenty of site owners have had that moment, and many of them talk about it on Hacker News.

We found 348 stories since January 2023 whose titles are about stopping AI crawlers and scrapers, from robots.txt rules to tools that trap or slow bots down. We compared each one with every other story posted on the site in the same month.

On average these stories earned 51.3% more points and 70.5% more comments. The topic peaked in 2025, when self-hosters and small sites began writing up their fights with AI bot traffic.

What we measured

We used the public Hacker News table on Google BigQuery and kept live stories posted from January 2023 to September 2026. A story counted if its title combined an AI or crawler term, such as AI crawler, AI scraper, LLM bot, GPTBot or robots.txt for AI, or named a tool built to stop them, with a blocking or defense term such as block, trap, poison, ban, opt out, protect or DDoS. We removed titles that used scrap in other senses.

For each story, the baseline is the average for all live stories posted in the same calendar month. Points and comment counts are the values recorded at the October 2026 refresh. We grouped linked domains into broad categories and name only widely known platforms.

Crawler-blocking stories outscore the average

Most Hacker News stories get a handful of points and vanish. Stories about AI crawlers did better than that on every measure we checked.

The median story scored 4 points, against a typical monthly median of about 2.6 for all stories. The gap is larger in the averages because a few crawler stories drew hundreds of points and comments.

The most-discussed threads covered reports of undeclared AI crawlers ignoring no-crawl rules, open-source tarpits that trap AI scrapers, and default AI-bot blocking rolled out by a large content delivery network.

2025 was the boiling point

The number of these stories rose from 55 in 2024 to 146 in 2025. That year many small sites, code forges and open-source projects described AI bot traffic overwhelming their servers, and proof-of-work tools for stopping bots took off.

In 2026 the count stayed high at 98 stories in nine months, but average engagement dropped back below the site average. That suggests the topic has become routine on the site rather than a surprise.

Independent site owners lead the conversation

We sorted the linked sites into broad groups to see who is writing about the problem.

Half the stories came from independent blogs, small company sites and other sites rather than news media or large platforms. Among widely known platforms, GitHub hosted 37 stories (688 comments) and Cloudflare's blog 9 stories (931 comments). Ars Technica led news outlets with 18 stories.

What this means

People who run websites care a lot about AI crawlers, and the loudest voices are small site owners dealing with server load. Blocking every AI bot can protect your bandwidth, but it can also keep your business out of the AI answers your customers read. The decision deserves a deliberate choice, not a default. An AI SEO consultant can help you sort the bots that bring visibility from the ones that only take bandwidth.

A sensible middle path is to allow the crawlers that power AI answers and search, limit training-only bots if you prefer, and watch your logs. Checking your robots.txt for AI crawlers is part of every free AI visibility audit we run, and it should come before any spending on generative engine optimization services.

Limits

Stories were matched on title keywords. Some relevant stories use other wording, and a few matched titles may be only partly about crawler blocking.

Points and comments are values at the October 2026 refresh. Hacker News ranking also depends on timing and moderation, which the dataset does not record.

Averages are pulled up by a few very popular stories; medians show a smaller gap.

This measures discussion, not how many sites actually block AI crawlers.

2026 covers January to September only.

Frequently asked questions

Related studies

• Developer conversation on Hacker Newsllms.txt Splits Developers: 54% For, 46% Against
• Developer conversation on Hacker NewsAI SEO Draws 3x the Positive Developer Comments of SEO
• Developer conversation on Hacker News'Written by AI' Comments on Hacker News Up 4.6x Since 2023
• Developer conversation on Hacker News'Slop' Now Shows Up in 1 of Every 149 Hacker News Comments