17.7% of Sites That Publish Their Own llms.txt Also Restrict AI Crawlers in robots.txt, Nearly Twice the Rate of Sites Without One

Summary

Picture an online course creator who adds an llms.txt file so AI assistants can find her lessons. A year earlier, worried about AI training, her developer added GPTBot and ClaudeBot to robots.txt with a Disallow rule. Now her site invites AI tools in with one file and turns them away with another.

llms.txt is a welcome mat for AI tools. robots.txt is the door policy that crawlers check first. When the two disagree, the robots.txt rule usually wins, because well-behaved crawlers read it before anything else.

We compared both files on 630,161 popular sites. Sites that added their own llms.txt are about 1.8 times as likely to have a Disallow rule for an AI crawler as sites without llms.txt: 17.7% versus 9.7%.

What we measured

We used the HTTP Archive October 2026 mobile crawl (httparchive.crawl.pages) for top-1M homepages and kept the 630,161 sites whose robots.txt returned 200. HTTP Archive parses robots.txt and counts Allow and Disallow lines per user agent. We flagged a site when any of 20 AI crawler user agents had at least one Disallow line.

We split llms.txt sites in two. 'Platform default' means the site runs on one of seven platforms where 95% or more of homepages serve the file. 'Self-added' means every other site with a 200 text response at /llms.txt.

Self-added llms.txt sites restrict AI crawlers almost twice as often

Each bar is the share of sites with a Disallow rule for at least one AI crawler.

Site owners who think about AI enough to write llms.txt seem to be the same owners who add AI rules to robots.txt. That overlap is where mixed signals come from.

GPTBot and ClaudeBot top the list

Crawler names are user-agent tokens as written in robots.txt.

Even the search-side crawlers are affected. 7.1% of self-added llms.txt sites restrict OAI-SearchBot and 8.4% restrict PerplexityBot. Those bots fetch pages to cite in live answers, so restricting them works against the point of llms.txt.

The gap is widest among mid-tier and long-tail sites

Same comparison, split by popularity rank.

In the top 10,000 the two groups are close, and the 1K to 5K bucket even reverses. Below that the gap widens as rank falls, reaching about twice the rate below rank 100,000.

What this means

Read your robots.txt and llms.txt side by side. If you want AI assistants to recommend you, do not disallow the crawlers that fetch pages for answers, such as OAI-SearchBot, ChatGPT-User, PerplexityBot and Claude-SearchBot. If you only object to model training, restrict the training crawlers and leave the search ones open. If the crawler names are unfamiliar, an AI SEO consultant can sort them for you in a few minutes.

A Disallow line may only cover one folder, so check what each rule actually blocks. This is a short fix and one of the more common mixed signals we see. Our free AI visibility audit reads both files and tells you whether AI assistants can still find and quote your business.

Limits

HTTP Archive records how many Disallow lines each user agent has, not the paths. A Disallow rule may block one folder rather than the whole site, so 'restrict' does not always mean a full block.

We checked 20 AI user agents; rules written only under '*' are not counted as AI-specific.

Platform-default vs self-added is inferred from platform adoption rates, not from file contents.

The comparison shows a pattern, not a cause. We cannot tell which file was added first.

Frequently asked questions

Related studies

• llms.txt and AI instruction files46% of llms.txt Files Are Added Automatically by Platforms
• llms.txt and AI instruction files28% of Top Online Stores Now Publish an AI Checkout File
• llms.txt and AI instruction filesllms.txt Grew From 1.1% to 14.3% of Top Sites in 15 Months
• llms.txt and AI instruction filesllms.txt Adoption 2026: 14.3% of Top Sites Have One