Data study · llms.txt and AI instruction files
49.3% of Websites Whose /llms.txt URL Loads Send Back an HTML Page, Not a Text File
219,833 top-1M homepages answered /llms.txt with a 200 response. 108,384 of those (49.3%) were HTML pages, and only 53,495 (24.3%) were clean plain-text files.
Analysis of 219,833 sites whose /llms.txt returned HTTP 200 · HTTP Archive, October 2026 crawl
Key findings
- 1
Of 219,833 sites where /llms.txt returned a 200 response, 108,384 (49.3%) served an HTML page instead of a text file.
- 2
52,751 files (24.0%) were sent as text/markdown, a content type that strict checkers, including HTTP Archive's own validator, flag as invalid.
- 3
Only 53,495 responses (24.3%) were plain-text files that passed HTTP Archive's checks with no warnings.
- 4
Another 3,942 plain-text files (6.9% of plain-text files) had markdown problems such as unmatched brackets or uneven code fences.
- 5
Among the top 1,000 sites, 80 of 139 that returned 200 at /llms.txt (57.6%) sent HTML.
Summary
Picture a marketing manager who types her site address followed by /llms.txt into a browser and sees the page load. Job done, she thinks. Then a developer points out that what loaded was the site's normal homepage layout, not a file at all.
llms.txt is meant to be a small text or markdown file that AI tools can read in one pass. When a server answers that address with a web page instead, an AI tool gets HTML, menus and scripts, or a polite "page not found" dressed up as a success.
That mix-up is common. Of 219,833 popular sites where /llms.txt returned a success code, 108,384 (49.3%) sent back an HTML page. Only about a quarter served a clean plain-text file.
What we measured
We used the HTTP Archive October 2026 mobile crawl (httparchive.crawl.pages), limited to homepages in the Chrome UX Report top 1 million. HTTP Archive requests /llms.txt on each site and records the status code, the content type and basic markdown checks in its llms_txt_validation metric.
We kept the 219,833 sites that answered with HTTP 200 and grouped them by what came back: plain text that passed, plain text with markdown warnings, text/markdown, HTML, or another content type. We did not read the file bodies, so HTML responses are classed by content type alone.
Half of the 'working' llms.txt URLs are web pages
Every row below returned a 200 status, so a quick browser check would make each one look fine.
% of 200 responses
| Response type | Sites | % of 200 responses |
|---|---|---|
| HTML page | 108,384 | 49.3 |
| Plain text, passed checks | 53,495 | 24.3 |
| Sent as text/markdown | 52,751 | 24 |
| Plain text with markdown warnings | 3,942 | 1.8 |
| Other content types (JSON, images, blank) | 1,261 | 0.6 |
Source: aisearchoptimization.net analysis of 219,833 sites whose /llms.txt returned HTTP 200 · HTTP Archive, October 2026 crawl
An HTML answer usually means the server has a catch-all rule that returns a normal page, or a custom error page, for any address it does not recognize. AI tools asking for llms.txt on those sites get page markup rather than a list of links.
The HTML problem shows up at every size of site
We split the same responses by popularity rank.
| Rank bucket | 200 responses | HTML | % HTML | Text file (plain or markdown) | % text file |
|---|---|---|---|---|---|
| Top 1K | 139 | 80 | 57.6 | 58 | 41.7 |
| 1K-5K | 639 | 372 | 58.2 | 263 | 41.2 |
| 5K-10K | 827 | 481 | 58.2 | 341 | 41.2 |
| 10K-50K | 7,163 | 4,242 | 59.2 | 2,869 | 40.1 |
| 50K-100K | 9,677 | 5,714 | 59 | 3,877 | 40.1 |
| 100K-500K | 86,683 | 45,229 | 52.2 | 40,908 | 47.2 |
| 500K-1M | 114,705 | 52,266 | 45.6 | 61,872 | 53.9 |
Source: aisearchoptimization.net analysis of 219,833 sites whose /llms.txt returned HTTP 200 · HTTP Archive, October 2026 crawl
From the top 1,000 down to rank 100,000, between 57.6% and 59.2% of these responses are HTML. Only the two bottom buckets drop lower, to 52.2% and 45.6%, because automatic files from store platforms raise the share of real text files there.
Most sites still return nothing at all
For context, here is every outcome across the full sample, including sites that did not return 200.
| Outcome | Sites | % of all homepages |
|---|---|---|
| 404 or 410 (no file) | 515,300 | 66.7 |
| Text file (plain or markdown) | 110,188 | 14.3 |
| HTML page with 200 | 108,384 | 14 |
| Other error status (401, 403, 429, 5xx and others) | 21,934 | 2.8 |
| Request failed or timed out | 15,345 | 2 |
| Other content type with 200 | 1,261 | 0.2 |
Source: aisearchoptimization.net analysis of 219,833 sites whose /llms.txt returned HTTP 200 · HTTP Archive, October 2026 crawl
A clean 404 is a better answer than an HTML page with a 200 code. It tells AI tools there is no file, rather than handing them a page that looks like one. 21,934 sites (2.8%) blocked or errored on the request, sometimes because a firewall treats unknown bots with suspicion.
What this means
Check your own file the way a machine would. Request yoursite.com/llms.txt and check the response headers as well as the page. You want a 200 status and a text/plain or text/markdown content type. If you see text/html, your server is probably returning a normal page, and you need a real file at that path or a 404. It is the kind of five-minute check that AI SEO services should include without a separate fee.
If you publish the file in markdown, keep the syntax clean: close every bracket, match your code fences, and keep the list of links short. Our free AI SEO audit checks your llms.txt headers along with the bigger question of whether AI assistants can find and quote your business.
Limits
Classification uses the content type the server reported. We did not read response bodies, so a few real llms.txt files served with an HTML content type are counted as HTML.
text/markdown is a reasonable content type for llms.txt. We list it separately because HTTP Archive's validator only accepts text/plain, not because AI tools reject it.
Only the homepage origin's /llms.txt was requested, once, from HTTP Archive's crawler; some servers answer bots differently from browsers.
Results describe one crawl in October 2026.
Embed this research
Add the charts from this study to your page. The embed links back here.
<iframe src="https://aisearchoptimization.net/embed/data-studies/llms-txt-returns-web-page-instead-of-text" title="Half of llms.txt URLs That Load Return a Web Page" width="100%" height="720" style="border:0" loading="lazy"></iframe>
Cite this study
APA
Grenyo, K. (2026). Half of llms.txt URLs That Load Return a Web Page. AI Search Optimization. https://aisearchoptimization.net/resources/data-studies/llms-txt-returns-web-page-instead-of-text
Plain link
Half of llms.txt URLs That Load Return a Web Page - AI Search Optimization - https://aisearchoptimization.net/resources/data-studies/llms-txt-returns-web-page-instead-of-text
Frequently asked questions
Why does my llms.txt show my homepage?
Your server likely sends a normal page for any unknown address. In our data 108,384 of 219,833 sites (49.3%) that answered /llms.txt with 200 sent HTML instead of a text file.
What content type should llms.txt use?
text/plain is the safest choice. 52,751 sites (24.0% of 200 responses) used text/markdown, which some validators flag even though the file itself may be fine.
How many llms.txt files have markdown errors?
3,942 of 57,437 plain-text files (6.9%) triggered markdown warnings in HTTP Archive's checks, such as unmatched brackets or uneven code fences.
Is a 404 at /llms.txt bad?
No. 66.7% of popular homepages return 404 or 410 there. A clear 404 is more honest to AI tools than an HTML page returned with a success code.
Related studies
- llms.txt and AI instruction filesllms.txt Adoption 2026: 14.3% of Top Sites Have One
- llms.txt and AI instruction filesllms.txt Grew From 1.1% to 14.3% of Top Sites in 15 Months
- llms.txt and AI instruction filesllms.txt Now Outnumbers security.txt Nearly 5 to 1
- llms.txt and AI instruction files46% of llms.txt Files Are Added Automatically by Platforms
See where AI search leaves you out
Run the free AI visibility audit. If you want the fixes done for you, the plan is a flat $299/month.