Only 1 in 4,544 Top Websites Sends a noai or noimageai X-Robots-Tag Header With Its Homepage

Summary

Picture a photographer whose web host offers a setting called "AI protection." She switches it on and nothing on her pages looks different. If the setting works through HTTP headers, the only place to see it is in the server's response.

X-Robots-Tag is a response header that carries the same directives as a robots meta tag, such as noindex. Some sites use it to send noai or noimageai, two informal opt-out values. Like the meta tag version, these values do not appear in the crawler documentation of OpenAI, Google or Anthropic, which point to robots.txt.

Header-based AI opt-outs are rare. 170 of the top 1 million homepages send one, about 1 in 4,544. Most site owners who try a page-level AI opt-out use a meta tag instead.

What we measured

We used the HTTP Archive October 2026 mobile crawl (httparchive.crawl.pages) for top-1M homepages and read the X-Robots-Tag values that HTTP Archive extracts from the main document response (wpt_bodies.robots.x_robots_tag). A site counts when any header value contains noai or noimageai as a word. We report only these AI-specific values and leave out noindex and other search directives.

We compared header users with sites using the same values in a meta tag, and listed the hosting or CDN technologies HTTP Archive detected on header users.

Very few sites send any X-Robots-Tag, and fewer still for AI

Counts by popularity rank.

The top 1,000 has one site with an AI header. Below it, the 10K to 50K bucket has the most sites with one, 25 of 28,831. Every bucket stays below 0.2%.

Most header users skip the meta tag

Overlap between the header and meta tag methods across all top-1M homepages.

Meta tags are far more common, with 1,027 sites in total. Header-only sites are likely set at the server or CDN level, where the owner may not even see the setting in their CMS.

The values sites send

Header values grouped. Hosting names below show where these sites run, not who set the header.

Hosting and CDN technologies detected on the 170 sites: Cloudflare (89), Nginx (56), Amazon Web Services (48), Amazon S3 (22), Next.js App Router (19), Imperva (14), Apache HTTP Server (13), Vercel (11). Sites can run on several at once.

What this means

Check your response headers if you have ever switched on an "AI protection" option in your host or CDN. You can see them in your browser's developer tools on the network tab. Make sure the setting does what you expect and does not add noindex or nosnippet by accident.

For an opt-out the major AI companies document, use robots.txt rules for their training crawlers. For visibility in AI answers, leave their search crawlers open. To see whether any header or setting is holding you back, get a free AI visibility audit; we run it the way a careful GEO agency would, starting with what the assistants say about you.

Limits

HTTP Archive records headers for the homepage document only, not for images or other pages, so image-level headers are missed.

We count noai and noimageai only. Other AI-related header experiments are not included.

Hosting and CDN detection is by HTTP Archive fingerprint and does not show which layer added the header.

Single crawl, October 2026.

Frequently asked questions

Related studies

• AI training opt-out signalsOnly 0.9% of News Homepages Mark Content as Paywalled
• AI training opt-out signalsOnly 0.13% of EU Sites Declare a TDM Reservation for AI
• AI training opt-out signalsWeb Bot Auth Keys: 0.01% of Top Sites Publish Them
• AI training opt-out signalsnoai Meta Tag Stats: 0.13% of Top Sites Use It