Only 89 of 772,412 Top Websites Publish a Web Bot Auth Key Directory, and 87.6% of Them Run on Cloudflare

Summary

Picture a software company that launches an AI agent that books appointments on other websites. Some of those sites block it as an unknown bot. The fix being discussed in standards groups is for the agent to sign its requests, so sites can check who it is.

Web Bot Auth is a draft standard for exactly that. A bot operator publishes its public keys at /.well-known/http-message-signatures-directory and signs each request. Websites and CDNs can then verify the signature instead of guessing from a user-agent string. The directory belongs on the bot operator's domain, so most ordinary business sites have no reason to publish one.

That makes adoption a count of who is building signed agents and crawlers. In October 2026, 89 of the top 1 million homepages published a key directory. HTTP Archive recorded none in any earlier crawl from March to September 2026.

What we measured

We used the HTTP Archive October 2026 mobile crawl (httparchive.crawl.pages) for top-1M homepages. HTTP Archive requests /.well-known/http-message-signatures-directory and records whether it found a key directory, how many keys it lists, the key types and the algorithms. We counted a site only when at least one key was present.

To show how often servers answer unknown addresses with a success code, we also used HTTP Archive's check of a deliberately made-up /.well-known/ path. Hosting and CDN come from HTTP Archive technology detection.

Signed-bot directories are rare and new

Counts by popularity rank.

No site in the top 1,000 publishes one. Most directories sit on smaller sites, which fits a picture of startups and developers building their own agents. The same field returned zero sites in each of the seven crawls from March to September 2026.

Publishers lean heavily on Cloudflare and llms.txt

How the 89 sites compare with the whole sample. Cloudflare is named as an adoption fact.

Cloudflare has promoted Web Bot Auth for verifying bots, which may explain the overlap. The llms.txt overlap suggests these are teams that follow AI tooling closely.

Most 'found' responses are empty pages

What servers send back at /.well-known/http-message-signatures-directory.

Many servers answer any address with a 200 code. That habit makes automated checks for new AI files unreliable unless the checker reads the content, which is why we only counted directories with real keys.

What this means

Most business owners do not need to publish a key directory. It matters if you run a bot or AI agent that visits other sites. What does matter for everyone is the made-up-address problem: if your server returns a success page for any URL, AI tools checking for files like llms.txt can be misled. A proper 404 for missing pages is the cleaner setup.

If you use a CDN or firewall that verifies bots, check how it treats AI search crawlers so you are not blocking the ones that could recommend you. Our free AI SEO audit includes a crawler check like that and shows whether AI assistants can find and quote your business today.

Limits

Only the homepage origin is checked; key directories on a separate subdomain are missed.

The sample is small (89 sites), so percentages for this group can move a lot between crawls.

HTTP Archive recorded zero directories with keys before October 2026; we cannot tell whether that reflects adoption or a change in the metric.

Hosting detection shows where a site is served from, not who configured the directory.

Frequently asked questions

Related studies

• AI training opt-out signalsOnly 0.9% of News Homepages Mark Content as Paywalled
• AI training opt-out signalsOnly 1 in 4,544 Top Sites Sends an AI Opt-Out Header
• AI training opt-out signalsInvented AI Meta Tags: 0.09% of Top Sites Use Them
• AI training opt-out signalsOnly 0.13% of EU Sites Declare a TDM Reservation for AI