Only 0.9% of News Site Homepages Mark Any Content as Paywalled in Their Structured Data

Summary

Picture a regional news site that sells subscriptions. An AI assistant answers a reader's question with a summary of a story that sits behind the paywall, and the editor wonders how it got there and how to signal what is paid.

Schema.org gives publishers a way to say which content is paywalled: the isAccessibleForFree property, set to false, usually with a hasPart block that points to the paid section. Google documents this markup for paywalled content so it is not mistaken for cloaking. AI systems that read structured data can use the same flag.

We checked news homepages in the top 1 million. Only 77 of 8,821 (0.9%) mark anything as paywalled. Far more news sites reach for robots.txt rules against AI crawlers.

What we measured

We used the HTTP Archive October 2026 mobile crawl (httparchive.crawl.pages) for top-1M homepages. We flagged a homepage as a news site when its structured data included the NewsArticle or NewsMediaOrganization type, then searched the structured data for isAccessibleForFree and its value.

For context we checked robots.txt rules for 20 AI crawler user agents on sites whose robots.txt returned 200. This is markup data, not data about what AI tools actually cite.

Paywall markup is almost absent from news homepages

How news homepages use the isAccessibleForFree property.

Homepages often carry markup for the publisher rather than for each article, so the property may live on article pages instead. Even so, only 570 news homepages mention the property at all.

News sites restrict AI crawlers nearly four times as often

Sites with a readable robots.txt only.

Many publishers appear to rely on blunt controls rather than precise ones. Blocking AI search crawlers can protect paid stories, but it also removes free stories from AI answers that could send readers back.

Larger news sites use the markup a little more

News homepages by popularity rank.

The differences are small and the counts in the top buckets are low, so treat the rank pattern as a hint.

What this means

If you publish paid content, mark it properly on article pages: isAccessibleForFree set to false and a hasPart block pointing at the paywalled section. That tells search engines and any AI system reading structured data what is paid, without hiding your free work. It is a small markup job that an AI search optimization agency or your developer can finish in an afternoon.

Think twice before blocking AI search crawlers across the whole site. Restricting training crawlers and marking paid sections lets your free stories keep earning citations. A free AI SEO audit from our team checks your paywall markup and whether AI assistants can find and quote your free stories.

Limits

We checked homepages only. Many publishers add paywall markup to article pages, which this crawl does not cover, so true article-level use is likely higher.

News sites are identified by structured data types and miss news sites without that markup.

Our search for isAccessibleForFree uses text matching inside the recorded structured data and may miss unusual formats.

robots.txt counts are Disallow lines per user agent, not full-site blocks.

No growth trend: we measured one crawl to keep query costs down.

Frequently asked questions

Related studies

• AI training opt-out signalsInvented AI Meta Tags: 0.09% of Top Sites Use Them
• AI training opt-out signalsWeb Bot Auth Keys: 0.01% of Top Sites Publish Them
• AI training opt-out signalsOnly 1 in 4,544 Top Sites Sends an AI Opt-Out Header
• AI training opt-out signalsOnly 0.13% of EU Sites Declare a TDM Reservation for AI