99.9% of Top Websites That Block Google-Extended Still Let Google Quote Them in Snippets and AI Overviews

Summary

Picture a publisher who adds Google-Extended to robots.txt after reading that it keeps content away from Google's AI. A month later he sees his articles summarized in an AI Overview and assumes the rule failed.

The rule did what Google says it does. Google's crawler documentation says Google-Extended does not affect inclusion in Google Search. Google's AI features documentation (developers.google.com/search/docs/appearance/ai-features) lists nosnippet, data-nosnippet, max-snippet and noindex as the controls that limit what appears from a page in Search, including AI Overviews and AI Mode. The two controls cover different things.

Our data shows how often owners use one without the other. Of 30,735 indexable homepages that restrict Google-Extended, 30,707 (99.9%) use no snippet limit. Of 736 that limit snippets, 708 leave Google-Extended open.

What we measured

We used the HTTP Archive October 2026 mobile crawl (httparchive.crawl.pages) for top-1M homepages with a robots.txt that returned 200. A site restricts Google-Extended when robots.txt has at least one Disallow line for that user agent. A site restricts snippets when its robots or googlebot meta tag or X-Robots-Tag header contains nosnippet, or max-snippet between 0 and 159.

We left out homepages with noindex, because those are hidden from Search anyway. The cross is the point of the study; standalone robots.txt rates are only context.

Blocking Google-Extended does not limit AI Overview quotes, and owners rarely add a snippet limit

Indexable homepages only.

For most of these owners the goal is probably to stay out of Gemini training while keeping search traffic, which is a reasonable choice. The risk is owners who expect Google-Extended to keep them out of AI Overviews. It does not.

Snippet blockers mostly ignore Google-Extended

Indexable homepages with nosnippet or max-snippet from 0 to 159, by rank.

These sites limit how Google quotes them in Search but still let their content be used for Gemini training and grounding outside Search. Only 28 indexable sites in the whole sample do both.

At every rank, Google-Extended blockers leave snippets open

Google-Extended blockers by rank, with the share that leave snippets open.

The rules are uneven in other ways too. 33.4% of owners who restrict OpenAI's training crawler do not restrict Google's equivalent token, which suggests many rules get added one crawler at a time.

What this means

Decide what you want before you edit robots.txt. If you want to stay out of Gemini training, Google-Extended is the tool. If you want to stay out of AI Overviews, Google's documentation points to nosnippet, data-nosnippet, max-snippet or noindex, and every one of them also changes your normal search snippets. Most businesses that pay for AI SEO services want to be in AI Overviews, not out of them, so these settings usually stay untouched.

For most small businesses the better goal is to be quoted, not hidden. Keep snippets open, keep Googlebot allowed, and make your pages easy to quote. A free AI visibility audit from our team checks those settings and shows whether AI assistants already quote you, which is the starting point for any AI Overviews SEO decision.

Limits

robots.txt data counts Disallow lines per user agent, not paths, so some 'restrictions' cover only part of a site.

Snippet settings are read from the homepage; inner pages may differ.

We cannot see intent. Many owners may want exactly the setup they have.

Google's documentation was checked on October 21, 2026 and may change.

Frequently asked questions

Related studies

• Snippet controls that also govern AI Overviews99.5% of Sites That Set max-snippet Choose No Limit
• Snippet controls that also govern AI Overviewsdata-nosnippet Appears on 8.9% of Top Homepages
• Snippet controls that also govern AI OverviewsOnly 0.16% of Top Sites Use nosnippet