In the Year After Pay-Per-Crawl Launched, 15.6% of News About AI Crawlers Mentioned Payment or Licensing, Up From 1.9% the Year Before

Summary

Picture a small travel blog owner in Portland who notices her server bills going up and her analytics full of bot visits. A forum thread says AI companies are crawling sites to train their models and power their answers, and that some hosts now let you block them or charge them. She wonders whether blocking would hurt her chances of showing up in AI answers.

We counted news articles about AI crawlers, scraping, robots.txt and bot blocking. GDELT logged 1,937 of them between January 2022 and September 2026. We compared the 12 months before July 2025, when pay-per-crawl (a way for sites to charge AI crawlers for access) was introduced, with the 12 months after.

Coverage volume rose modestly, by 19.7%. The language changed more. Before July 2025, only 1.9% of AI crawler stories mentioned payment or licensing in the headline URL. In the following year it was 15.6%, as the debate widened from whether to block AI bots to whether they should pay.

What we measured

We matched URL slugs that contain an AI term (AI, LLM, OpenAI, ChatGPT, Perplexity, Anthropic, Claude, Gemini, generative) and a crawler term (crawler, crawling, scrape, scraping, robots.txt, GPTBot, bot blocker, bot traffic, blocking AI bots). We then flagged payment or licensing words (license, pay, paid, payment, deal, compensate, charge, toll, marketplace, monetize) and blocking words (block, robots.txt, opt out, ban, stop).

A spot check of 50 matches found about 42 on topic; the rest were about an AI company whose name contains gptbots, or AI web scraping software. Counts are unique URLs, with distinct headlines reported alongside. This counts news coverage, not blocking rates on actual websites.

Payment and licensing entered the crawler story after July 2025

Coverage of AI crawlers in the 12 months before and after July 2025:

January 2026 alone had 71 articles with payment or licensing words, but only 16 distinct headlines that month, so one widely syndicated story drove most of them. Even without it, payment language appeared three times as often as before. The shift coincides with the introduction of pay-per-crawl and with lawsuits and licensing deals between AI companies and content owners.

Coverage comes in bursts

Monthly AI crawler articles:

The first big burst came in August 2023 with 104 articles, when many news sites began blocking AI crawlers in robots.txt. July 2024 (92 articles) had the most negative tone of any month at -4.87. August 2025 (106) and October 2025 (98) followed accusations of hidden crawling and lawsuits over scraping. June 2026's 158 articles came from just 31 distinct headlines.

It is a publisher story first

Coverage by year:

Publisher and website words appeared in 68.7% of 2025 articles (470 of 684), up from 29.9% in 2024. The top outlets were The Manila Times (48), WebProNews (42), MarketScreener (34), Search Engine Journal (27), GlobeNewswire (26) and The Verge (23). English sources wrote 1,683 of the 1,937 articles (86.9%). Average tone across all years was -0.30, with 24.4% of articles below -2.

What this means

For most small and mid-size businesses, AI crawlers are how your content gets into AI answers. Blocking them can protect content you sell, but it can also keep your business out of the answers customers see. The news has moved toward paid access for big publishers. For a local business, being readable is usually worth more than being paid. Open access is the starting point for SEO for AI search, since nothing else matters if the crawlers cannot reach your pages.

Check your robots.txt and any bot-blocking settings at your host or CDN to make sure you are not blocking AI search crawlers by accident. Our free AI visibility audit includes that check, and a crawler setting is one of the first things an AEO agency should look at before touching content.

Limits

This counts news coverage, not how many websites block or charge AI crawlers.

Payment and licensing words in a URL can describe proposals or complaints as well as actual payments.

Syndicated stories inflate some months, which is why we report distinct headlines.

We matched URL slugs, so coverage that mentions crawlers only in the text is missed.

Frequently asked questions

Related studies

• How the news covers AI searchLocal Business AI Search News: From 6 in 2024 to 157 in 2026
• How the news covers AI searchAds in AI Answers: News Coverage Up 841% in 2026
• How the news covers AI searchOnly 0.3% of AI Search News Mentions Small Businesses
• How the news covers AI searchGEO, AEO and AI SEO News: 183 Articles a Month in 2026