Data study · The open-source tooling around AI search
4.6% of New Web Scraper Repos on GitHub Now Put AI or LLM in Their Name, More Than Five Times the 2023 Rate
Across 355,289 new scraper and crawler repos since April 2023, the share naming AI, LLMs, GPT, agents or MCP rose from 0.8% in 2023 to 4.6% in 2026. Those AI-named scrapers draw far more stars per repo than ordinary ones.
Analysis of 355,289 original scraper and crawler repositories first seen Apr 2023 to Oct 2026 · GH Archive via BigQuery, October 2026
Key findings
- 1
The share of new scraper and crawler repos with AI terms in their name rose from 0.8% in 2023 (640 of 75,763) to 4.6% in 2026 (3,021 of 66,294).
- 2
Most of the rise came in one step, from 1.9% in 2024 to 4.3% in 2025, followed by a smaller rise to 4.6% in 2026.
- 3
AI-named scrapers from 2025 averaged 2.78 recorded stars per repo, against 0.42 for other scrapers, more than six times as many.
- 4
In 2025, AI-named scrapers were 4.3% of new scraper repos but collected 22.6% of the stars those repos earned.
- 5
MCP has become a common format: 533 AI-named scrapers in 2025 (10.8%) and 358 in 2026 (11.9%) are MCP servers, against 12 in 2024.
- 6
Developers created 115,567 new scraper or crawler repos in 2025, up 18.3% from 97,665 in 2024.
Summary
Picture a roofing contractor who notices that a competitor's website copy shows up, almost word for word, inside a chatbot answer. Somewhere a scraper pulled that page and handed it to a language model. More and more of the scraping code on GitHub is built for exactly that job.
Web scrapers and crawlers are programs that fetch web pages and pull out their content. For years they fed price trackers, lead lists and research datasets. Now a growing number exist to feed AI models and agents.
We looked at every new scraper and crawler repo on GitHub since April 2023. In 2026 4.6% of them put AI, LLM, GPT, agent or MCP right in the name, more than five times the 2023 rate. Those AI scrapers also draw much more developer attention.
What we measured
We used GH Archive, the public log of GitHub events, in Google BigQuery. We kept repositories with scrap or crawl in their name and removed probable forks (repos with no forks and at most two stars that share a name with an earlier forked repo). We started in April 2023 so that older repos showing their first activity in early 2023 do not distort the baseline. A repo's first recorded event stands in for its creation date.
We flagged a repo as AI-named when its name contains AI, LLM, GPT, RAG, agent, MCP or a major model or AI company name. We compared shares and recorded stars between AI-named and other scrapers. This is a name-only test, so it undercounts scrapers that feed AI without saying so.
AI-named scrapers went from rare to routine
Share of new scraper and crawler repos with AI terms in their name, by half year.
Share (%)
| Period | New scraper repos | AI-named | Share (%) |
|---|---|---|---|
| 2023 Q2 | 27,549 | 207 | 0.75 |
| 2023 H2 | 48,214 | 433 | 0.9 |
| 2024 H1 | 48,922 | 656 | 1.34 |
| 2024 H2 | 48,743 | 1,153 | 2.37 |
| 2025 H1 | 60,270 | 2,407 | 3.99 |
| 2025 H2 | 55,297 | 2,506 | 4.53 |
| 2026 H1 | 51,734 | 2,376 | 4.59 |
| 2026 H2 (to Oct 20) | 14,560 | 645 | 4.43 |
Source: aisearchoptimization.net analysis of 355,289 original scraper and crawler repositories first seen Apr 2023 to Oct 2026 · GH Archive via BigQuery, October 2026
The share rose in every period from 2023 to 2025, from 0.9% in late 2023 to 4.53% in late 2025, then leveled off in 2026. The total number of new scraper repos also rose, from 97,665 in 2024 to 115,567 in 2025.
AI scrapers get far more attention per repo
Recorded stars and the share of repos with at least one star, by the year a repo first appeared.
| Year first seen | AI-named repos | Stars per AI-named repo | Other repos | Stars per other repo | AI-named share of stars |
|---|---|---|---|---|---|
| 2023 (Apr to Dec) | 640 | 39.4 | 75,123 | 1.59 | 17.4% |
| 2024 | 1,809 | 47.3 | 95,856 | 1.79 | 33.2% |
| 2025 | 4,913 | 2.78 | 110,654 | 0.42 | 22.6% |
| 2026 (to Oct 20) | 3,021 | 0.29 | 63,273 | 0.1 | 12.2% |
Source: aisearchoptimization.net analysis of 355,289 original scraper and crawler repositories first seen Apr 2023 to Oct 2026 · GH Archive via BigQuery, October 2026
The 2023 and 2024 groups include several of today's best known AI crawling tools, which is why their averages are so high. Even in 2025, 11.9% of AI-named scrapers earned a star against 8.7% of others. 2026 figures are low across the board because GH Archive recorded far fewer stars that year.
New AI scrapers increasingly ship as MCP servers
MCP servers let AI assistants call a scraper directly as a tool.
Repos
| Year | AI-named scraper repos | MCP servers | MCP share (%) |
|---|---|---|---|
| 2023 (Apr to Dec) | 640 | 6 | 0.9 |
| 2024 | 1,809 | 12 | 0.7 |
| 2025 | 4,913 | 533 | 10.8 |
| 2026 (to Oct 20) | 3,021 | 358 | 11.9 |
Source: aisearchoptimization.net analysis of 355,289 original scraper and crawler repositories first seen Apr 2023 to Oct 2026 · GH Archive via BigQuery, October 2026
The format barely existed before 2025. Now more than one in nine new AI scrapers plugs straight into an AI assistant, so the assistant can fetch any page it needs while answering a question.
What this means
More of the bots reading your site are there to feed AI systems, and many of them run on the fly while someone asks an assistant a question. You cannot see these visitors in a normal analytics report, but they shape what AI says about your business.
Decide which AI bots you want to welcome and make your key facts easy for them to extract: clear headings, plain text and consistent business details. Which pages do AI assistants rely on when they describe your business? Our free AI visibility audit answers that, and it is the groundwork for any LLM SEO plan.
Limits
We matched repo names only. Many scrapers that feed AI models do not say so in their name, so the true share is higher than 4.6%. A few name matches may be unrelated.
GH Archive stopped including descriptions for newly created repos after October 8, 2025, so a description-based count was not possible across the full period. Programming language is not in the event data.
GH Archive recorded far fewer star events in 2026 and had collection gaps in October 2025 and August to October 2026. Star figures for recent repos are understated.
First seen is the first recorded public event. Fork removal is a name-based heuristic.
Embed this research
Add the charts from this study to your page. The embed links back here.
<iframe src="https://aisearchoptimization.net/embed/data-studies/ai-web-scraper-repos-share" title="AI Scraper Repos: 4.6% of New Scrapers Name AI, Up 5x" width="100%" height="720" style="border:0" loading="lazy"></iframe>
Cite this study
APA
Grenyo, K. (2026). AI Scraper Repos: 4.6% of New Scrapers Name AI, Up 5x. AI Search Optimization. https://aisearchoptimization.net/resources/data-studies/ai-web-scraper-repos-share
Plain link
AI Scraper Repos: 4.6% of New Scrapers Name AI, Up 5x - AI Search Optimization - https://aisearchoptimization.net/resources/data-studies/ai-web-scraper-repos-share
Frequently asked questions
How many new web scraper repos mention AI?
In 2026 through October 20, 3,021 of 66,294 new scraper and crawler repos (4.6%) had AI, LLM, GPT, agent or MCP in their name. In 2023 the share was 0.8%.
Are AI scrapers more popular than ordinary scrapers?
Yes. AI-named scrapers first seen in 2025 averaged 2.78 recorded stars per repo, against 0.42 for other scrapers. They were 4.3% of new scraper repos but drew 22.6% of the stars.
How many new scraper repos appear each year?
We counted 97,665 in 2024 and 115,567 in 2025, an 18.3% rise. 2026 had 66,294 through October 20, with gaps in the event record late in the year.
What does this mean for my website?
A growing share of the bots that fetch your pages exist to feed AI systems. In 2026, 11.9% of new AI-named scrapers were MCP servers that let assistants read pages while answering a question.
Related studies
- The open-source tooling around AI searchGEO and AI SEO Repos on GitHub Hit 322 a Month in 2026
- The open-source tooling around AI searchSEO Data MCP Servers on GitHub More Than Doubled in 2026
- The open-source tooling around AI searchOne List Holds 81% of AI Bot Blocklist Stars on GitHub
- The open-source tooling around AI searchAI Browser Agent Repos on GitHub Grew 22-Fold Since 2024
See where AI search leaves you out
Run the free AI visibility audit. If you want the fixes done for you, the plan is a flat $299/month.