Dedicated Robots.txt Parsers Were Downloaded About Once for Every 10 Downloads of Python Scraping and Browser Automation Tools in 2026, Up From About 1 in 24 in 2022

Summary

Picture a dentist who adds a line to robots.txt asking AI crawlers to skip the patient forms folder. Whether that request is honored depends on the crawler's code actually reading the file and following it.

In Python, the language most scrapers are written in, a crawler can read robots.txt with the built-in standard library, with a feature inside its framework, or with a dedicated parser package. We counted downloads of the four dedicated parser packages on PyPI and compared them with downloads of popular scraping, extraction and browser automation tools.

In the Q3 2026 sample days, dedicated parsers were downloaded 15,348,592 times, about one for every ten scraper and browser automation downloads (10.0%). In 2022 the figure was closer to one in 24. Parser downloads are rising faster than scraper downloads, but they still trail scraper downloads by a wide margin.

What we measured

We queried the public PyPI download log in Google BigQuery (bigquery-public-data.pypi.file_downloads), which records one row for every file downloaded from the Python Package Index. To keep the query affordable we sampled the first seven days of every month from January 2022 through September 2026, 399 days in total, and summed downloads by month, package, country, CI flag, installer and weekday versus weekend into one working table. A quarter in this study therefore means 21 sampled days (days 1 to 7 of each of its three months), not a full quarter. We removed downloads made by mirroring and proxy tools (bandersnatch, Nexus, Artifactory, devpi), which copy packages in bulk rather than install them. To control for the growth of PyPI itself we also pulled total daily download counts for the whole index from the table's partition metadata for the same sampled days.

We compared four dedicated robots.txt parser packages (protego, robotspy, robotexclusionrulesparser, reppy) with nine scraping and extraction libraries and seven browser automation packages. Browser automation tools such as Selenium and Playwright are used heavily for software testing as well as scraping, so we show the ratio both with and without them.

Parsers are gaining ground on scrapers

Each row covers the 21 sampled days of the third quarter of the year.

The ratio stayed between 4.1% and 6.3% from 2022 to 2024, then rose to 7.5% in 2025 and 10.0% in 2026.

One package does nearly all the work

Download share by parser package, Q3 2026 sample days.

Protego ships as a dependency of Scrapy, so part of its count comes from Scrapy installs. Scrapy itself had 2,816,406 downloads in the period, far fewer than Protego's 15,061,708, which means most Protego downloads came from other projects or direct installs.

Parser downloads look increasingly automated

Installer and timing patterns give clues about who is downloading.

In Q3 2026 the daily download pace on weekends was 96.9% of the weekday pace, close to flat, compared with 65.5% for PyPI as a whole. A pattern that barely dips on weekends fits automated builds better than people at work. Only 11.1% were flagged as CI, so most of these automated-looking downloads do not announce themselves as test runs.

What this means

Robots.txt asks crawlers to stay out of certain areas; it cannot force them. Python crawlers can check it with the standard library, with a framework feature or with a parser package, and our data only sees the last of these, so it cannot say how many crawlers ignore the file. What it does show is that dedicated parser downloads still trail scraper downloads by a wide margin. If something on your site must stay out of AI tools, protect it with a login or remove it, and use robots.txt for preferences such as which AI crawlers you welcome.

Review your robots.txt rules for AI crawlers at least once a year, since new crawler names appear often. Every other part of LLM SEO depends on which AI crawlers your site allows or blocks, and a free AI visibility audit includes that check.

Limits

A download is not a user. One developer, one Docker build or one CI job can download the same package many times, and a single busy pipeline can move a package's numbers sharply from month to month.

Figures cover the first seven days of each month only (21 days per quarter), so they are sample totals, not full-month or full-quarter totals. Growth rates compare like-for-like sample periods.

PyPI as a whole grew from 12.1 billion downloads in the Q3 2022 sample days to 115.0 billion in the Q3 2026 sample days (9.5x), so part of every package's growth is the index growing. We report share of all PyPI downloads where it matters.

PyPI-wide downloads in the September 2026 sample week were 38.1% lower than in the August 2026 sample week (28.7 billion vs 46.4 billion), so September figures for individual packages should be read with that drop in mind.

The CI flag is only populated in this dataset from March 2024, and it only catches environments that announce themselves as CI. Automated downloads that do not set a CI variable are counted as non-CI.

Country codes come from the downloading IP address. Cloud servers sit mostly in the United States, so country splits describe where machines run, not where developers live.

Python's standard library includes a robots.txt parser (urllib.robotparser), and many frameworks have their own, so download counts of dedicated packages say nothing direct about how many crawlers obey robots.txt.

Browser automation tools are widely used for software testing, which inflates the scraper side of the ratio; we show the ratio without them as well.

Protego's count includes installs pulled in by Scrapy and other projects that depend on it.

Frequently asked questions

Related studies

• Downloads of the tools that read websites for AIWeb-to-Text Tool Downloads Grew 71x Since 2022
• Downloads of the tools that read websites for AIAI Agent Framework Downloads Topped 108M in One Week
• Downloads of the tools that read websites for AIMCP Downloads Now Exceed 10 Agent Frameworks Combined
• Downloads of the tools that read websites for AIAI Search API Downloads Grew 7.4x Faster Than PyPI