Python Tools That Turn Web Pages Into Text for AI Were Downloaded 70.6 Times as Often in 2026 as in 2022

Summary

Picture a bakery owner who asks an AI assistant for the best sourdough in town. The assistant does not look at the bakery's website the way a person does. Somewhere in the pipeline, a piece of code downloads the page, strips out menus and scripts, and keeps the plain text for the model to read.

Much of that stripping is done by small open-source Python libraries with names like markdownify, trafilatura and html2text. Every time a developer, a server or a test pipeline installs one of them, the Python Package Index logs a download. Those logs are public, so they give a rough but real view of how fast this kind of web reading is growing.

The answer is very fast. In the Q3 2026 sample days, 14 of these packages were downloaded 101,997,090 times, 70.6 times as often as in the same days of 2022. The whole package index grew 9.5x in that time, so web-to-text tools took a bigger slice of all Python downloads every year.

What we measured

We queried the public PyPI download log in Google BigQuery (bigquery-public-data.pypi.file_downloads), which records one row for every file downloaded from the Python Package Index. To keep the query affordable we sampled the first seven days of every month from January 2022 through September 2026, 399 days in total, and summed downloads by month, package, country, CI flag, installer and weekday versus weekend into one working table. A quarter in this study therefore means 21 sampled days (days 1 to 7 of each of its three months), not a full quarter. We removed downloads made by mirroring and proxy tools (bandersnatch, Nexus, Artifactory, devpi), which copy packages in bulk rather than install them. To control for the growth of PyPI itself we also pulled total daily download counts for the whole index from the table's partition metadata for the same sampled days.

We grouped 14 packages whose main job is turning web pages (or documents) into plain text or Markdown that a language model can read: markdownify, markitdown, html2text, trafilatura, justext, firecrawl-py, readability-lxml, crawl4ai, firecrawl, inscriptis, newspaper3k, newspaper4k, goose3, scrapegraphai. Some of them, such as html2text and markdownify, are also used outside AI work, and markitdown converts office files and PDFs as well as HTML.

Web-to-text downloads grew 71x in four years

Each point is the total number of downloads across the 14 packages during the 21 sampled days of that quarter (days 1 to 7 of each month).

Sampled quarterly downloads stayed between 1,248,042 and 2,386,525 from Q1 2022 through Q1 2024. They started to bend upward in 2024 and steepened again in 2026. Q3 2026 alone was 19.4x Q3 2024.

The growth beat PyPI's own growth by a wide margin

Python downloads overall have exploded as more software is built and tested automatically. To check that web-to-text tools are not just riding that wave, we divided their downloads by total PyPI downloads for the same days.

The share rose 7.4x. In plain terms, for every 10,000 files downloaded from PyPI in the Q3 2026 sample days, about 9 were one of these web-to-text tools, compared with about 1 in 2022.

Markdown converters lead the group

Markdown keeps headings, lists and links while dropping layout code, which is why it has become a common format for feeding web pages to language models. The download share by package reflects that.

Five packages (markdownify, markitdown, html2text, trafilatura and justext) made up 91.2% of the group. Several newer AI-first crawlers such as crawl4ai and the Firecrawl SDKs are smaller in download terms but did not exist before 2024.

A third of downloads now run inside automated test pipelines

PyPI marks a download as CI when the installing machine reports a known continuous-integration environment. It is a floor for automation, not a full count.

The CI share climbed from 21.2% to 33.7% in two years. Growth in CI downloads points to these tools being built into production software that is tested and redeployed regularly, which is the kind of software that reads websites on a schedule. By country, United States IP addresses led with 72.4%, followed by India (4.4%) and Germany (3.1%); these splits mostly show where cloud servers sit.

What this means

If AI tools read your site through converters like these, what they keep is your text, headings, lists and links. Content that only appears after scripts run, text baked into images, and key facts hidden in tabs or sliders are the parts most likely to be dropped or garbled on the way to the model. Clear headings, real HTML text and a plain summary near the top of important pages give these tools the best chance of passing the right facts along. An AI SEO consultant can run your key pages through the same kind of converter and show you what survives.

This is a fast-growing audience you never see in your analytics. If you want to know how AI assistants describe your business after reading your pages, start with a free AI visibility audit; it is the first step in our LLM SEO work.

Limits

A download is not a user. One developer, one Docker build or one CI job can download the same package many times, and a single busy pipeline can move a package's numbers sharply from month to month.

Figures cover the first seven days of each month only (21 days per quarter), so they are sample totals, not full-month or full-quarter totals. Growth rates compare like-for-like sample periods.

PyPI as a whole grew from 12.1 billion downloads in the Q3 2022 sample days to 115.0 billion in the Q3 2026 sample days (9.5x), so part of every package's growth is the index growing. We report share of all PyPI downloads where it matters.

PyPI-wide downloads in the September 2026 sample week were 38.1% lower than in the August 2026 sample week (28.7 billion vs 46.4 billion), so September figures for individual packages should be read with that drop in mind.

The CI flag is only populated in this dataset from March 2024, and it only catches environments that announce themselves as CI. Automated downloads that do not set a CI variable are counted as non-CI.

Country codes come from the downloading IP address. Cloud servers sit mostly in the United States, so country splits describe where machines run, not where developers live.

Some packages in the group (html2text, markdownify, justext) are used for non-AI work such as email rendering or search indexing, so not every download is AI-related.

Downloads of one package can pull in others as dependencies, so group totals can count one install more than once.

Frequently asked questions

Related studies

• Downloads of the tools that read websites for AIAI Agent Framework Downloads Topped 108M in One Week
• Downloads of the tools that read websites for AIMCP Downloads Now Exceed 10 Agent Frameworks Combined
• Downloads of the tools that read websites for AIAI Search API Downloads Grew 7.4x Faster Than PyPI
• Downloads of the tools that read websites for AIAI Browser Agent Downloads Grew 54x in a Year