Cloudflare has just launched Pay Per Crawl, a new feature that lets publishers and content creators charge AI crawlers – think OpenAI, Perplexity, and others – for access to their websites. The offering, currently in private beta, introduces a scalable way for content owners to monetize AI traffic without resorting to blanket bans or one-off licensing deals (the latter of which might be difficult to secure for smaller publishers and creators).
“After hundreds of conversations with news organizations, publishers, and large-scale social media platforms,” says Cloudflare, “we want to empower content owners to monetize their content at Internet scale.” Pay Per Crawl is the product of those conversations. Oh, and the company is also blocking AI scrapers by default, which will boost interest in Pay Per Crawl.
Why This Matters:
As AI companies race to build ever-more powerful models using web content, publishers have been in a tough position. Essentially, they can allow their work to be scraped for free (not good), or they can block access entirely (not good either). Cloudflare’s Pay Per Crawl introduces a third option: monetize it, and it does so by adding structure and standards around that opportunity, which is important. With this system, AI crawlers are shown a price for each request and must confirm they’re willing to pay before gaining access, shifting control back to content owners.
If widely adopted, Pay Per Crawl could shift expectations around AI data collection and maybe even reset the economics of internet publishing (maybe, maybe, maybe). It also revives existing HTTP status code 402—“Payment Required”—a rarely used web protocol that was originally reserved for digital payment systems, but was never widely adopted.
In a nutshell, here’s how it will work: Publishers set a flat, site-wide price for crawler access. When an AI crawler requests content, it can either include payment intent and gain access or receive a 402 Payment Required response with the content price. Cloudflare handles billing, authentication (via HTTP message signatures), and settlement. Critically, publishers retain full control and can choose to allow, block, or charge crawlers on a case-by-case basis.
Experts React:
Tony Katsur, CEO of the IAB Tech Lab, weighed in on Cloudflare’s announcement in a post on X, praising the move while pushing for broader industry action:
According to Tony, the IAB is developing an open standards initiative called the LLM Content Ingest API to enable pay-per-query models, allowing publishers to monetize not just AI training access, but also the output of generative tools. Of course, this feels like a longshot, but we’ll see what happens. In theory, these standards would help media companies regain control and revenue as AI-driven and zero-click search overhauls web traffic and audience development.
Our Take:
Cloudflare has been hinting at this for a couple of months now. In May, CEO and co-founder Matthew Prince said:
“Ten years ago, for every two pages Google crawled, they sent you one visitor. Six months ago, that was up to six to one. The crawl rate hasn’t changed, but what’s changed is now 75% of queries to Google, Google answers on Google without sending you back to the original source. In the last six months, the rate has increased even further, where now it’s up to 15 to one—fifteen crawls for every one visitor. And if you look at OpenAI, which is 251 to one, Anthropic is 6,000 to one. It’s putting a lot of pressure on the media companies that are there, that are making money through either subscriptions or ads on their pages.”
Pressure, of course, equals monetization opportunity.
Random thought: Pay Per Crawl feels a lot like media-buying, doesn’t it? Publishers have inventory – in this case, content – and crawlers now have to pay to access it. In some ways, it mirrors programmatic buying: request, bid, pay, access. The buyer just happens to be an AI model, not an advertiser.
