Cloudflare, which has quickly established itself as a key player in the AI scraping wars, acting as a White Knight for content owners, today announced a new capability called Content Signals Policy. The tool lets site owners set clearer preferences on how bots scrape and use on-site content for AI. It does so by making it easier to update their robots.txt file, including opting out of AI overviews and inference altogether.
Robots.txt files are already used to specify which crawlers are allowed to access a website and which sections they can reach. However, what crawlers can do with the content after accessing it has always been fuzzier, especially in the new AI era. With Content Signals Policy, site owners can now set more explicit instructions on whether bots are allowed to access content at all, define how that content can be used once accessed (search, AI input, AI training), and even include a reminder that preferences in robots.txt can be cited as a legal signal of intent.
Why This Matters:
Cloudflare is one of the foremost internet infrastructure providers for websites and content owners, and giving them a simple, free tool to manage how AI scrapers use their content could have wide-ranging effects. As The Information notes, this may even impact Google, which recently integrated AI Mode into the Chrome address bar, as well as challengers like Perplexity. With open web traffic already declining, Cloudflare is positioning itself as a key decision-maker in how AI access to content is governed.
They’re also, in our view, helping to create standards in a way the adtech industry has struggled to do—at least in a form that sticks or is enforceable. Cloudflare is effectively creating a standard by making it easier for publishers to control how their content is used by AI bots and crawlers. Flipping that switch carries real weight.
Experts React:
Quora, Reddit, and the News/Media Alliance were all quoted in the launch press release. Here’s what Reddit CTO Chris Slow had to say:
“For the web to remain a place for authentic human interaction, platforms that empower communities must be sustainable. We support initiatives that advocate for clear signals protecting against the abuse and misuse of content.”
(Reddit, obviously, has an interesting relationship with AI search and Google specifically.)
Our Take:
Cloudflare’s move underscores how infrastructure providers—not just DSPs, SSPs, or measurement vendors—are beginning to shape the rules of engagement in an AI-driven advertising ecosystem. But this isn’t purely altruistic. Cloudflare is also making itself more indispensable to clients by positioning as the intermediary that helps websites control and monetize AI access to their content.
With this approach, Cloudflare can both protect creators and enable publishers to charge AI companies for crawling content—taking a cut by acting as the merchant of record for pay-per-crawl transactions. Hey, the way things are going, it’s a smart business!
