Cloudflare Pushes Back on AI 'Strip mining'

Cloudflare announced yesterday that it set its network to block AI crawlers by default, but its bigger ambition is a new feature called pay per crawl (all lower-case), letting content providers charge AI companies for crawling their content.

The feature is part of an even bigger, longer-term hope: to rethink the way the industry values web content in the first place, getting away from the model where revenue comes from raw clicks. That sounds like a stretch goal, but it's encouraging. These are the kinds of audacious moves that Cloudflare is good at, and it supports the company's self-proclaimed mission of building a better internet.

It all stems from Cloudflare's observation that AI-based search is draining content sites of traffic. This was already a trend with search engines; people seek an answer in Google, see it among the search responses, and then walk away without clicking on anything. That deprives ad-dependent sites of revenue. It also means sites that rely on subscription models don't get to show off a free taste to potential readers.

'Stripmined'

AI intensified the problem, particularly in the last six months, based on dramatic numbers that Cloudflare CEO Matthew Prince shared recently. He summarized it in a blog this week: "With OpenAI, it's 750 times more difficult to get traffic than it was with the Google of old. With Anthropic, it's 30,000 times more difficult."

Further down, Prince puts it more bluntly:

"Instead of being a fair trade, the web is being stripmined by AI crawlers with content creators seeing almost no traffic and therefore almost no value."

Cloudflare can quantify the problem and do something about it because so much internet traffic goes through its global network. The same is true of rivals like Akamai and Fastly. Websites depend on these companies for services like DDOS protection and bot management, and all three offer CDN services that deliver web pages like this one.

Pay per crawl is executed as a network function. It's built on HTTP response codes such as 402 (payment required) or 403 (simply "forbidden," with no content returned). Of course, Cloudflare adds elements of control. Publishers can set conditions on certain bots, for example, keeping open the possibility to strike customized deals with particular AI companies.

The Deeper Problem

This is an effort worth cheering. Yes, we at Futuriom are content creators and therefore have a stake in the game. But there's also a societal issue here: As much as we all love using GenAI, there's no escaping the fact that it's making money off the work of others. AI has become the ultimate middleman, taking a cut of nearly 100%.

This lays bare a deeper problem: We shouldn't be valuing web content by page views in the first place. That thinking led to shameless, clickbaity headlines, and it let copycat aggregation become a substitute for journalism. (I've been guilty of both over the years, especially during my journalism days, and I'm not proud of it.)

Cloudflare wants to tackle the concept of content value too, although it's a trickier problem that won't be solved by technology alone. Still, Prince has an idea: a marketplace where web content is valued on how well it "furthers knowledge" — specifically, how well it fills the gaps in AI models. History is stacked against the "golden age of high-value content creation" that he's picturing, but it's a worthy effort.