Cloudflare just drew a line in the sand against Google, and it could reshape how the open web feeds AI. According to Hacker News, the content delivery network that sits in front of roughly one-fifth of the world’s websites has given Google a hard deadline: starting September 15, every new site signing up for Cloudflare, plus all customers on its free tier, will have their bot management default set to block “multi-purpose crawlers” on any page that runs ads.
Here’s what that phrase actually means. Some crawlers do two jobs at once. They index your site for search AND scrape it to train AI models. Google runs exactly that kind of dual-purpose bot. Cloudflare’s new default turns those crawlers away at the door unless the site owner explicitly opts back in. Apple and Bing get caught in the net too, but the unnamed primary target is Google.
Why this is a big deal
For 20 years, publishers optimized their entire existence around ranking higher in Google Search. The relationship was simple: Google sent traffic, publishers gave up content. That deal is breaking down.
“We’ve been clear about what we want,” said Cloudflare chief strategy officer Stephanie Cohen, as quoted by Hacker News. “We want a technical solution that allows you to be discoverable without having to give your content away for free.”
What stands out here is the leverage. One config change, flipped to “block by default” across millions of free-tier sites, is a coordinated action that no single publisher could pull off alone. Cloudflare is doing the collective bargaining for them.
The players lining up
This isn’t just Cloudflare acting alone. Per Hacker News, several heavyweights are moving in the same direction:
- USA Today Inc. is prepared to delist from Google within six to twelve months, according to CEO Mike Reed. The company already has licensing deals with Meta, Microsoft, and Amazon. Its stance: “For those with licensing agreements, they get our content. For those without, we block them.”
- Beehiiv, the creator network, partnered with Cloudflare so its creators can now block the Google crawler directly.
- Executives at “every major media company” reportedly have a model ready for what blocking Googlebot would look like.
The common thread: Google, unlike OpenAI, Anthropic, Microsoft, Amazon, and Meta, has not signed licensing deals with publishers. The hyperscalers and pure-play AI firms are paying. Google is using one crawler to both index and train, and refusing to negotiate.
The context Google won’t mention
Google did introduce Google-Extended, which nominally lets publishers opt out of AI training while staying in Search. But publishers don’t trust it. Executives at two media companies told Adweek they fear the opt-out quietly penalizes search visibility. That distrust is the whole ballgame. If opting out of AI training costs you traffic, it isn’t a real choice.
There’s also money on the table. The article notes Google pays Reddit millions for content access, and one blogger is owed $3,000 from a class action settlement with Anthropic. Scale that across the entire body of professional publishing and the unpaid bill is enormous.
What to watch next
A few things to track if you build, publish, or run AI products:
- September 15. The default flips. Watch whether Google blinks and offers a real technical separation between search indexing and AI training crawlers.
- The delisting dominoes. If USA Today actually pulls out of Google Search and traffic holds, expect others to follow fast. That anonymous “every major media company has a model” line is the tell.
- The search alternatives. The piece flags Kagi as a better site-specific search engine that doesn’t retain histories. Fragmentation of search itself is now in play.
- Your own content. If you rely on Cloudflare’s free tier, the default is changing under you. Check your bot settings before mid-September so you’re deciding, not defaulting.
The old bargain was discoverability in exchange for content. Cloudflare is betting it can force a new one: discoverability without the free giveaway. Whether Google negotiates or digs in will tell us who really holds the leverage in the AI content wars. Full details are available at the original source.