I’ve always wondered when the free lunch for AI companies scraping the web would end. Well, it looks like that day is here.
Cloudflare just dropped a bombshell, and it’s a game-changer for anyone who creates content online. They’re now automatically blocking AI crawlers by default on the new websites it protects, completely flipping the script on decades of open web policy.
But they didn’t just build a wall; they built a tollbooth.
They’ve launched a new marketplace called Pay per Crawl, and it’s exactly what it sounds like. Publishers finally get some control back.
Here’s how it works:
-
You Set the Price
You can now charge AI companies micropayments to access your content.
-
Control the Use
Decide whether bots pay for data to train their models, power search results, or something else.
-
Fair Value
The data is wild. OpenAI’s crawler scrapes a site 1,700 times for every single referral it sends back. This new system helps balance those scales.
This is a massive move. Cloudflare is positioning itself as a key gatekeeper for the data that will power the next wave of AI. It could force a healthier relationship between AI and publishers, but it also raises a big question: will we see an internet divided between premium, paid-for data and free sites that become the default AI fodder? It will be super interesting to watch this unfold.