Bots now make up more than half of all internet traffic, and Cloudflare CEO Matthew Prince thinks that changes how the web makes money. In a long interview on The Verge AI’s Decoder podcast, Prince said Cloudflare is no longer just “a service that makes the internet faster and protects it from bad guys.” He said the company now wants to help shape “what that future business model of the internet looks like.”
That’s a big claim from the company that sits between millions of websites and the AI crawlers hitting them.
🌐 What’s actually changing
For about 30 years, advertising has paid for the web. Prince calls Google “the hero” of that first generation. Search sent people to websites. Tools like DoubleClick and Google Analytics turned those visits into ad revenue. Publishers got traffic and Google got data, so both sides had a reason to keep going.
AI breaks that deal. Scrapers pull content to train models. Agents read pages on a person’s behalf and never see an ad. The Verge AI reports that Cloudflare’s own data showed bots passed 50% of traffic in June, and the share keeps rising as AI companies scrape more and agents do more tasks for users.
Prince said his team once estimated that non-human traffic would overtake human traffic in the second half of 2027. The June numbers suggest the timeline is moving faster than that forecast. A forecast from the company that runs the gate is still worth taking seriously, but it’s worth remembering that Cloudflare also sells the tools to manage that traffic.
🔒 Cloudflare’s play: the toll booth
Cloudflare runs in more than 350 cities with thousands of data centers. That gives it a view of a huge share of global traffic and a practical way to act on it. Website owners using Cloudflare can:
- Block AI crawlers and agents entirely
- Allow them freely
- Charge them, letting in only the bots willing to pay for access
The third option matters most. Cloudflare began blocking AI crawlers by default and launched pay-per-crawl experiments in 2025, and Prince is now framing that as the start of a new economic layer for the web. If it works, publishers get paid directly by AI companies and don’t depend on clicks that aren’t coming back.
🤔 The open question: is this good?
Decoder host Nilay Patel asked Prince directly whether the outcomes he describes are actually good. Patel says Prince’s answer stayed with him long after the conversation ended, and he meant that as a mixed signal.
The tension is real. A pay-to-crawl web could bring back money for the people creating content. It could also leave one infrastructure company with a lot of say over who reads what on the internet, and at what price. Small AI startups could end up priced out while the biggest labs sign bulk deals. Open access, which built the early web, could shrink.
🏢 AI is reshaping Cloudflare too
The shift isn’t only outside the company. According to The Verge AI, Cloudflare laid off more than 1,000 people earlier this year, about 20% of its workforce. Prince then wrote a Wall Street Journal op-ed headlined “How I choose which Cloudflare employees to replace with AI.”
So the company pitching itself as the web’s defense against AI disruption is also openly replacing people with it. That fits a wider pattern: infrastructure companies are some of AI’s loudest critics and also some of its fastest adopters.
📋 What practitioners should do now
- Publishers: Look at your bot traffic logs today. Know which crawlers hit you and how often before you decide to block, allow or charge.
- AI builders: Plan for paid data access as a normal cost. Free scraping is getting harder, and licensing budgets will become standard.
- Businesses running agents: Expect more friction. Agents that identify themselves clearly and pay when asked will get access that anonymous scrapers won’t.
- Everyone: Watch whether payment standards come from open protocols or from a few gatekeepers. That decision will shape the next decade of the web.
🔮 Looking ahead
Over the next one to three years, expect the web to split into tiers: open content, licensed content and content locked behind bot paywalls. Cloudflare is betting it can be the referee. Whether publishers, AI labs and regulators accept that role is the question to watch. The full conversation is on The Verge AI’s Decoder, and it’s worth hearing in full.