Opportunity assessment: high. For the first time, the four biggest AI labs have agreed on one signal that tells their crawlers to stay away. OpenAI, Google, Microsoft, and Anthropic say they’ll adopt a shared “policy token” that lets publishers opt out of AI scraping across all of them at once, according to The Verge AI. Search Engine Roundtable reported it first. This is the closest the industry has come to an off switch that works across every major lab.
🎯 Situation Report
The new signal lives in robots.txt. That’s the plain text file websites have used for decades to tell crawlers what they can and can’t touch, whether the crawler is doing search indexing, archiving, or AI training.
Kevin Kelley, a technical lead for standards at Anthropic, proposed the token through the Internet Engineering Task Force (IETF), the group behind many of the web’s core protocols. The four companies say their AI crawlers will stay off any site that carries the token.
The rollout is staggered:
- Anthropic: Already supports it.
- OpenAI: Says it will implement it in the “coming months.”
- Google: Plans to support it next year.
- Microsoft: Plans to support it next year.
Meta is the wild card. Communications head Andy Stone said the company backs the idea but didn’t say whether Meta will honor the token. “We believe creating a common signal that conveys such preferences to multiple providers at once is an important step,” Stone said. “And we look forward to working with other AI companies on the standards and their implementation across the industry.”
📍 The Old Way
Until now, publishers had to block each AI company one by one. That meant tracking a growing list of crawler names, updating robots.txt every time a new bot showed up, and hoping nothing slipped through.
That’s a lot of upkeep for a big newsroom. For a small publisher, it’s close to impossible. One shared token turns a moving target into a single line of text.
⚙️ Granular Controls
Microsoft is going further than a simple yes or no. It plans to let publishers opt out of two different things:
- AI training: Your content gets used to build or improve models.
- AI grounding: A model gets pointed at your content as a verified source while answering, which helps cut down on hallucinations.
The split matters. Some publishers may be fine with getting cited in AI answers but won’t want their archives feeding model training. Microsoft also says sites will keep showing up in Bing search results no matter what they choose. Google already offers similar controls that separate AI features and training from regular search visibility.
🧭 Why It Matters
Publishers have spent the past few years getting hammered by AI crawlers. The pushback has come from several directions:
- Really Simple Licensing (RSL): A standard that lets publishers charge crawlers for access.
- Cloudflare: Now blocks AI crawlers by default.
- Legal fights: Lawsuits and licensing deals between media companies and AI labs.
What stands out here is who’s offering the fix. The AI companies are building the opt-out themselves and running it through a neutral standards body. That gives the token more staying power than four separate corporate policies would.
There’s a strategic angle too. A clear, industry-wide way to say no makes it easier for AI companies to argue that everything else they crawl is fair game. Opting out gets simpler. Staying silent starts to look like consent.
⚠️ Gaps in the Line
Know the limits before you rely on this:
- It’s voluntary. Robots.txt has never been enforceable. It only works on crawlers that choose to obey it.
- The timeline is long. Full coverage from the big four won’t arrive until sometime next year.
- Meta hasn’t committed. Support for the idea isn’t the same as honoring the signal.
- The past is the past. Nothing here suggests the token touches content that’s already been scraped and trained on.
📋 Action Items
If you run a website, check your robots.txt now and keep an eye on the IETF draft as it develops. Hold on to your company-specific blocks until each provider confirms it supports the token. And decide what you actually want, because “no training, yes citations” is about to become a real option.
The next checkpoint is OpenAI’s rollout, followed by Google and Microsoft next year. If Meta and the smaller labs sign on, robots.txt finally becomes the consent layer publishers have been asking for. Full details are available at The Verge AI.