Google, Anthropic and others to support new AI crawling opt-out standard

What happened

Google, Microsoft, Anthropic, and OpenAI announced joint support for a new crawling opt-out standard designed to help website operators control how AI developers use their content. The standard builds on the existing “robots.txt” system, a simple text file used to tell web crawlers what parts of a site they can access. This updated protocol will specifically allow sites to block AI companies from scraping and using their content to train large language models.

Why it matters

AI models today rely heavily on massive amounts of web data for training. Until now, website owners had limited options to prevent their content from being used without consent. This new standard shifts power back to site operators by letting them explicitly opt out of AI training data pipelines. For businesses, publishers, and creators, this means better control over intellectual property and data usage. At the same time, AI developers will face constraints on the data inputs they can freely scrape, potentially raising costs and slowing the pace of model updates.

What to watch next

The rollout of this opt-out standard will be a key signal for how data rights clash with AI innovation. Watch how many sites adopt the new robots.txt directives and whether AI companies incorporate those restrictions in practice. Enforcement and compliance will shape which datasets remain accessible for training the next generation of models. Operators and founders should prepare for more granular data governance demands, while investors should track how these limits might affect AI companies’ model development pipelines.

AI Quick Briefs Editorial Desk