The biggest names in AI are backing a new way to opt out of scraping

What happened

OpenAI, Google, Microsoft, and Anthropic are backing a new technical standard that allows publishers to opt out of having their content scraped for AI training. This standard uses a shared “policy token” signal, proposed by Anthropic’s Kevin Kelley through the Internet Engineering Task Force (IETF). Publishers can embed this token on their websites to communicate their AI training preferences in a unified way, instead of negotiating terms with each AI company individually.

Why it matters

This agreement shifts some control over web data usage back to publishers and content owners, who have long worried about their material being scraped without permission. Centralizing the opt-out process simplifies compliance for both publishers and AI companies, reducing friction and enforcing clearer boundaries on training data collection. This could slow down or limit the growth of AI model training pools drawn indiscriminately from the web, leading to more respect for content ownership and licensing. It also raises the bar for AI providers to respect explicit content boundaries, which potentially lowers legal and reputational risks around using scraped materials.

What to watch next

The key thing to monitor is adoption and enforcement. Will other AI players follow these big names, and will publishers actually implement the policy token widely? Also, watch if this standard prompts publishers to revisit their web licensing models, or if it leads to clearer industry norms on ethical data sourcing. Finally, regulators could take note if self-regulation starts altering web scraping practices, potentially influencing future data use laws affecting AI training.

AI Quick Briefs Editorial Desk