OpenAI, Google, Microsoft and Anthropic to adopt AI scraping opt-out standard

OpenAI, Google, Microsoft, and Anthropic will adopt a new standard that allows publishers to opt out of having their content scraped for AI training.

This shared control enables publishers to apply their specific AI training preferences across several different crawlers simultaneously.

Kevin Kelley proposed the shared "policy token" signal as part of the Internet Engineering Task Force (IETF). The signal will reside within a website’s robots.txt file. If these companies encounter the shared token, they will restrict their AI crawlers from accessing the site.

Microsoft will provide more granular controls, giving publishers the option to opt out of AI training or AI grounding specifically. Microsoft stated that websites will still appear in Bing’s search results regardless of the preferences they select. Anthropic has already adopted the signal for its systems.

Google and Microsoft plan to support the standard next year.

OpenAI intends to implement the standard in the "coming months." The Really Simple Licensing (RSL) standard gives publishers the ability to charge crawlers that visit their websites. Cloudflare is currently blocking AI crawlers by default.