Google, Microsoft and OpenAI Say They Will Honor One Shared Robots.txt Line to Opt Out of AI Training
The companies announced plans Friday to read a single policy token in robots.txt as a site's training preference, instead of requiring a separate rule for each crawler. None of the three supports the line yet, and their timelines and handling differ.
Google, Microsoft and OpenAI announced plans on Friday to honor a single shared line in robots.txt that states a website’s preference on AI training, so a site could state one preference to every provider that takes part instead of writing a separate rule for each company’s crawler.
The line comes from an Internet Draft that Anthropic’s Kevin Kelley submitted to the AI Preferences working group at the Internet Engineering Task Force (IETF) on the same day, according to Search Engine Journal. The IETF’s document tracker lists it as an individual draft, which does not represent official IETF standardization activity.
How the shared line works
Kelley’s draft adds no new directive to robots.txt. A site would write an ordinary rule group under what the draft calls a policy token, __usage_ai-training, using Allow and Disallow lines. The draft’s example opts a whole site out of AI training except for a press-kit folder.
In the draft’s vocabulary, AI training means using content to modify a generative model’s learned parameters. Models that only do non-generative work, such as classification or ranking, fall outside that definition. A wildcard group (User-Agent: *) can never match the token, a file with no __usage_ai-training group signals no preference, and the token does not control crawling.
None of the three companies supports the line yet, based on the timelines they gave, Search Engine Journal reported. Google and Microsoft described different ways of handling it.
Different handling at Google, Microsoft and OpenAI
Google’s post, by John Mueller, says that where a site opts out through the shared line and does not specify Google-Extended, Google plans to read it as if the publisher had disallowed Google-Extended. That control covers future Gemini model training and grounding in Gemini Apps and on Vertex AI. Google aims to support the control in 2027, and its crawler documentation says early 2027. Google-Extended does not affect a site’s inclusion in Google Search.
Microsoft’s post, by Bing principal product manager Krishna Madhavan, adopts the shared line for training and adds a separate AI Grounding preference that is set only at the page level. It governs whether content can help inform AI-generated answers. Microsoft also lets a meta tag or HTTP header set the training preference for a single page, overriding robots.txt for that page. Grounding defaults to Allow, and a new option called DisallowExtended allows grounding in Microsoft’s own tools, such as Copilot and Bing’s AI Mode, but not in grounding services Microsoft sells to other businesses.
Microsoft will begin honoring the new options as it crawls from February 1, 2027, and will stop recognizing the older NoArchive and NoCache tags as training or grounding signals after March 1, 2027. Sites that use the old tags and want to stay out of grounding also need to record a temporary preference in Bing Webmaster Tools. That setting opens December 15 and is available only to sites still using those tags on that date. Microsoft is the only one of the companies whose announcement replaces controls sites already use. According to Google, OpenAI and Kelley’s account of Anthropic, existing controls will continue to work.
OpenAI aims to honor the signal in the next few months, while its current robots.txt policy stays in effect. Neither OpenAI’s blog post nor its crawler documentation, checked October 30, states what happens when a site’s shared rule conflicts with its existing GPTBot rule. Kelley said Anthropic has started development on support, and that rolling it out across all of Anthropic’s services will take time. As of October 30, Anthropic had no newsroom post about the signal, and its crawler help page listed only three controls: ClaudeBot, Claude-User and Claude-SearchBot.
One shared group could replace a stack of per-company training rules once each company reads it, but an opt-out through the shared line will not reach the same uses at Google and Microsoft. At Microsoft, training and grounding are separate settings. At Google, both sit under Google-Extended, and the crawler documentation describes no way to keep content out of Gemini training while still letting Gemini Apps use it for grounding. Kelley’s draft expires May 3, 2027, unless a new version replaces it.
Reporting based on coverage by Search Engine Journal.
