Google, OpenAI back a shared website AI-training opt-out

OpenAI, Google, Microsoft and Anthropic backed a shared signal on October 30 that lets website operators express one AI-training preference across participating providers. The voluntary commitments would give publishers a common robots.txt rule alongside company-specific controls. Rollout is staggered, so the announcements establish different implementation schedules for the same proposed mechanism.

OpenAI targets the coming months; Google’s crawler documentation specifies early 2027. Microsoft will begin honoring preferences as it crawls pages from February 1, 2027. Anthropic’s Kevin Kelley said implementation had started, with more time needed to extend support across services and handle edge cases.

The proposed policy identifier is __usage_ai-training. An Internet-Draft puts it in the robots.txt field normally used to name a crawler, with existing Allow and Disallow directives. Its example withholds training permission across a site while allowing a press-kit directory. Existing path matching and longest-match precedence carry over. The draft also excludes wildcard groups from being interpreted as this training preference.

Training has a specific meaning here: using content to change a generative model’s learned parameters. The IETF working-group vocabulary referenced by the proposal treats training, AI use and search as distinct categories. It describes communicating preferences, while leaving enforcement mechanisms outside its scope.

Training, search and answer use have separate controls

OpenAI’s current crawler guide already separates those purposes. GPTBot collects material that may be used for foundation-model training; OAI-SearchBot surfaces websites in ChatGPT search. A publisher can allow search while disallowing training. OpenAI says that when both uses are allowed, results from one crawl may serve both purposes to avoid duplicate collection. These are existing controls.

Google plans to interpret the shared training opt-out as a Google-Extended disallow when that company-specific control is unspecified. Explicit provider settings continue alongside the common signal. Google says sites specifically allowing Google-Extended remain eligible to supply grounding material in Gemini Apps and receive clicks from those responses.

Google’s crawler reference describes Google-Extended as governing future Gemini training and grounding in Gemini Apps and Vertex AI. It has no separate HTTP client identifier: existing Google crawlers perform the collection. The reference also states that Google-Extended affects neither inclusion in Google Search nor its ranking signals, separating this use control from ordinary search eligibility.

Microsoft’s page-level AI Grounding preference is separate from training. DisallowExtended permits its own services while excluding third-party grounding. Ordinary Bing search eligibility remains.

Microsoft’s NoArchive and NoCache lose their training/grounding meaning after March 1, 2027. Temporary transition settings cover sites using those tags on December 15, 2026. Continued grounding restrictions require page settings plus a temporary Bing Webmaster Tools preference, lasting until recrawl or September 30, 2027.

The proposal is an individual Internet-Draft submitted by Anthropic’s Kevin Kelley on October 30, with a current expiry of May 3, 2027. The IETF tracker identifies it as unendorsed work in progress. Kelley presented it as input to the working group’s attachment discussions and said Anthropic’s implementation would complement existing crawler controls.