On October 30, 2026, OpenAI, Google, and Microsoft each said they will honor a shared robots.txt signal that lets website operators express preferences about foundational generative AI training to multiple providers at once. The signal uses an ordinary robots.txt group keyed by the policy token __usage_ai-training, as described in an individual IETF Internet-Draft authored by Anthropic's Kevin Kelley. Anthropic said through Kelley that it has already started implementing support. The new token does not replace existing per-crawler robots.txt controls.

Key Facts

Details
What Shared robots.txt “policy token” __usage_ai-training for generative AI training preferences
Who announced OpenAI, Google, Microsoft (company blogs); Anthropic via Kevin Kelley on the IETF AI Preferences list / draft
Date October 30, 2026
Example opt-out User-Agent: __usage_ai-training then Disallow: /
Replaces existing controls? No — OpenAI, Google, and Microsoft say company-specific crawler rules remain
OpenAI timing Aims to implement “in the coming months”
Google timing Aims to support in 2027 (Search Central); crawler docs notes cited by secondary reports say early 2027
Microsoft timing Honors new preferences as it crawls starting February 1, 2027
Anthropic timing Kelley: “already started implementing”; full support across services will take time
What it is not Not a crawl-block by itself; absence of the token means no preference expressed (draft); defaults differ by vendor for other uses

Background

Website operators today often list each AI crawler separately in robots.txt. The draft keeps robots.txt grammar unchanged and reuses path matching: a Disallow under __usage_ai-training expresses a preference against AI training for those paths; an Allow expresses a preference in favor. Wildcard User-Agent: * groups must not be read as matching the policy token.

OpenAI said the signal will not replace its existing crawler controls, which continue to apply. Google framed the work as building on Google-Extended (offered since 2023): if a site opts out with the shared token and does not explicitly set Google-Extended, Google plans to treat that like a Google-Extended disallow. Microsoft's Bing blog adds a separate page-level AI Grounding preference (Allow, Disallow, DisallowExtended) and says neither training nor grounding preferences change eligibility for traditional Bing search results. Microsoft will start honoring the new preferences on February 1, 2027, and asks sites that still rely on NoArchive / NoCache for training or grounding to migrate by March 1, 2027.

The Verge reported that Meta communications head Andy Stone called a common signal “an important step” without saying Meta will honor it.

Reactions and Impact

Publishers get one syntax aimed at many participating trainers, while keeping the option to fine-tune per provider. Rollout dates are staggered, so a robots.txt change today will not bind every major model trainer on the same schedule. Microsoft's grounding controls are product-specific and sit beside the shared training token rather than inside the IETF draft.

What We Don't Know Yet

  • When OpenAI's “coming months” implementation will go live, and how it maps to GPTBot and related bots.
  • Google's exact early-2027 cutover date beyond the 2027 aim.
  • When Anthropic's partial implementation becomes complete across ClaudeBot and related services.
  • Whether Meta, Amazon, or other large trainers will commit to honoring the same token.
  • How courts or regulators will treat a voluntary robots.txt preference that is still an individual Internet-Draft, not a finished IETF standard.

Sources