One robots.txt line for AI training opt-outs: what OpenAI, Google, Microsoft and Anthropic agreed to

OpenAI, Google, Microsoft and Anthropic said on October 30 that they will honor a single robots.txt signal that lets a site say "don't train AI models on this" once, instead of writing a separate rule for every company's crawler. The token is __usage_ai-training, and it comes from an Internet-Draft published the same day at the IETF.

Why it matters: until now, opting out of AI training meant knowing each provider's product token (GPTBot, Google-Extended and so on) and keeping that list current. A shared token gives site owners one place to state a preference. The catch is that the providers are switching it on at different times, and it is a request, not a lock.

What the signal is

The draft, Machine-Readable Training Preference Signal, was written by Anthropic's Kevin Kelley and is dated October 30, 2026, with an expiry of May 3, 2027. It is listed under the IETF's AI Preferences (AIPREF) work. AI Weekly notes that an individual draft like this has no formal standing in the IETF standards process yet.

The mechanism reuses ordinary robots.txt groups. Instead of a crawler name, the User-Agent line carries a "policy token." The draft's own example:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

How the draft says this is read:

• Disallow on a path means a preference against AI training for that path; Allow means a preference in favor.
• If no group names the token, no preference is expressed.
• A wildcard User-Agent: * group does not count as a match for the token, so an existing blanket rule won't be read as a training opt-out.
• "AI training" follows the AIPREF vocabulary: using content to change the learned parameters of a generative model.
• The draft stresses that it "introduces no new directive names and no new grammar," so existing robots.txt parsers should cope.

How it differs from GPTBot and other crawler controls

Product tokens like GPTBot control crawling by a specific bot. The new token expresses a usage preference and, according to the draft, is not a crawling directive. If you want to stop a bot fetching pages at all, the draft says to keep using rules keyed to that crawler's product token. Providers still check their own product-token rules as before.

OpenAI's crawler documentation already separates its bots by purpose, and says each setting is independent of the others:

OpenAI also says robots.txt changes can take about 24 hours to show up in its search systems.

Google told Search Engine Roundtable's Barry Schwartz that if a site uses the shared token and doesn't explicitly name Google-Extended, Google will treat Google-Extended as disallowed. Google-Extended and other company-specific controls keep working alongside it, so a site can set one broad preference and override it per provider.

Who has signed on, and when

Microsoft is also offering a separate "grounding" preference, covering whether live content can be used to generate AI answers, distinct from training. Thurrott reports that Microsoft says choosing either won't affect eligibility for Bing search results.

AI Weekly reports OpenAI describes the token as a coordination mechanism, not a security control. Under RFC 9309, robots.txt is advisory, and a scraper that isn't participating can simply ignore it.

What site owners should do now

1. Add the shared group if you want to opt out of training: User-Agent: __usage_ai-training plus Disallow: /, with Allow: lines for anything you're happy to have used.
2. Keep your existing crawler rules. Rollouts are staggered into 2027, so don't delete GPTBot or Google-Extended lines yet. They also remain the way to block crawling itself.
3. Decide on search separately. In OpenAI's setup, you can disallow GPTBot and still allow OAI-SearchBot to stay visible in ChatGPT search.
4. Don't expect retroactive removal. AI Weekly notes that changing robots.txt doesn't pull content out of datasets or models that already exist. It's also still unclear whether opt-outs cover older corpora.

What to watch

The draft is a working document and could change. Search Engine Roundtable quotes IETF participant Nate Hake as saying significant disputes remain. Watch for OpenAI to publish a firm date, for the token to show up in each provider's crawler documentation (OpenAI's doesn't mention it yet), and for whether crawler operators beyond these four commit to it.