# One robots.txt line for AI training opt-outs: what OpenAI, Google, Microsoft and Anthropic agreed to

> Four AI providers say they will honor a shared robots.txt token, __usage_ai-training, that states a training preference once for all of them. It is not live everywhere yet, so here is what it does, what it doesn't, and what to put in your robots.txt today.

Published Nov 2, 2026 by TechMeasured Staff · AI · Canonical: https://techmeasured.com/ai/robots-txt-shared-ai-training-opt-out-guide

OpenAI, Google, Microsoft and Anthropic said on October 30 that they will honor a single robots.txt signal that lets a site say "don't train AI models on this" once, instead of writing a separate rule for every company's crawler. The token is `__usage_ai-training`, and it comes from an Internet-Draft published the same day at the IETF.

Why it matters: until now, opting out of AI training meant knowing each provider's product token (GPTBot, Google-Extended and so on) and keeping that list current. A shared token gives site owners one place to state a preference. The catch is that the providers are switching it on at different times, and it is a request, not a lock.

## What the signal is

The draft, *Machine-Readable Training Preference Signal*, was written by Anthropic's Kevin Kelley and is dated October 30, 2026, with an expiry of May 3, 2027. It is listed under the IETF's AI Preferences (AIPREF) work. AI Weekly notes that an individual draft like this has no formal standing in the IETF standards process yet.

The mechanism reuses ordinary robots.txt groups. Instead of a crawler name, the `User-Agent` line carries a "policy token." The draft's own example:

```
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
```

How the draft says this is read:

- **Disallow** on a path means a preference *against* AI training for that path; **Allow** means a preference *in favor*.
- If no group names the token, **no preference is expressed**.
- A wildcard `User-Agent: *` group does **not** count as a match for the token, so an existing blanket rule won't be read as a training opt-out.
- "AI training" follows the AIPREF vocabulary: using content to change the learned parameters of a generative model.
- The draft stresses that it "introduces no new directive names and no new grammar," so existing robots.txt parsers should cope.

## How it differs from GPTBot and other crawler controls

Product tokens like GPTBot control **crawling** by a specific bot. The new token expresses a **usage** preference and, according to the draft, is not a crawling directive. If you want to stop a bot fetching pages at all, the draft says to keep using rules keyed to that crawler's product token. Providers still check their own product-token rules as before.

OpenAI's crawler documentation already separates its bots by purpose, and says each setting is independent of the others:

| OpenAI agent | What OpenAI says it does |
|---|---|
| GPTBot | Crawls content that may be used to train its foundation models |
| OAI-SearchBot | Surfaces sites in ChatGPT search results |
| ChatGPT-User | Fetches pages when a user asks; OpenAI says robots.txt rules may not apply |

OpenAI also says robots.txt changes can take about 24 hours to show up in its search systems.

Google told Search Engine Roundtable's Barry Schwartz that if a site uses the shared token and doesn't explicitly name Google-Extended, Google will treat Google-Extended as disallowed. Google-Extended and other company-specific controls keep working alongside it, so a site can set one broad preference and override it per provider.

## Who has signed on, and when

| Provider | Stated timing (as reported) |
|---|---|
| Anthropic | Already implementing, per AI Weekly and Thurrott |
| OpenAI | "In the coming months," per AI Weekly; Thurrott reports 2027 |
| Google | Early 2027, according to Search Engine Roundtable |
| Microsoft | February 1, 2027, according to AI Weekly |

Microsoft is also offering a separate "grounding" preference, covering whether live content can be used to generate AI answers, distinct from training. Thurrott reports that Microsoft says choosing either won't affect eligibility for Bing search results.

AI Weekly reports OpenAI describes the token as a coordination mechanism, not a security control. Under RFC 9309, robots.txt is advisory, and a scraper that isn't participating can simply ignore it.

## What site owners should do now

1. **Add the shared group** if you want to opt out of training: `User-Agent: __usage_ai-training` plus `Disallow: /`, with `Allow:` lines for anything you're happy to have used.
2. **Keep your existing crawler rules.** Rollouts are staggered into 2027, so don't delete GPTBot or Google-Extended lines yet. They also remain the way to block crawling itself.
3. **Decide on search separately.** In OpenAI's setup, you can disallow GPTBot and still allow OAI-SearchBot to stay visible in ChatGPT search.
4. **Don't expect retroactive removal.** AI Weekly notes that changing robots.txt doesn't pull content out of datasets or models that already exist. It's also still unclear whether opt-outs cover older corpora.

## What to watch

The draft is a working document and could change. Search Engine Roundtable quotes IETF participant Nate Hake as saying significant disputes remain. Watch for OpenAI to publish a firm date, for the token to show up in each provider's crawler documentation (OpenAI's doesn't mention it yet), and for whether crawler operators beyond these four commit to it.


## Sources

1. [IETF — Internet-Draft: Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00)](https://www.ietf.org/archive/id/draft-kelley-aipref-training-preference-signal-00.html)
2. [OpenAI — Overview of OpenAI crawlers](https://developers.openai.com/docs/bots)
3. [Search Engine Roundtable — Google to adopt IETF AI training control](https://seroundtable.com/google-ai-training-control-42430.html)
4. [AI Weekly — OpenAI, Google, Microsoft, Anthropic back one AI opt-out token](https://aiweekly.co/alerts/openai-google-microsoft-anthropic-back-one-ai-opt-out-token)
5. [Thurrott.com — Anthropic, Google, Microsoft, and OpenAI agree to let sites opt out of AI training](https://www.thurrott.com/a-i/343407/anthropic-google-microsoft-and-openai-agree-to-let-sites-opt-out-of-ai-training)