For years, keeping your content out of AI training sets meant memorizing a different crawler name for every AI company, like a bouncer with a guest list that changed every week. On October 30, Google, Microsoft, OpenAI and Anthropic finally agreed on a single robots.txt signal for it.

Meet __usage_ai-training

The four companies said they will honor a shared robots.txt policy token, __usage_ai-training, which tells AI developers you'd prefer your content not be used to train generative models. It's defined in an IETF Internet-Draft written by an Anthropic engineer and reuses existing robots.txt syntax, so you can block training site-wide and still carve out exceptions, such as a press kit folder. One gotcha: a plain User-agent: * block does not count. You have to name the token on purpose.

Rollout timing varies. Anthropic says it has already started implementing support, OpenAI says "in the coming months," Microsoft will honor AI training preferences starting February 1, 2027, and Google says it intends to support the signal in early 2027. Meta voiced support for the idea but hasn't committed to implementing it.

A Preference, Not a Padlock

Read the fine print before you celebrate. The token covers training only, and the draft says plainly that it "is not a directive for crawling behavior." It doesn't apply retroactively, private licensing deals can override it, and there's no enforcement beyond companies keeping their word. The draft also has no formal standing at the IETF yet.

The bigger catch is that training isn't where most of the traffic fight is happening. Answer-time retrieval, where an AI pulls your page into a response, generally falls outside the training definition. Microsoft is adding a separate page-level grounding control, and Google's AI Overviews still follow Googlebot rules. Search rankings aren't affected either way, so opting out of training shouldn't hurt your SEO, but it also won't keep your content out of AI answers.

For the first time there's one standard line to add instead of a dozen vendor-specific ones. Whether to add it is now a business call for your team.

Not sure whether you want AI training, AI answers, both or neither? Talk it through with us, and we'll set up your robots.txt and CMS to match.

Source: SiliconANGLE, PPC Land