Big Four AI Labs Back Shared Training Opt-Out Standard

30-Second Summary

Google, Microsoft, OpenAI, and Anthropic have committed to a shared robots.txt signal that lets any website opt out of having its content used for AI training. The standard, drafted by Anthropic and submitted to the IETF on 30 October 2026, uses a new User-Agent token called __usage_ai-training. Anthropic has already started honouring it. Microsoft plans full implementation by 1 February 2027, and Google and OpenAI have each given themselves until 2027 without specifying a date. This is the first time all four major AI labs have agreed on a single content consent mechanism.

At a Glance

• Topic: AI Strategy
• Companies: Anthropic, Google, Microsoft, OpenAI
• Date: 30 October 2026
• Announcement: Shared robots.txt AI training opt-out standard
• What Changed: For the first time, four major AI labs have committed to a single opt-out signal for training data. Previously each company had its own mechanism, if any.
• Why It Matters: Every business with a website now has a single, standardised way to tell AI companies not to use their content for training, and the signal will have teeth once all four companies implement it.
• Who Should Care: Any business that publishes content online, including case studies, thought leadership, product documentation, and pricing information.

Key Facts

• The opt-out signal is User-agent: __usage_ai-training with Disallow: / in a website's robots.txt file.
• Anthropic's Kevin Kelley, an Anthropic standards lead, authored the Internet-Draft submitted to the IETF on 30 October 2026.
• The draft builds on the IETF AI Preferences working group, which has been developing a formal standard.
• Anthropic says it has already started implementing the signal. OpenAI says it will comply "in the coming months." Microsoft targets 1 February 2027. Google targets early 2027.
• The signal covers training data collection only. It does not affect AI answer engines like Google AI Overviews, Perplexity, or ChatGPT search when those systems fetch a page to answer a query.
• Google's existing Google-Extended token and Microsoft's separate "AI Grounding" setting remain distinct mechanisms for controlling AI answer fetching.

What Happened

On 30 October 2026, representatives from Anthropic, Google, Microsoft, and OpenAI each announced their intention to honour a new shared robots.txt signal that website owners can use to opt out of AI training data collection. The announcement came as Anthropic published an Internet-Draft through the IETF proposing __usage_ai-training as the canonical User-Agent for AI training crawlers, and called on all major AI labs to adopt it.

The draft sits within the IETF AI Preferences working group, an ongoing standards effort that has been developing a formal mechanism for content owners to specify their preferences for AI use. The working group has not yet adopted a final standard, so what the four companies announced is a commitment to the draft rather than a ratified specification.

Anthropic stands apart from the others by saying it has already begun implementing the signal in its training pipelines. Google, Microsoft, and OpenAI have each provided dates ranging from February to some point in 2027, which means the practical effect of this announcement is still months away for most companies.

The scope of the standard is narrower than much of the coverage suggests. Setting __usage_ai-training tells AI companies not to include your pages in future training datasets. It does not instruct their systems to stop fetching your content to generate live answers. A user asking Claude, ChatGPT, or Perplexity a question can still receive a response that draws on or cites your pages, even after you add the signal.

Why It Matters

Content rights are being defined right now. The absence of a standard has meant AI companies have generally trained on publicly accessible web content unless a site took active steps to block their crawlers. This standard, even in draft form, shifts the default expectation. Once all four companies implement it, opt-out becomes a genuine choice rather than a technical workaround.

It changes the calculation for thought leadership content. Professional services firms, consultancies, and agencies have built content programmes on the assumption that publishing builds visibility. This standard forces a more deliberate question: is your goal to influence AI training (which shapes the quality of answers these models give) or to be cited in AI answers (which puts your content in front of buyers)? These are now separable objectives.

Implementation timelines matter for enterprise procurement. If you are negotiating a content licensing arrangement, or if your legal team is reviewing data use agreements with AI vendors, the February 2027 date is now a meaningful reference point. Microsoft is the only company to have named a specific date. That date is likely to become a de facto deadline for the others.

This is a governance moment, not just a technical one. Adding a line to robots.txt is a five-minute task. Deciding what to add requires a conversation between your marketing, legal, and technology teams about what your content is worth and to whom. Organisations that have that conversation now are in a better position than those that treat it as a default setting.

The AI answer visibility trade-off is real. There is evidence from search engine optimisation research that content used in training datasets tends to be surfaced more often in AI-generated answers. If that correlation holds, opting out of training may reduce how often AI systems recommend your firm unprompted. Measure before you decide.

Regulatory alignment is likely. The EU AI Act and emerging Australian AI governance frameworks both grapple with training data consent. This voluntary standard is likely to become the reference point for regulatory compliance in this area, which means early adopters will have a structural advantage when those frameworks are enforced.

The David and Goliath View

This is a significant moment, but the gap between announcement and implementation is wide enough to be suspicious. Anthropic is the only company that has actually done anything. Google, Microsoft, and OpenAI have committed to a timeline that puts real compliance between three and twelve months away, with no enforcement mechanism if they miss it. The cynical reading is that this announcement lets all four companies claim they support content rights while changing nothing before the next EU AI Act review cycle.

That said, the fact that all four companies have publicly named the same mechanism is genuinely new. The history of web standards suggests that once major platforms commit to a shared signal, it tends to stick, even when the initial implementations are slow. The SEO parallel is instructive: when Google and Bing jointly committed to structured data standards in the early 2010s, the web moved relatively quickly to adopt them, not because anyone enforced it but because the incentive structure became clear.

The most important thing for a business operator is not to treat this as a set-and-forget decision. The standard is still a draft, the implementation dates are soft, and the trade-offs between training data exclusion and AI answer visibility are not yet well understood. Set a review date for February 2027, when Microsoft's implementation is due, and revisit your position then with actual data on how your content is performing in AI answer engines.

Where This Fits in the AI Stack

This standard sits at the data ingestion layer of the AI stack. Most operators think about AI at the application layer, which is where tools like Claude, Copilot, and Gemini are deployed for daily work. Training data governance is three layers below that, and its effects play out over months and years rather than days. The more relevant context for an operator is that this is the beginning of a longer regulatory and commercial negotiation between AI companies and content owners, not the end of it.

Questions Operators Are Asking

Will this stop AI companies from using my content in their answers? No. The __usage_ai-training signal applies only to training datasets. AI systems that fetch pages to answer live queries, including Google AI Overviews, Claude web search, and Perplexity, are governed by separate mechanisms. You would need to use Google-Extended, ClaudeBot disallow rules, or equivalent signals to restrict live answer fetching.

Can I set it now, before all companies implement it? Yes. Anthropic says it has already started honouring the signal. Adding it now will have immediate effect for Anthropic's training crawlers and will be in place when Google, Microsoft, and OpenAI implement their versions.

Does this affect my SEO? No, this does not affect search engine crawlers. Search crawlers use different User-Agent strings and are not covered by this standard. Your pages will continue to be indexed by Google and Bing as normal.

What if I only want to protect some pages, not my whole site? The IETF draft allows path-level exclusions, the same way standard robots.txt works. You could restrict training data access to specific directories while leaving others available. This is particularly relevant if you have a mix of proprietary methodology content and general marketing pages.

Does this mean AI companies were previously doing something wrong? No. Training on publicly accessible web content has generally been legal in most jurisdictions, subject to ongoing litigation in several countries. This standard is voluntary and prospective. It does not establish that past training without consent was impermissible.

Citable Summary

On 30 October 2026, Google, Microsoft, OpenAI, and Anthropic committed to a shared robots.txt signal, User-agent: __usage_ai-training, allowing website owners to opt out of AI training data collection. Drafted by Anthropic's Kevin Kelley and submitted to the IETF, the signal is already being honoured by Anthropic, with Microsoft targeting 1 February 2027 and Google and OpenAI targeting 2027. The standard covers training data only, not live AI answer fetching. Enterprise operators should review their content strategy before setting the signal, as there is evidence that training data inclusion correlates with AI answer visibility.