OpenAI, Google, Microsoft and Anthropic agree to shared website opt-out for AI training

Website owners have long faced a messy chore when trying to keep their content out of artificial intelligence models. To stop developers from scraping articles, guides and products, administrators had to identify every crawler individually, adding separate blocking directives for GPTBot, ClaudeBot, Google-Extended and others.

On 30 October 2026, OpenAI, Google, Microsoft and Anthropic announced they will honour a single shared instruction. The proposal uses an ordinary robots.txt rule directed at a specific token, written as User-Agent: __usage_ai-training followed by Disallow: /. When placed on a site, all four companies agree to treat it as an instruction not to use the material on those pages to train future AI systems.

The technical design was authored by Anthropic standards lead Kevin Kelley and submitted through the Internet Engineering Task Force (IETF). While the proposal remains an early draft without formal standards endorsement, the four companies decided to move ahead with voluntary adoption.

Rollout dates and technical limits

The four developers are introducing the change on different timelines:

• Anthropic has already started supporting the rule.
• OpenAI states it will activate support in the coming months.
• Microsoft plans to start honouring the directive on 1 February 2027.
• Google plans to introduce support in early 2027.

The rule comes with important boundaries. It addresses model training rather than basic web fetching. Automated crawlers may still download a page to index it for traditional web search or show live search snippets. The rule also has no retrospective effect. Any material gathered before the directive is added will remain in older training sets. Other crawlers outside these four companies will also continue to ignore the token until they choose to adopt it.

What this means for Australian businesses

For small and medium business owners running websites in Australia, this announcement simplifies website maintenance, but it requires sensible handling over the coming months.

First, do not delete existing blocking rules immediately. Because Google, Microsoft and OpenAI will not enforce the shared token until early 2027, removing existing blocks against specific bots would expose your pages to training in the meantime. If your web team already blocks specific agents, keep those rules active.

Second, add the new token alongside your existing lines. Inserting User-Agent: __usage_ai-training and Disallow: / into your robots.txt file today signals your intention to Anthropic immediately and ensures your site is protected as each other company turns on enforcement.

Finally, understand what this does not protect. The directive does not hide your site from search engines or prevent competitors from reading your public pricing. It only asks participating AI developers not to ingest your proprietary articles, case studies or product catalogues into their future foundation models. If your competitive edge relies on proprietary material published on your site, pairing robots.txt rules with login walls or paid access remains the safest route.