A Single AI Training Opt-Out for Publishers, Backed by Four Big Labs

A single AI training opt-out for publishers sounds almost too simple after years of per-bot blocking lists. On 30 October 2026, OpenAI, Google, Microsoft and Anthropic said they will honour one shared signal, set once in a website’s robots.txt file, instead of making every site negotiate separately with every crawler (AFP via Malay Mail). Information verified through 31 October 2026.

For a small newsroom, the old routine went like this. A new crawler appeared. Someone had to find its name, add a block, check the file had not broken, then do the whole thing again for the next company. Multiply that by dozens of bots and you get a chore that never stayed finished.

How the single AI training opt-out for publishers works

The shared signal is a policy token written into robots.txt, the plain-text file sites have used for decades to tell automated visitors where they may go. The token is named `__usage_ai-training`, and a site that adds it with a Disallow line is telling every participating company the same thing: do not use this content for training (AI Weekly). Publishers can carve out exceptions with Allow lines for particular paths, and company-specific controls, such as Google-Extended for Google, stay in place alongside it (AI Parabellum).

The proposal did not come from a marketing department launch. It was written by Kevin Kelley, a technical lead for standards at Anthropic, as an individual draft for the Internet Engineering Task Force, building on work in the IETF’s AI Preferences working group (AI Parabellum). That detail matters. Robots.txt itself began as a convention, not a law, and this token follows the same path: a draft dated 30 October 2026, expiring in May 2027, with no formal standing in the IETF standards process yet (AI Weekly).

One distinction trips people up. The token governs training, not fetching. A crawler may still visit a page; the instruction says the content it finds there may not be fed into model training (AI Weekly). Search indexing, live retrieval and training are three different uses, and this signal speaks to only one of them.

Four companies, four different clocks

Anthropic says it has already started rolling the signal out. OpenAI says its support will arrive “in the coming months.” Microsoft has set a date, 1 February 2027, and Google says sometime in 2027 (AFP via Malay Mail). So a publisher who edits robots.txt this weekend is writing a rule that only part of the industry will read straight away.

Microsoft is going one step further than its peers. Alongside the training signal, it plans page-level controls for AI grounding, the separate step where a system pulls live web content into an answer rather than training model weights on it (Zot News). Microsoft notes that a site’s training and grounding choices will not remove it from Bing search results, though both Microsoft and Google warn that opting out could make a publisher’s content less likely to surface inside AI apps and experiences (AFP via Malay Mail).

Google, for its part, says that where a site uses the shared token without naming Google-Extended directly, it will read that as a block on Google-Extended too (AI Weekly). Read that carefully if you run a site. Silence, under this scheme, gets interpreted.

Meta is not in the group of four. Its communications head, Andy Stone, welcomed a common signal as “an important step” and said Meta looked forward to working with other AI companies on the standards, without saying whether Meta will honour the opt-out itself (Zot News).

Why publishers pushed for one switch

The pledge lands in the middle of open legal fights. The New York Times sued OpenAI and Microsoft in December 2023, alleging that millions of its articles were used without authorisation or payment to train ChatGPT; the companies invoke fair use, an exception in copyright law (AFP via Malay Mail). Other publishers have chosen different weapons. Cloudflare has blocked AI bots by default on new customer sites since July 2025, and the Really Simple Licensing standard lets publishers charge bots that visit their pages (AFP via Malay Mail).

Against that backdrop, a shared token is less a surrender than a truce line. It gives publishers one public, checkable statement of preference. It gives regulators something concrete to point to. And it cuts the busywork for the thousands of small sites that never had a licensing team to begin with.

Analysis: the limits are real, and they are stated plainly by the companies themselves. The setting applies only to future training; it does not pull material out of models already trained, and scraping stays allowed by default until a publisher changes the setting (AFP via Malay Mail). OpenAI frames the mechanism as a coordination device rather than a security control, which means a scraper that chooses to ignore robots.txt faces no technical barrier at all (AI Weekly). The honest test comes in 2027, when Google and Microsoft are due to switch their support on and publishers can check, crawler by crawler, whether a one-line promise in a text file was kept.

Photo: Florian Hirzinger, via Wikimedia Commons (CC BY-SA 3.0)

Sources

1. https://www.bangsar.net.my/after-scraping-the-internet-ai-giants-pledge-no-scraping-tool-for-publishers/
2. https://aiparabellum.com/openai-google-microsoft-anthropic-shared-ai-training-opt-out/
3. https://aiweekly.co/alerts/openai-google-microsoft-anthropic-back-one-ai-opt-out-token
4. https://zot.news/article/big-ai-firms-back-shared-publisher-opt-out-standard-abm55