Website owners who don't want their pages used to train AI models may soon need just a single entry in robots.txt instead of a growing list of crawler-by-crawler rules. According to TechRadar, OpenAI, Google, Microsoft and Anthropic have agreed to support a shared opt-out signal in a site's robots.txt file, one that tells AI crawlers whether they may use its content for training.
The companies' own announcements aren't among the sources Xarmo cites for this story, so every detail below is attributed to the outlets reporting it: Engadget and TechRadar.
What the signal is
Engadget reports that the four companies have agreed to honour a new robots.txt directive that lets publishers state their preferences about content being scraped to train generative AI. The directive uses a special user-agent name, __usage_ai-training, and Engadget gives this example:
```
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
```
In that example, a site opts its whole domain out of AI training but leaves a press-kit folder available. The syntax is ordinary robots.txt; what changes is that one entry is meant to be recognised by several AI companies at once.
That is the problem it tries to solve. Engadget explains that publishers currently have to add separate rules for each company's crawler. The directive builds on a proposal at the IETF, the internet standards body, written by Kevin Kelley, Anthropic's technical lead for standards, according to Engadget.
When each company starts honouring it
Support isn't arriving all at once. Engadget reports this timeline:
- Anthropic has already started supporting the directive.
- Microsoft plans to start on 1 February 2027.
- Google plans to support it in early 2027.
- OpenAI has pledged to implement it in the coming months.
In TechRadar's report, OpenAI is quoted encouraging the rest of the industry to follow: "We encourage other AI providers to honor the signal."
What it doesn't do
The limits matter as much as the headline. According to Engadget, the directive doesn't stop crawling altogether, and it doesn't keep a site out of Google Search. Engadget says Google will read it as a stop signal for its Google-Extended user agent, which gathers material for Gemini training.
It also can't reach back in time. Engadget notes that if a company has already crawled a page, it keeps that content; the directive only applies to scraping from now on.
And it is only a request. TechRadar points out that robots.txt rules only work on scrapers that actually follow them. For bots that don't, it lists other measures site owners use: requiring visitors to log in, limiting how many requests a single IP address can make, and captchas. Each has a cost, TechRadar says, because login walls and puzzles can drive real visitors away, and some scrapers rotate between IP addresses to dodge limits.
What site owners can do now
For anyone running a blog, a portfolio, a fan wiki or a small studio site, the practical upshot is a single entry that should eventually cover four of the biggest AI companies, rather than a list that has to be kept up to date by hand. Because three of the four haven't switched it on yet, on the timeline Engadget reports, adding the entry today mostly prepares a site for 2027.
Cole's take: The useful part here is the split between training and everything else. A site owner who wants to stay in search results but stay out of training datasets has had to work out which crawler does which job at each company. One named signal for "training" is easier to reason about. The weakness is the same one robots.txt has always had: it's a polite request, and it only binds the companies that choose to honour it. I'd add the entry when your platform lets you, keep any crawler-specific rules you already rely on until each company's support is live, and treat the 2027 dates as the point where it starts doing real work.
Reporting notes
