PANONDA
Últimas
← News
Regulação·3 min read

One robots.txt line to opt out of AI training — if the companies actually honor it

OpenAI, Google, Microsoft and Anthropic are reportedly weighing a shared opt-out token publishers could add to robots.txt. There's no spec, no launch date and no enforcement yet. Here's what to set up now.

By PANONDA Newsroom

Foto: Andrew Neel · Pexels

What happened

OpenAI, Google, Microsoft and Anthropic are reportedly looking at adopting a shared policy token publishers could drop into robots.txt to signal that their content should not be used to train AI models. The reporting ran November 1, 2026, updated November 3.

Today, that preference has to be expressed separately for each company — different user agents, different documentation, different products. The proposal would let you say it once.

Why the distinction matters

The signal would sit next to your existing crawler rules, but it would answer a different question. Existing robots.txt lines mostly govern whether a crawler may fetch and index a page. This one would govern whether the content may be used to train a model.

That gap is where publishers have been losing money for two years. Blocking a search crawler costs you search traffic. Blocking an AI-answer crawler costs you AI-surface visibility. Blocking a training crawler costs you, in theory, nothing in referrals — but there has never been a clean way to do only that across all four major labs at once.

A single shared token would cut the configuration work. It would not create an entitlement to be paid, and it would not bind any company that hasn't adopted it.

What this story is not

It is not a live control. The reporting says the companies are looking to adopt it. It does not establish that all four currently honor the signal, or any of them.

It is not a published standard. There is no confirmed technical specification, no launch date, no syntax you can copy into a file this afternoon. Anyone showing you the exact line to paste is guessing.

It is not enforcement. robots.txt has never been legally binding. It is a request. The entire value of this proposal rests on voluntary compliance by companies currently being sued over scraping by multiple newspaper groups.

It is not a licensing deal. Opting out of training is a refusal, not a negotiation. If your plan is to get paid, a blanket opt-out is the starting position in that conversation, not the outcome. Note the context: publishers were reportedly outraged in late October over book sales into Anthropic training sets — that fight is about money, not about file syntax.

And it says nothing confirmed about search visibility. Whether using the token affects how you rank or appear in AI answers is not established.

The part you can act on now

The standard isn't real yet. Your crawler posture is. Most independent publishers have never audited what they're currently allowing, which means they can't tell whether a new token would change anything for them at all.

Do the inventory now so that when the spec lands — if it lands — the deployment is ten minutes instead of a month.

What publishers should configure now

  1. Pull your current robots.txt and read it line by line. Most were written for Google in 2019 and never touched again.
  2. List every crawler control you actually have: robots.txt, CDN rules, Cloudflare bot settings, paywall logic, WAF rules. They often contradict each other.
  3. Write down three separate business decisions, on paper: do you allow search indexing, do you allow AI answers, do you allow model training. Most publishers want yes, maybe, no. Few have it configured that way.
  4. Pull 90 days of server logs and rank AI crawler user agents by request volume. You need a baseline before and after any change.
  5. Email your platform and vendor contacts one question: which specific crawler identities and which products will obey a shared opt-out token. "We support the standard" is not an answer.
  6. Save dated copies of your policy file and any crawler or licensing correspondence. If compliance is ever litigated, the record is the evidence.
  7. Don't blanket-block before the spec exists. Some crawlers drive link previews and referral traffic. Stage any change and watch two weeks of referrals.
  8. Skip the revenue modeling. No payment mechanism is attached to this proposal.

Sources

Everything above was checked against these pages. Open them and see for yourself — that is why they are here.

  1. 1indianexpress.com
  2. 2master-ia.fr
  3. 3staradvertiser.com
Share

Get the stories by email

What changes in AI for people who publish — sourced, and to the point.

No spam. Unsubscribe anytime.