PANONDA
Últimas
← News

#adThis story contains affiliate links — we may earn a commission, and the verdict does not change because of it.

Plataformas·3 min read

Four AI companies are reportedly backing one robots.txt opt-out. Don't touch your file yet.

OpenAI, Google, Microsoft and Anthropic are said to support a shared "policy token" in robots.txt that would let publishers refuse AI training in one line. The syntax, the launch date and the enforcement are all unconfirmed.

By PANONDA Newsroom

Foto: Daniil Komov · Pexels

What happened

OpenAI, Google, Microsoft and Anthropic have reportedly agreed to support a shared standard for robots.txt: a "policy token" that tells AI crawlers whether a site's pages can be scraped or used for training. One line, honored by all four, instead of a separate rule per company. The report is dated November 5, 2026.

That's the whole confirmed fact. Everything else — the syntax, the launch date, which crawlers it covers, whether the commitments are binding — is not public.

Why it matters anyway

Right now, saying no to AI scraping is clerical work. You track user-agent strings for GPTBot, Google-Extended, Bingbot variants, ClaudeBot and a long tail of others. Each company names its crawlers differently, splits training from search differently, and changes the names without telling you. You also have contract language sitting on top of the technical rules, usually written by someone who never looked at the robots.txt file.

A shared token wouldn't give you more rights — it would make the rights you already have cheaper to exercise. That's a real gain for a two-person newsletter operation or a podcast network with a site nobody has touched in a year. It is a much smaller gain for anyone with a lawyer already negotiating licensing terms directly.

What this is not

It is not a payment system. The reported standard governs permission signals. It does not create a licensing market, a rate card or a check.

It does not reach backward. A new robots.txt line can affect future crawls. It cannot pull your archive out of a model that already trained on it, and it cannot unwind the copies sitting in someone's dataset.

It is not a legal opt-out. A technical signal matters in court only if a judge decides it's evidence of intent. Robots.txt has never been a contract. Nothing in this report changes that.

It probably doesn't treat all AI uses the same. Training, search indexing, retrieval for chatbot answers, evaluation runs and caching are different activities. A single token that blocks all of them would also likely cost you visibility in AI search surfaces — which is where some publishers are now being paid. That tradeoff is the part nobody has specified.

And it is one report. No company has published a spec. Treat this as a direction of travel, not a product.

The questions that decide whether it's worth anything

When the specification lands, these are the six things to read for:

Which companies honor it on day one, and which merely "support" it. Whether the token separates training from search answers — if it doesn't, opting out may delist you. Whether affiliated products and third-party crawlers that resell data are covered. Whether you can verify compliance in your own server logs. Whether a licensed partner can still get access while unlicensed training stays blocked. And whether there's any audit or penalty when a crawler ignores it.

If the answer to the verification question is no, the standard is a politeness convention. That's not nothing — the original robots.txt was also a politeness convention — but price it accordingly.

What to do this week

  1. Change nothing in your robots.txt on the strength of this report. There is no syntax to adopt yet.
  2. Pull your current file and write down what it actually blocks. Many publishers who think they opted out blocked one crawler two years ago and nothing since.
  3. Turn on crawler logging if you haven't. When a standard arrives, log evidence is the only way you'll know who respects it.
  4. Keep per-company rules in place. A shared token should be additive, not a replacement, until you've seen it work.
  5. Check your existing licensing contracts for language that overrides technical signals. A signed deal beats a robots.txt line.
  6. Decide now, internally, whether you want to be in AI search answers at all. That's the question the token will force, and it's a business call, not a technical one.
  7. Diary a check for when a draft spec publishes. Read the training-versus-search distinction first.

Sources

Everything above was checked against these pages. Open them and see for yourself — that is why they are here.

  1. 1luongnv.com
  2. 2ashutoshai.com
  3. 3aiweekly.co
  4. 4note.com
Share

Get the stories by email

What changes in AI for people who publish — sourced, and to the point.

No spam. Unsubscribe anytime.