Cloudflare switched on crawler blocking by default. Ad-supported pages are the test case.

What happened

Cloudflare's crawler policy change took effect September 15. On ad-supported pages, it blocks by default the crawlers that both index content and feed model training or agent retrieval. Search-only crawlers still get through.

At the same time, the company reframed its pay-per-crawl experiment as Pay Per Use. The shift is in what triggers payment. Pay-per-crawl paid when a bot fetched the page. Pay Per Use pays when your content shows up inside an AI answer.

That is the whole confirmed change. How many sites are affected, what the rates are, and which specific bots land on which side of the line are not confirmed in the reporting available.

Why the trigger change matters more than the blocking

Blocking is old news. Robots.txt has existed for thirty years and AI companies have treated it as a suggestion. What is new is the billing event.

A fetch is cheap and frequent. An appearance in an answer is rare and valuable. Moving payment from the first to the second means the number of paid events drops sharply, and the price per event has to rise to compensate. Nobody has published that price.

If you are paid per crawl, you get paid for existing; if you are paid per use, you get paid for being the answer — and most pages are never the answer.

This is the same logic Google is using. Digiday reported on September 14 that Google's pay-per-value licensing program pays publishers when their content significantly contributes to an answer in Gemini, AI Overviews and AI Mode. Two infrastructure-layer players, same model, same unpublished formula. That convergence is the story.

What this is not

It is not a wall. Search crawlers still index ad-supported pages. If you were worried this would drop you out of Google Search, the reporting says the opposite — search-only crawlers are explicitly still allowed.

It is not a payment you can count on. Pay Per Use is a framework announcement. No confirmed rate card, no confirmed participating AI companies, no confirmed payout schedule. Treat it as a channel that might open, not revenue you can model.

It does not cover video or audio. This is web-page infrastructure. If your work lives on YouTube, in a podcast feed, or inside a newsletter platform you don't control, Cloudflare sits nowhere in that path. Your Substack's CDN decisions are Substack's.

And it is not automatic for everyone. Default settings apply to pages Cloudflare classifies as ad-supported. If your site is subscription-gated or you self-host outside Cloudflare, nothing changed for you on September 15.

The part publishers keep getting wrong

Blocking a crawler and getting paid are separate decisions with separate consequences. Block everything and you may vanish from AI answers that currently send some referral traffic. Allow everything and you have no leverage in any negotiation.

Nobody has published the trade ratio. Until someone does, you are guessing with your own traffic.

What to do about it

1. Log into your Cloudflare dashboard this week and check which bot management settings are now on by default for your zone. The change was applied; the question is whether it applied to you.
2. Pull 30 days of crawler logs before the block took effect and 30 days after. That before/after is the only evidence you will have if a rate negotiation ever happens.
3. Baseline your referral traffic from AI surfaces now — Gemini, AI Overviews, Perplexity, ChatGPT. If blocking costs you clicks, you need the number from before.
4. Decide, in writing, which sections of your site are worth blocking and which are cheap promotion. Not every archive page deserves the same policy.
5. Do not sign anything that prices your content per crawl. The market moved to per-use in one week. A per-crawl deal locks you into the old unit.
6. Set a 45-day check for a published Pay Per Use rate. If no rate exists by then, the program is an announcement, not a revenue line.