Cloudflare Starts Charging AI Crawlers. What It Means If You Run Your Own Server.

On 1 July 2026, Cloudflare said its crawler marketplace, Pay Per Crawl, is evolving into Pay Per Use: publishers get paid when their content is actually used, when it turns up in an AI search result or an agent buys access to it on demand, rather than only when a bot fetches it. The same announcement put a date on a bigger change. From 15 September 2026, on pages carrying ads, Cloudflare's defaults will allow search crawlers but block training and agent use, and block outright any mixed crawler that doesn't let a site owner choose between the three. That applies to new customers, to new sites on existing accounts, and to free-tier customers who haven't changed their settings by then. 1

Pay Per Crawl itself is a year old. It went into private beta on 1 July 2025, letting a site set a flat per-request price and return an HTTP 402 Payment Required to any crawler that hasn't presented payment intent, with Cloudflare acting as merchant of record. 2

HTTP 402 sat in the HTTP specification for three decades, reserved and essentially unused, waiting for someone to find a use for it. Here we are.

A working toll booth, at last

We wrote last month about the underlying arithmetic. Cloudflare Radar put automated requests at 57.5% of HTML traffic in June, the first time machines have outnumbered people on the web. 3 Cloudflare's own 1 July announcement makes the same point more loosely: bots and agents now drive more than half of all web requests. 1

Until now the entire cost of serving that has fallen on origin operators, with no mechanism to recover it and no realistic way to refuse, robots.txt being a request rather than a control. Pay Per Crawl is the first time the web has had a working toll booth rather than a polite notice.

Whether you think that's good depends on whether you're a publisher or a model builder. What isn't debatable is that it changes the default from crawl freely and argue later, to negotiate first.

The catch: you have to be behind Cloudflare

One practical limitation runs through all of it. This only works if you're behind Cloudflare.

The enforcement lives at the edge, in a network that sees the request before your origin does, holds the crawler's identity and payment relationship, and can return 402 on your behalf. That's a real service. It's also a further concentration of power in the least visible part of the stack, which a fortnight ago spent an afternoon demonstrating how little anyone can see of it, as we covered last week.

So the web gets a payment mechanism, and the price of using it is a deeper dependency on one company sitting between you and your readers. None of that requires bad intent. A hard coordination problem gets solved by whoever is big enough to solve it, and whoever solves it sets the terms.

Without a toll booth

If you run your own servers and would rather not hand the relationship to an intermediary, the tools are less elegant but they're yours.

Rate limit by ASN and prefix rather than user agent, because crawlers rotate user agents but don't rotate out of their address space nearly as easily. Google, Bing and OpenAI publish machine-readable IP ranges for their crawlers, and forward-confirmed reverse DNS will tell you whether something calling itself Googlebot actually came from Google. The crawlers most likely to be costing you money are the ones that do neither.

Serve cheap responses to expensive visitors. Aggressive caching means a crawl costs you bandwidth rather than CPU and database time.

Kill infinite URL spaces. Faceted search, calendars and filter permutations turn a hundred pages of content into a million fetchable URLs.

Publish what you want crawled and gate the rest. A generous public summary plus an authenticated full version is a blunt instrument that works everywhere and needs nobody's cooperation.

And measure it first, because you can't make a policy decision about a cost you haven't quantified.

We sell the machine and get out of the way

Ours is the unglamorous position: we aren't in the path, and we don't want to be.

Every AVS ISP VPS and VDS is full KVM virtualisation with real root access, so every control in that list is yours to implement: your nginx, your rate limits, your caching, your fail2ban, your rules, readable and modifiable by you. There's no vendor WAF whose logic you can't inspect and no control panel deciding which traffic you're permitted to refuse.

DDoS protection is included on every plan, which covers the hostile end of automation. The polite but costly end is a capacity and configuration question, and that's why owning root matters more than it used to.

Plans start at €5/mo for a VPS, €40/mo for a 4 vCPU VDS rising to €160/mo at 16, and €150/mo for bare metal. VPS and VDS run in Tirana, Skopje, Amsterdam and London, bare metal in Tirana, Skopje and Amsterdam. Crypto accepted, no KYC.

What the web is buying

Pay Per Crawl is a clever answer to a real problem, and we expect it to work. We also think it's worth naming the trade clearly. The web is buying a functioning content market by accepting more intermediation, at a moment when the industry has just spent a fortnight unable to say which intermediary had actually broken.

If you use it, use it deliberately. And keep a path to your origin that doesn't depend on it.

Sources

1. Cloudflare Allows the Agentic Internet to Flourish with a Simple Philosophy: Your Content, Your Rules, Cloudflare, 1 July 2026
2. Introducing pay per crawl: enabling content owners to charge AI crawlers for access, Cloudflare, 1 July 2025
3. Bot Traffic Passes Humans Online: Cloudflare Says Agentic AI Drove 57.5% Share, Tech Times, 5 June 2026