Bots Are Now the Majority of Web Requests: Cloudflare Puts It at 57.5%

On 3 June 2026, Cloudflare CEO Matthew Prince said automated systems now account for 57.5% of HTTP requests for web pages, against 42.5% from people, the first time Cloudflare has measured the crossover, and put the shift down to agentic AI. 1 At SXSW in March he had said he didn't expect it until 2027. 2

Worth being precise about what that number is, because it gets repeated as something bigger. It counts requests for web pages across Cloudflare's network. It isn't bandwidth, it isn't time spent online, and it isn't the whole internet: one bot fetching a 5 KB page and one person opening a video both count once. Humans have not stopped using the web.

What it does mean is narrower and more expensive. If you operate anything with a public HTTP endpoint, more than half of what gets asked of you is no longer asked by a person.

Bandwidth, CPU, cache, and your graphs

Bot traffic isn't free just because it isn't malicious.

Bandwidth is the obvious one, since every crawl is transfer, and on metered hosting that's a direct bill. CPU is the less obvious one: a crawler hitting dynamic pages can run your application logic, your database queries and your template rendering just like a real request, except it doesn't stop at one page and it has no session cache.

Then there's cache pollution. Crawlers tend to request long-tail URLs that get little or no human traffic, which evicts hot content from your cache and degrades performance for real users. And log volume distorts your analytics, so if half your traffic graph isn't customers and you haven't separated it, every capacity decision you make rests on a number that's wrong.

The pattern that hurts small operators most is a crawler discovering a dynamically generated URL space, whether that's faceted search, calendars or filter combinations, and enumerating it. A site with a hundred pages of content can present a million URLs, and something will eventually try to fetch all of them.

Five things that help

Look at your logs before anything else. Split traffic by user agent and by ASN. Most people are surprised by the ratio.

Treat robots.txt as a request rather than a control. Well-behaved crawlers honour it. The ones causing you problems frequently don't, and some disguise themselves behind rotating user agents.

Rate limit by network rather than by user agent. A crawler that lies about who it is still comes from somewhere, and per-ASN and per-prefix limits survive user-agent spoofing.

Close off infinite URL spaces. Stop emitting links to filter permutations in the first place, since nofollow discourages crawling rather than preventing it, block query-parameter combinations you never intended to be crawled, and return 404 rather than generating a page for every possible input.

Cache aggressively and serve from cache. A crawler hitting a cached response costs you bandwidth only. A crawler hitting your application costs you everything.

Root, and rules you can read

Two things matter here, and neither is a feature we invented for a blog post.

The first is predictable resources you control. Every AVS ISP VPS and VDS is full KVM virtualisation with root access, so you run your own nginx, your own rate limiting, your own fail2ban rules and your own caching layer, tuned to your actual traffic mix. There's no shared control panel deciding what you may block, and no vendor WAF whose rules you can't read.

The second is that DDoS protection is included on every plan, which handles the volumetric end of hostile automation. The polite but expensive end, the crawlers that aren't attacking you and are just relentlessly reading you, is a capacity and configuration question. That's exactly why root matters.

Plans start at €5/mo for a VPS, €40/mo for a 4 vCPU VDS rising to €160/mo at 16, and €150/mo for bare metal. VPS and VDS run in Tirana, Skopje, Amsterdam and Dublin, bare metal in Tirana, Skopje and Amsterdam. If serving robots is now half your workload, being able to size and tune for that honestly is worth more than a plan that looks cheaper until the transfer bill arrives.

Who pays for the majority

There's a live argument about whether crawling for AI training is fair use of a public web page, and it isn't going to be settled by a hosting blog. What isn't in dispute is the arithmetic. Most of the requests hitting the web are now automated, somebody is paying to serve them, and right now that somebody is whoever owns the origin server.

Whatever you conclude about the ethics, decide deliberately what share of your capacity you're willing to donate to it. If you don't, your infrastructure absorbs the cost by default.

Sources

1. Bot Traffic Passes Humans Online: Cloudflare Says Agentic AI Drove 57.5% Share, Tech Times, 5 June 2026
2. Online bot traffic will exceed human traffic by 2027, Cloudflare CEO says, TechCrunch, 19 March 2026