Two Config-Driven Outages, A Year Apart: CloudFront and 1.1.1.1
Last Thursday, 16 July, AWS CloudFront started returning 5xx errors for any distribution using VPC Origins, and kept doing it from 07:45 to 11:18 UTC: three hours and thirty-three minutes. AWS put the root cause at "an internal constraint on the fleet that manages connections to private VPC origins", which left that fleet unable to load its updated network configuration. 1 Distributions using other origin types were untouched, but CloudFront sits in front of enough of the internet that one feature failing broke or degraded services including Canvas, Blackboard, Hugging Face and Ubiquiti. 2
Two days before that, on 14 July, the first anniversary of a failure with the same shape passed without much comment. In 2025, Cloudflare's 1.1.1.1 public DNS resolver went dark for 62 minutes. A pre-production Data Localization Suite service had been wired to the 1.1.1.1 prefixes by mistake in early June, and the mistake sat dormant until a second change added an offline test location and triggered a global configuration refresh, withdrawing those prefixes from every Cloudflare data centre at once. Despite the speculation at the time, it was not a BGP hijack. Cloudflare's postmortem is explicit that its own automation withdrew the routes. 3
No fibre was cut. No datacentre burned. In both cases the machines were fine and the instructions were wrong.
The control plane is the fragile part now
We've spent this year writing about physical failures: the AWS Middle East events in March, the Zayo fibre cut that degraded half the web's apps on 22 June, submarine cables generally. Those are easy to understand and, in an odd way, comforting. A cable is a thing, and things can be made redundant.
Configuration distribution is harder, for reasons that are structural rather than a matter of competence.
It's global by design. The entire value of a distributed config plane is that a change reaches everywhere quickly, which is exactly the property that makes a bad change reach everywhere quickly.
It's also the recovery path. If your config system is unhealthy, the mechanism you'd normally use to fix things is the mechanism that's broken. Physical redundancy doesn't help here; you have plenty of working servers, all obediently doing the wrong thing.
And it has no natural blast-radius boundary. Regions and availability zones bound physical failure. One fleet inside CloudFront failed to load one configuration, and the errors were worldwide. A config push doesn't respect those boundaries unless someone engineered it to, and staged rollout is the kind of discipline that erodes quietly under delivery pressure.
DNS is the same story one layer up. A route withdrawal is a configuration statement. The resolver was healthy; it stopped being reachable because the network was told it wasn't there.
Five habits worth copying
The uncomfortable part is that these are two of the most operationally sophisticated organisations on the internet. If distributed configuration bites them, an ordinary team's Ansible run at 5pm on a Friday isn't somehow safer.
Stage everything, including config. If a change reaches 100% of your fleet at once, you have a global blast radius regardless of how many regions you run in.
Keep a recovery path that doesn't use the config plane: console access, a static fallback, a manually editable file, something that works when the automation doesn't.
Don't buy your authoritative DNS from the provider you hide your origin behind. When that vendor has a bad day it removes the problem and your ability to route around it in the same stroke.
Configure a second recursive resolver, publicly and internally. July 2025 is the argument for this one: a resolver can be entirely healthy and still unreachable, a second one costs nothing, and if resolution fails then nothing else you built matters.
And know your rollback time, measured rather than estimated. "We can roll back" isn't a plan until someone has timed it.
A way in when the automation fails
We're a small ISP and we won't pretend our config plane is more sophisticated than Cloudflare's. It's smaller, which changes the trade-offs rather than settling them, and occasionally works in our favour: fewer moving parts, fewer places for a global push to go global.
What we give customers is the thing that matters during an incident of this shape, which is a way in that doesn't depend on our automation being healthy.
Every VPS and VDS currently in our catalogue has noVNC console access through the client area, independent of the guest's own networking. When the network config is wrong, the firewall locked you out, or the machine won't finish booting, you get a screen and a keyboard. No SSH required, no ticket, no credential sent to anybody.
Full KVM with root access means you can recover the machine yourself, with no waiting on a vendor to push a fix to a control panel you can't bypass. And the looking glass lets you check our routing yourself, from Tirana, Skopje, Amsterdam and London, rather than taking a status page's word for it.
Then there's the standing advice we keep repeating because it keeps proving useful. Put a โฌ5/mo VPS somewhere that doesn't share a control plane with your primary. Not another region of the same provider, but a different provider in a different country. On both of those days, that machine would have been the one still able to tell your users what was happening.
The trend worth watching
Most of this year's outage news has been physical: the AWS Middle East disruption in March, the Zayo fibre cut in June. Last Thursday's was neither, and neither was the one whose anniversary it nearly shared. The industry has spent twenty years getting good at surviving hardware failure, and comparatively little effort on surviving its own instructions.
The failure mode of a well-engineered distributed system is increasingly not that it breaks. It's that it works perfectly, everywhere, on the wrong input.