As SREs, we commit to building reliable systems. That means SLAs, error budgets, 99.95% availability month over month, the usual promises. Reliability is about meeting expectations under normal conditions.
Resilience is different.
A reliable system delivers when things go according to plan. A resilient system, when hit by something that should break it such as a lost Availability Zone, a dependency going dark, a sudden 10x traffic spike, bends instead of snapping, and recovers without sweating (too much).
Think of an octopus. It’s not resilient because it’s tough. It’s resilient because of how it’s designed: distributed, adaptive, capable of losing parts and continuing to function.
A resilient system doesn’t just work or not-work. It sheds non-critical load, serves stale data when fresh data is unavailable, disables expensive features under pressure, and tells users honestly what’s happening. The fallback becomes a feature.
Nowadays everyone run on the cloud and by default cloud architecture offers redundancy: multiple regions, availability zones, load balancers, managed services that replicate data. But that redundancy is only useful if you actually test whether your system can use it under stress.
Cloud Outages
You can define the most bullet-proof system you can think of, and you’re still underestimating what might come at you. You’re basically building on infrastructure you don’t own, don’t operate, and can’t fully observe. AWS, Azure, or GCP will have incidents, there is no way around it. The hyperscaler’s blast radius becomes your blast radius unless you’ve explicitly designed otherwise.
Cloud provider outages are a certainty.
From the end of 2023 to the end of 2025, all major hyperscalers experienced notable outages. When a region or zone goes down, your traffic has to go somewhere.
Malicious actors are constantly probing:
-
Bot/scraping attacks can triple requests per second in a matter of minutes.
-
DDoS from botnets spawned from the tiniest IoT devices. Yes, you might get attacked by smart fridges.
-
Aggressive crawling doesn’t always come from malicious actors. Sometimes is behemots like Google, Meta, Anthropic perform massive crawls that can bring down your infra.
-
And the new kids on the block: attacks powered by LLMs scripts that adapt and change patterns so quickly that even the sharpest tools can’t detect them.
Development speed in the age of AI is exploding. More people have coding superpowers, which means more applications, more A/B tests, more features, more code on GitHub. The pace of change itself becomes a threat: more deployments, more dependencies, more hidden failure modes.
There are things you wouldn’t have imagined a year ago, threats nobody planned for, and yet, here we are. AWS datacenters set on fire by Iranian drone attacks. Military actions on datacenters. What if that happens to a zone where you have deployed critical workloads? These scenarios are no longer unthinkable. You cannot really go unprepared. Disaster scenarios can be a reality, and you need a plan. A plan by itself doesn’t give you confidence, but rehearsing the plan, testing the plan, is what makes you good at it.
What would an SRE do?
The most useful tools we have at our disposal are failovers and fault tolerance tests. Each helps answer a very specific question:
-
Failover: Can your airplane fly with a burning engine?
-
Fault tolerance: If a tiny little cog starts failing, can you still operate your engines?
Failovers: testing cloud redundancy
Failover means putting the redundancy that the cloud offers to the test. If you run on the cloud chances are you operate in multiple regions and multiple availability zones within each region. When you execute a failover test, you simulate a failure, shifting traffic to a different regions or a different zone.
A Failover test answers the following:
- What happens when a region goes down?
- What happens when a zone goes down?
The answer isn’t just “we survive.” It’s more about measuring how you survive: error rates, latency, how long recovery takes, and whether business metrics take a hit.
Fault injection: breaking things on purpose
Fault injection instead, is the more surgical version. You can deliberately break a dependency, make a service return 5xx, add latency, kill a pod, sit and watch the world burn and watch how the system reacts. This surfaces hidden contracts and false assumptions that static architecture diagrams never show.
I see you puzzled, thinking: What about load tests?
Thing is, a failover is secretly a load test. When you shift all production traffic from one region to another, the receiving region suddenly gets a massive surge. That’s why high-traffic failovers test are so important. Low-traffic tests might pass beautifully, but high-traffic tests often expose the harsh truth.
Final Thoughts
Every test will produce a list of follow-up items. If you don’t document and track them, you’ll keep re-learning the same lessons. Failover and fault tolerance testing isn’t about proving your system is perfect. It’s about finding the weak links before reality does. Each test will break something. Each broken component will teach you something. The goal is not a totally smooth experience. The goal is to see the system recover by itself, and to make that recovery boring, predictable, and automatic.
Because one day, the engine will catch fire for real. And when it does, we want to know we can still land the plane.