Failure modes and the patterns that contain them — every question, written out
Retry storms, circuit breakers, bulkheads, timeout budgets and the metastable failures that outlast their own cause.
A backend fails for two seconds. Forty seconds later it is still failing under normal load. Why?
Code diagnosis
Client retries are now offering it several times its usual traffic.
This is metastable failure: a brief shove moves the system into a second stable state it stays in under load it previously handled. The retries are load, and while they persist the backend cannot get far enough ahead to stop producing the errors that cause them.
See it run — The fault ends at 17 seconds. Watch the error rate afterwards.
Exponential backoff without jitter fixes less than people expect. Why?
Trade-off & selection
Every client waits the same interval, so the herd arrives together again.
Backoff spaces out one client’s attempts; jitter decorrelates different clients from each other. Without it, a thousand clients that failed together retry together, and the recovering backend meets the same wall a little later.
See it run — Same seed, same fault, and it recovers.
What does a retry budget guarantee that backoff alone does not?
Invariant identification
A hard ceiling on amplification regardless of how backoff is configured.
A budget caps retries at a small share of successful traffic — ten per cent is common. When success falls, the retry allowance falls with it, so a failing dependency cannot be handed a multiple of its normal load however each client is configured.
See it run — Amplification stays near one throughout.
What is a circuit breaker actually protecting?
Invariant identification
Your own threads, which a hanging dependency would otherwise consume.
A dependency that hangs takes one of your workers per in-flight call for the whole timeout. Within seconds the pool is full and every endpoint fails, including ones that never touch that dependency. The breaker gives the threads back.
See it run — Watch the worker pool saturate.
A breaker admits all traffic in half-open to test recovery. What goes wrong?
Edge case reasoning
The probe is a herd aimed at something that just stopped being broken.
A dependency that has just recovered has cold caches, empty pools and a backlog. Full traffic knocks it straight back down, the breaker reopens, and the cycle repeats. Admit a handful of probes and require several to succeed before closing.
See it run — Compare pool saturation against the probe policy.
The recommendations service slows down and the checkout endpoint starts failing. What connects them?
Code diagnosis
A shared thread pool that the slow endpoint has filled.
One shared pool means one endpoint holding threads for two seconds instead of thirty milliseconds starves every other endpoint. This is the failure most often called "cascading" in a post-mortem, and the fix is isolation rather than a timeout.
See it run — Checkout failing while its own dependencies are healthy.
You give each endpoint its own slice of the pool. What have you given up?
Trade-off & selection
Pooling: an endpoint can be rejected while another slice sits idle.
Bulkheads trade utilisation for containment. You will reject requests on a full slice while other slices idle, and that is the point: you are choosing which thing breaks rather than letting the busiest endpoint decide for you.
See it run — The slow endpoint still fails; nothing else does.
Gateway waits 1s, service 2s, database 5s. What does that combination produce?
Code diagnosis
Work continues after the caller has gone, wasting capacity on unwanted answers.
Timeouts must decrease as you go inward. Otherwise the gateway gives up while the service waits on a database call with four seconds left, and that capacity is spent on an answer with nowhere to go — at exactly the moment capacity is scarce.
See it run — The share of capacity spent on abandoned work.
Why is a propagated deadline better than a set of ordered timeouts?
Comparison
Each hop knows the time left and refuses work it cannot finish.
Ordered timeouts are a set of independent numbers that must be kept consistent by hand across every service and every change. A deadline is one number that travels with the request, so no hop ever begins work that cannot be delivered.
See it run — Wasted work goes to zero at the same goodput.
Database load spikes to thousands of identical queries for one row. What has happened?
Code diagnosis
A hot key expired and every request for it missed at once.
Thousands of copies of one query is the signature of a cache stampede. The fix is request coalescing — the first miss fetches, everyone else waits for that answer — and it is a few lines in any real client library.
See it run — Database queries per second at each expiry.
Why can a deep request queue make an overload worse rather than better?
Edge case reasoning
It fills with work whose clients have already given up.
If queue wait exceeds the client timeout, every request served is one nobody is waiting for, so goodput collapses while the server stays fully busy. Bound the queue to roughly the timeout times the service rate, and drop anything already expired.
See it run — The share of completed work that was already abandoned.
Why is a node that is slow harder to handle than one that has crashed?
Edge case reasoning
Health checks pass, so it keeps receiving traffic it cannot serve.
A crashed node is removed instantly; a slow one answers its health check and keeps taking its share. Everything routed to it waits out a timeout, so a single degraded node poisons the tail across the whole fleet. Latency-based ejection, not liveness checks, is the answer.
Load spikes at exactly the top of every hour. Where would you look first?
Code diagnosis
Scheduled jobs and TTLs landing on the same boundary.
Everything scheduled at a round number fires together: cron jobs on the hour, TTLs set during a deploy expiring together, clients polling on a shared interval. Jitter every schedule — the fix is one random offset and it is almost never applied by default.
See it run — Spread expiries and the spike flattens.
What should a service do when it is genuinely beyond capacity?
Trade-off & selection
Reject a share of requests immediately, cheaply, and by priority.
Beyond capacity the only question is who gets served. Rejecting early and cheaply preserves capacity for the requests you do serve, and prioritising means the checkout survives while the recommendations do not.
A two-second database blip caused a forty-minute outage. Explain the mechanism to someone who was not on call.
Explain it plainly
The database blip is the trigger, not the cause. During those two seconds every in-flight request failed, and our clients are configured to retry three times immediately. So the instant the database recovered it was offered four times its normal traffic — the new requests plus everyone else’s retries. That was more than it could serve, so those requests failed too, and were retried in turn. The system had moved into a second stable state where the retries generate the failures that generate the retries, and it stays there under load it had been handling comfortably a minute earlier. Two things kept it there. The retries had no jitter, so they arrived in waves rather than spread out. And our request queue was deep enough that the wait exceeded the client timeout, which meant we were spending all our capacity on answers nobody was waiting for any more. The fix is not a bigger database — it is jittered backoff, a retry budget capping retries at a fraction of successful traffic, and a queue short enough that anything we serve is still wanted.
The answer has to separate the trigger from the mechanism that sustained it, because those are different things and only the second one is fixable.
What is the argument for injecting failures into production deliberately?
Invariant identification
Recovery paths only work if they are exercised, and untested ones do not.
Failover, retries and degradation are code paths, and code paths that never run do not work. Exercising them at a chosen moment, at a bounded blast radius, with everyone awake, is strictly better than discovering them at three in the morning.