Retry storms

The problem it solves

A backend has a two-second problem. A deploy, a lock contention spike, a garbage collection pause, a brief network blip — something that would have been invisible in a log if nobody had reacted to it. Five hundred clients notice, and every one of them retries.

The retries are new load. They arrive at the exact moment the backend has the least capacity to serve them, they push its queue deeper, which makes responses slower, which makes more requests time out, which produces more retries. The original fault clears after two seconds. The outage continues for minutes, sometimes until someone stops the traffic, because the system has entered a state where its own recovery attempts are what prevent recovery.

This is a metastable failure: a system that is stable under normal load and stable under the failure state, with no path back on its own. It is the signature failure of distributed systems, and it appears in post-mortems constantly, usually described without being named.

The mechanism

Retries multiply load. A client that retries up to three times offers a failing backend up to four times its normal traffic — precisely when the backend can least take it. That is the amplification, and it is arithmetic, not misfortune.

What turns amplification into a self-sustaining outage is the queue. Requests time out at the client after, say, 800 ms, but the request itself sits in the backend’s queue until a worker picks it up. If the queue is 4,000 deep and the backend serves 1,000 a second, the work at the head is four seconds old — its client left three seconds ago and has already retried, twice. The backend now spends all of its capacity computing answers nobody is waiting for, and every completed unit of work produces no progress. Goodput reaches zero while utilisation reads 100%.

Three mechanisms break the loop:

Exponential backoff with jitter. Doubling the wait after each attempt reduces the offered rate geometrically; jitter is what stops the retries from arriving in synchronised waves. Backoff without jitter merely rearranges the herd — every client that failed at the same moment waits the same interval and returns together. AWS’s published analysis makes the case for full jitter: sleep a uniform random amount between zero and the exponential cap, not the cap plus noise.

A bounded queue. If the backend refuses work rather than accepting it into a deep queue, requests fail fast, clients learn immediately, and the backend’s capacity is spent on work whose clients are still there. This is backpressure, and it is startling how much it helps.

A retry budget. Cap retries as a fraction of successful traffic — say, 10%. When success collapses, the budget collapses with it and retries stop automatically, regardless of what each client’s backoff policy says. This is the only mechanism here that bounds amplification in the limit, and it is the one almost nobody implements.

What the simulation shows

The default is immediate retries. Watch the offered-load panel against the original request rate: at the moment of the fault, offered load multiplies. Then watch the fault clear — the shaded band ends — and observe that the error rate does not. The system never comes back. Roughly seven requests in ten fail, indefinitely, from a two-second fault.

Now change one parameter. Exponential backoff with jitter, same seed, same fault, same load. Error rate spikes during the fault, and recovers essentially to zero the moment it clears. The two runs are identical in every respect except the retry policy, and the difference between them is the difference between an incident and a log line.

Then take away the retry policy and fix the queue instead: keep immediate retries but bound the queue at 200. It recovers. The deep queue was half the problem all along, because a shallow queue cannot hold enough abandoned work to starve the useful kind.

Finally, turn on a retry budget with naive retries: amplification is bounded whatever the backoff does.

The numbers worth carrying

Amplification is 1 + attempts. Three retries means 4× offered load at the worst possible moment. Compute this for your own clients and note that it compounds through tiers: a gateway that retries three times in front of a service that retries three times offers the database up to sixteen times its traffic.

Queue age is depth divided by drain rate. If that number exceeds your client timeout, every unit of work you complete is wasted, and the fix is a smaller queue rather than a bigger one — one of the few places where reducing a buffer improves throughput.

Budget share is the parameter that actually bounds things: retries capped at 10% of successes means a fully failed backend receives, at most, 10% extra load, forever.

Where it breaks down

Backoff still has a herd. Clients that failed together and back off by the same schedule return together. Jitter is not optional; it is the part that does the work.

Idempotency. A retry of a request that already succeeded but whose response was lost will apply the effect twice unless the operation is idempotent. Retries and idempotency are the same conversation, and any retry policy without an idempotency key is a duplicate-charge policy.

Layered retries multiply. Three tiers each retrying three times is up to 64× at the bottom. Retry at one layer — usually the outermost that can meaningfully recover — and pass failures through everywhere else.

Deadlines beat retry counts. A retry that starts after the caller’s deadline is pure waste. Propagating a deadline down the chain makes this automatic; see timeout cascades.

What people get wrong

“We retry three times, that is standard.” Standard and unbounded are compatible; three retries per client across five hundred clients is a 4× multiplier on a backend already failing.

“Backoff is enough.” Backoff without jitter synchronises. The simulation’s exponential and exponential-jitter settings are one click apart and the difference is visible.

“The fault was two seconds, so the outage was two seconds.” The whole point of this page is that it was not, and that the extra minutes were self-inflicted.

“Retries improve availability.” They improve availability against independent, transient failures. Against a saturated dependency, they are the load that prevents recovery. The distinction is whether the failure is correlated across clients — and during an incident it always is.

In production

The reference material is AWS’s Exponential Backoff and Jitter article and the Google SRE book’s chapter on handling overload, which introduces the retry budget and the client-side throttle. gRPC has retry policies with backoff and a retryThrottling budget in its service config; Envoy has per-route retry budgets; Finagle shipped retry budgets years before most people knew they needed them.

Operationally, the metric to instrument is the ratio of attempts to unique requests. It sits at ~1.0 in normal times and climbs at the start of every storm, and it is the earliest unambiguous signal that a slowdown is becoming a metastable failure. Pair it with a queue-age metric and you can see both halves of the loop.

The follow-up questions

“Your dependency has a blip. What do your clients do?” — Exponential backoff with full jitter, a bounded number of attempts, a deadline, and a budget. Say all four.

“Why jitter?” — Otherwise the failures were synchronised and the retries stay synchronised.

“The fault has cleared but errors continue. What is happening?” — Metastable failure. Explain the queue of abandoned work and the amplification, then say how you break it: shed load, drain the queue, or throttle at the entry point.

“How do you bound retries across a whole fleet?” — A retry budget as a fraction of successes. This is the answer that shows you have operated one of these.