Timeout cascades

The problem it solves

Three teams set three timeouts. The gateway team picks one second, because that is the longest a user should wait. The service team picks two seconds, because their calls occasionally take a while. The database team picks five seconds, because some queries are genuinely slow. Every number is defensible. Together they are nonsense.

When the database slows down, the gateway gives up after one second and returns an error to the user. The service is still waiting — it has another second to go. The database is still working — it has another four. The user’s request is over, and the machinery behind it grinds on for four more seconds producing an answer with nowhere to go. Under load, a large fraction of your capacity is spent on work that has already been abandoned.

The mechanism

Timeouts must decrease as you go inward. The outermost caller has the longest budget, and each hop inside gets less, because each hop must complete inside its caller’s remaining patience. Any inner timeout longer than its caller’s is unreachable — it can never fire before the caller has already given up, so it is not a timeout, it is decoration.

Arranging the numbers by hand works, and it is fragile: it breaks the moment a service gains a hop, a retry, or a second caller with a different budget. A service called by a gateway with a one-second budget and by a batch job with a five-minute budget cannot have one correct timeout.

The robust version is a deadline. The entry point computes an absolute time by which the answer is useless — now plus one second — and passes it down with every call. Each hop computes its remaining budget as deadline − now, uses that as its timeout, and refuses to start work that cannot finish in time. Deadlines compose automatically, survive retries (a retry inherits the remaining budget, not a fresh one), and make abandoned work structurally impossible rather than merely unlikely.

The other half is cancellation. A deadline that only stops waiting still leaves the downstream working. The deadline must propagate as a cancellation signal — gRPC does this natively, Go’s context carries it, and HTTP requires you to notice the closed connection — so the database actually stops rather than merely being ignored.

What the simulation shows

The default is the pathological configuration: gateway 1s, service 2s, database 5s. When the fault arrives, watch the panel counting capacity spent on answers nobody is waiting for. It climbs immediately and stays high. The sequence view shows one request through the chain: the gateway’s bar ends, and the two bars inside it keep going.

Now reverse the ordering so the gateway waits longest and each inner hop less. The abandoned-work counter largely collapses. Same fault, same latency, same load — the only change is the relationship between three numbers.

Then turn on deadline propagation, which is the version you should actually build. Wasted work goes to essentially nothing. Note what does not change: goodput. The database really is too slow, and no amount of timeout arithmetic makes it fast. What changes is that not one cycle is spent on an answer with nowhere to go — so the capacity you have is applied entirely to requests whose callers are still present, and the system degrades gracefully instead of collapsing.

The numbers worth carrying

Wasted work per abandoned request = (inner timeout − outer remaining budget) × the resources it holds. At 300 requests a second with a 4-second overhang, that is 1,200 request-seconds of capacity per second of overload, which is why the effect is not marginal.

Budget arithmetic for a chain: the outermost budget must cover every hop plus retries plus network. If the gateway has 1,000 ms and there are three sequential hops, no hop can be given more than about 300 ms once you leave room for network and serialisation — and if any hop retries once, its budget must cover two attempts. Retries consume the budget; they do not extend it. Doing this arithmetic out loud is the whole answer to the interview question.

Where it breaks down

Clock skew. Absolute deadlines cross machine boundaries, and machines disagree about the time. Send a remaining duration rather than an absolute timestamp, or accept skew-sized error. gRPC sends grpc-timeout as a duration for exactly this reason.

Non-cancellable work. A database that has begun a query will finish it whether or not you are listening — Postgres needs an explicit statement_timeout or a cancel request; many drivers only stop reading the socket. A deadline that does not reach the actual worker saves your thread but not the downstream’s capacity.

Queueing time counts. The deadline is spent while a request waits in a queue, not only while it is being served. A request that spends 900 ms of its 1,000 ms budget queued should be rejected on dequeue, not started — this is one of the highest-value cheap optimisations available during overload, and it is exactly the mechanism that keeps the retry storm from becoming metastable.

Fan-out. Parallel calls share a budget rather than each getting a fresh one, and the barrier means the slowest sets the pace — see tail latency.

What people get wrong

“We have timeouts.” The follow-up is: are they consistent down the chain, and do they include retries? Almost nobody has done the arithmetic, and the answer is what the question is actually asking.

“Set generous timeouts to avoid failing good requests.” Generous inner timeouts hold resources during an incident and produce answers nobody wants. A timeout is a statement about how long the answer stays valuable, not about how long the work might take.

“The client timed out, so we stopped.” Only if cancellation propagated. Check.

“Retries are independent of timeouts.” A retry inside a deadline must fit in the remaining budget. A retry policy specified in attempts rather than time is how a 1-second budget turns into a 6-second request.

In production

gRPC deadlines propagate through metadata and are cancelled automatically — this is the single strongest argument for gRPC in a deep service mesh. Go’s context.Context carries deadline and cancellation together and is threaded through every standard library call that blocks. Envoy has per-route timeouts and honours upstream deadline headers. On the storage side, statement_timeout in Postgres and maxTimeMS in MongoDB are the mechanisms that make cancellation real.

The operational metric worth building is abandoned work: requests that completed after their caller gave up. It is usually zero and it is the fastest way to detect a misordered chain, because it goes non-zero the instant an incident starts and stays zero the rest of the time.

The follow-up questions

“Your gateway times out at 1 second. What is the database timeout?” — Less than one second minus everything above it, and be ready to do the subtraction including retries.

“How do timeouts and retries interact?” — Retries consume the budget. Two attempts inside a 1-second deadline means each gets under 500 ms.

“What happens to the work after a timeout?” — If the answer is not “it is cancelled”, you have found the wasted capacity.

“What is better than per-hop timeouts?” — Deadline propagation, and say why: composability, retry correctness, and the ability to reject queued work that can no longer be delivered.