Queueing, latency and backpressure — every question, written out
Little’s Law, the utilisation curve, what a queue is really for, and why the tail is the only percentile that matters at scale.
A service handles 200 requests a second and each takes 50ms end to end. How many are in flight?
Complexity derivation
About 10, from L = λW.
L = λW: 200 per second times 0.05 seconds is 10 concurrent requests. It holds for any arrival pattern, any service distribution and any queue discipline, which is why it is the fastest way to sanity-check a capacity claim.
See it run — The two lines sit on top of each other whatever you turn.
Your thread pool holds 200 threads and the service takes 40ms. What throughput can it reach?
Trade-off & selection
5,000 a second, from λ = L / W.
200 threads divided by 0.04 seconds is 5,000 a second at full occupancy. Any two of the three numbers give you the third, which is how you catch a capacity claim that does not add up without instrumenting anything.
Going from 70% to 95% utilisation multiplies the average wait by roughly how much?
Complexity derivation
About six times, because wait grows as 1/(1 − ρ).
Waiting time scales with 1/(1 − ρ). At 70% that factor is 3.3; at 95% it is 20 — roughly six times more. The last few percent of capacity cost more than all the rest put together, which is why utilisation targets exist.
See it run — The measured curve lands on the theoretical one.
Two services run at the same utilisation but one has a much worse p99. What differs?
Code diagnosis
The variability of its service times.
The Pollaczek-Khinchine formula makes the wait proportional to (1 + cv²), where cv is the coefficient of variation of service time. Two services at identical utilisation can differ by an order of magnitude in wait if one has a heavy tail — which is why reducing variance is often cheaper than adding capacity.
See it run — Same utilisation, much longer waits.
Why does one pool of sixteen servers beat sixteen pools of one at the same total utilisation?
Comparison
A free server is far more likely to exist when a request arrives.
With separate pools a request can queue behind a busy server while another sits idle. Pooling removes that waste, which is why the queueing penalty falls sharply with server count at fixed utilisation — and why it is worth resisting the urge to partition a pool without a reason.
See it run — The cliff moves right, and never disappears.
A backend has a p99 of 100ms. A request fans out to 100 of them and waits for all. What is the user p99?
Complexity derivation
Far worse: the backend p99 becomes roughly the user median.
The chance that none of 100 calls hits the p99 is 0.99^100, about 37% — so roughly two thirds of user requests contain at least one slow call. The rare case for the backend is the common case for the user, and nothing about the backend changed.
See it run — Compare the two histograms.
What does a hedged request buy, and what does it cost?
Trade-off & selection
It cuts the tail sharply for a few percent more backend work.
Send a second copy of anything still outstanding after roughly the p95 and take whichever answers first. Only about five per cent of requests trigger one, so the extra load is small while the tail improvement is large.
See it run — Extra work percentage against the p99.
A producer runs 25% faster than its consumer, feeding an unbounded queue. What happens?
Code diagnosis
The queue grows without bound and the data gets steadily older.
A 25% mismatch is arithmetic, not a burst: the backlog grows at a fixed rate for as long as it lasts. The queue converts an error you would have noticed into a latency you will not, until memory runs out or somebody looks at the age of the head.
See it run — Queue depth and the age of the oldest message.
A metrics pipeline queue is full. Do you drop the oldest item or the newest?
Trade-off & selection
The oldest — recent data is what a dashboard needs.
The right drop policy follows from what the data is for. Monitoring wants the present, so drop the oldest; a payment log wants completeness, so reject at the door instead. What is never acceptable is dropping silently — emit a counter and alert on it.
See it run — Bounded depth, and a count of what was dropped.
What does backpressure actually do to a system under sustained overload?
Invariant identification
It makes the overload visible where something can act on it.
Backpressure propagates the constraint upstream: the queue is bounded, so the ingest layer stalls, so the client gets a fast rejection rather than a slow acceptance. It does not create capacity — it converts hidden latency into visible failure.
See it run — Rejections at the door, and a bounded backlog.
Why is a mean latency SLO close to useless?
Code diagnosis
Latency is long-tailed, so the mean sits above the median and describes nobody.
Service times are roughly log-normal, so the mean is pulled up by the tail and describes no actual request. A mean of 100ms is consistent with 95% of users at 40ms and 5% at 1.3 seconds, which are different systems with the same SLO.
Why does a queue form at 70% utilisation when there is clearly spare capacity?
Edge case reasoning
Arrivals clump, so requests overlap even when the average leaves room.
With Poisson arrivals, gaps are exponentially distributed and clumps are ordinary rather than exceptional. A queue forms whenever two requests arrive closer together than one service time, which happens constantly well below saturation.
What are you buying when you keep a fleet at 60% rather than 85%?
Trade-off & selection
A latency margin and room to lose capacity without collapsing.
The gap between 60% and 85% is a latency margin and a failure margin. Losing a third of a fleet at 60% takes you to 90%, which is survivable; the same loss at 85% takes you past saturation, and the queue never drains.
Why bound the number of in-flight requests rather than only setting timeouts?
Invariant identification
A timeout bounds one request; a concurrency limit bounds the resources.
With a one-second timeout and no concurrency limit, a thousand requests a second against a hung dependency means a thousand in flight. The timeout bounds each one and nothing bounds their number — which is how a slow dependency exhausts a pool.
Explain to a product manager why the team will not run servers at 90% utilisation to save money.
Explain it plainly
Waiting time does not rise smoothly with load — it rises with one over the spare capacity. At 70% busy the average request waits about three times its own service time; at 90% it waits ten times; at 95%, twenty. So the last stretch of capacity we would be buying back is the stretch that costs the most latency per unit saved. Concretely, moving from 70% to 90% would roughly triple our p99, and our p99 is what customers experience as the site being slow. There is also a failure argument: at 70% we can lose a third of the fleet and still be under 100%, which happens routinely during a deploy or an instance failure. At 90% the same loss takes us past saturation, and past saturation the queue never drains — the site does not get slower, it stops. The saving is real and it is smaller than it looks, because we would need more headroom elsewhere to compensate.
The answer has to convert a queueing result into a business consequence without arithmetic.
Your p99 looks healthy during an outage in which most requests are timing out. Why?
Code diagnosis
The percentile is computed over successes, and the slow requests failed.
Excluding failures from a latency percentile is survivorship bias, and it produces a metric that looks best when the system is worst. Record the latency of failed requests too, and watch success rate beside latency rather than instead of it.