The utilisation curve

The problem it solves

There is a graph that explains more production incidents than any other single idea in system design, and most engineers have never seen it drawn. It plots latency against utilisation. It is flat, flat, flat — and then it goes vertical.

The practical consequence is that capacity is not linear in the way budgets assume. Going from 50% to 70% utilisation costs you almost nothing in latency. Going from 90% to 95% roughly doubles your queueing delay. The last few percent of a server’s capacity are not the same goods as the first few percent, and any conversation about “running our servers hot” is a conversation about latency whether or not anyone says so.

The mechanism

Waiting happens because arrivals and service are irregular. If requests arrived exactly every 100 ms and each took exactly 80 ms, you could run at 80% utilisation forever with zero queueing — every request would find the server free. Queues form because sometimes two arrive close together, and at high utilisation the server has no idle time in which to recover from that.

For a single server, the classical result is the Pollaczek–Khinchine formula. Written in terms of utilisation ρ, mean service time S, and the coefficient of variation cv of service time, the mean waiting time is

W = S × ρ / (1 − ρ) × (1 + cv²) / 2

Everything important is visible in that expression. The ρ / (1 − ρ) term is the hockey stick: at ρ = 0.5 it is 1, at 0.9 it is 9, at 0.95 it is 19, at 0.99 it is 99. Waiting time does not creep up as you approach capacity; it diverges. And the (1 + cv²) / 2 term says variability is a first-class cause of waiting, entirely separate from load — a workload with cv = 2 waits two and a half times as long as a constant-time one at the same utilisation.

What the simulation shows

The curve above is measured, not drawn from the formula. The simulation runs many independent lanes at a range of utilisation levels, lets each reach steady state, and records what actually happened; the Pollaczek–Khinchine prediction is overlaid so you can see theory and measurement agree. That agreement matters, because the shape is so aggressive that it reads as an exaggeration until you watch it happen.

Three things to do with it.

First, read the p99 at 70% and the p99 at 95% off the chart. Same server, same code, same request mix — several times the latency. There is no configuration change hiding in that gap; it is the arithmetic of waiting.

Second, set the server count to sixteen and watch the whole curve shift right. Pooling is real: at 95% utilisation, sixteen servers behave far better than one, because when a request arrives the chance that some server is free is much higher. This is the mathematical justification for shared pools over dedicated per-tenant capacity, and for load balancers that can actually find the free server — see load balancing.

Third, hold utilisation fixed and widen the service-time spread. Load has not changed by a single request per second, and the wait grows anyway. Variance is half the wait, and it is the half nobody budgets for.

The numbers worth carrying

The queueing multiplier 1/(1 − ρ) is worth memorising as a table: 50% → 2×. 80% → 5×. 90% → 10×. 95% → 20×. 99% → 100×. That is the factor by which the waiting component grows relative to a lightly loaded system; total latency is service time plus that wait, so the effect on the number your users see depends on how much of it was queueing to begin with — which is exactly why the effect is invisible until suddenly it is not.

The practical target follows. For a latency-sensitive service on a single-threaded work unit, keep steady-state utilisation in the 50–70% band. For a large pool with good balancing, 80% is defensible. Above 90%, you are trading tail latency for hardware cost, and you should be able to say what that trade buys.

The second number is headroom for response: if your autoscaler takes 90 seconds to add capacity, your steady-state utilisation must be low enough that the fleet you have can absorb 90 seconds of growth without crossing the knee. That is the connection to autoscaling, and it is why “target 80% CPU” is usually wrong.

Where it breaks down

The formula assumes Poisson arrivals and a single server, and real systems violate both. Multi-server systems follow the Allen–Cunneen approximation, which the simulation uses when the server count is above one; it is an approximation, and it is optimistic for heavy-tailed service times. Real arrivals are frequently more bursty than Poisson — batch jobs, cron alignment, retries after a blip — and burstiness moves the knee left.

Finite queues change the shape entirely. Once a queue is bounded, latency cannot diverge because requests are rejected instead: the hockey stick becomes a plateau plus an error rate. That is not the formula failing, it is a different system, and it is usually the better one — see backpressure.

Finally, the curve is about a bottleneck resource. A service at 40% CPU can be at 99% on its database connection pool, and the latency you measure follows the pool, not the CPU. Finding the actual constrained resource is the difficult part of applying this in production; the maths is the easy part.

What people get wrong

“We run at 90% because hardware is expensive.” That is a coherent position for a batch system, where throughput is the goal and nobody is waiting. For an interactive service it is a decision to have a bad p99, and it should be made out loud.

“We are only at 70%, so we have 30% headroom.” You have 30% of capacity and much less than 30% of latency headroom, because the remaining capacity is the expensive kind. Additionally, 70% average over a minute frequently means 95% for several seconds inside that minute, and the queue notices the seconds.

“Latency rose, so the code got slower.” Or the load rose slightly while you were already near the knee. The curve makes small load changes produce large latency changes, and it is a common false trail during an incident.

“Adding a server gives us one more server’s worth.” At high utilisation it gives you much more, because you move back down the curve. This is one of the few places where the intuitive estimate is too pessimistic.

In production

The operational form of this page is: alert on utilisation, not only on latency, and set the threshold below the knee. By the time latency alerts fire you are already on the steep part, and the steep part is where a small further increase produces a large further degradation — which is how a service goes from “a bit slow” to “timing out” in ninety seconds.

The second operational form is to reduce cv where you can. Splitting a mixed workload into fast and slow queues, capping request sizes, and moving batch work off the interactive path all reduce variability, and the formula says that reducing variability buys latency at constant load. That is often cheaper than buying capacity.

The follow-up questions

“Your servers are at 85%. Is that fine?” — Depends on the pool size, the variability, and how fast you can add capacity. Give the curve, not a number.

“Latency doubled and traffic went up 5%. How?” — You were near the knee. Show 1/(1 − ρ) going from, say, 10 to 20.

“How much headroom do you need?” — Enough to cover the time it takes to add capacity, plus enough that a normal burst does not cross the knee. Then say what your boot time is.

“Why does a bigger pool help more than the maths suggests?” — Because a free server is much more likely to exist. Sixteen servers at 95% are not sixteen copies of one server at 95%.