Autoscaling

The problem it solves

Demand is not constant. It has a daily shape, a weekly shape, a marketing-email shape, and occasionally a shape nobody predicted. Provisioning for the peak wastes most of your money most of the time; provisioning for the mean fails during the peak. Autoscaling is the promise that capacity will follow demand, so you pay for the mean and serve the peak.

The promise is real and it comes with a term nobody puts on the slide: dead time. Between the moment demand rises and the moment new capacity is serving traffic there is a delay — detect, decide, provision, boot, warm, register, pass a health check. That delay is not a tuning parameter. It is a property of your image, your runtime, your dependency graph and your cloud provider, and it sets a hard floor on what reactive scaling can achieve.

The mechanism

A reactive autoscaler is a control loop: sample a metric, compare it to a target, compute a desired instance count, act, then wait out a cooldown so the loop does not fight itself. Every one of those steps adds latency, and the loop’s total dead time is what determines whether it helps.

The simulation models the parts honestly. Instances take bootMs to become useful — the default is 90 seconds, which is unremarkable for a JVM service with a warm-up phase and generous for a container that pulls a large image. There is a cooldown between decisions. There is a floor and a ceiling on the fleet. And there is a queue in front of each instance with a bound, so when demand exceeds capacity, requests are shed rather than accepted into an unbounded backlog — which is the correct behaviour, for reasons on the backpressure page.

The two policies differ in what they measure. Request count scales on offered load per instance, which is a quantity with no upper bound: if you are ten times behind, the metric says ten times. Target utilisation scales on how busy the instances are, which is a quantity that saturates: once every instance is pinned, utilisation reads 100% whether you are 2× behind or 10× behind, and the policy asks for the same modest increase in both cases.

What the simulation shows

Run the default spike and watch the capacity-against-demand panel. Demand rises; the policy notices; instances are ordered; and then a flat 90 seconds passes during which nothing arrives. The shed-request counter climbs through that entire window. By the time the new capacity is serving, the spike is most of the way over — the fleet that actually served the peak is very nearly the fleet you already had. Reactive scaling cannot beat its own boot time, and for a spike shorter than the boot time it is nearly a no-op.

Now compare the policies. Switch to target utilisation under heavy overload and watch the fleet size settle short of what is needed while p99 stays broken. The signal saturated; the loop is asking for a fraction of the deficit and will keep asking, one cooldown at a time. The request-count policy under the same load asks for the whole gap at once.

Then take the other side of the argument seriously: widen the spike so demand ramps over minutes rather than seconds. Now the same policy, the same boot time, tracks demand closely and the tail stays flat. Autoscaling works — it works on the timescale of your boot time and slower. The failure is not the mechanism; it is applying it to a spike faster than the mechanism can respond.

Finally, turn scaling off entirely to see the counterfactual. The point of the comparison is that the honest answer to “will autoscaling save us” is “over what interval”.

The numbers worth carrying

Your boot time is the number. Measure it end to end — not “the container starts in 2 seconds” but from scaling decision to first successful request served, including image pull, JIT warm-up, connection pool fill, and the health check’s own interval. Typical figures: a Lambda cold start in the low hundreds of milliseconds; a small container on warm nodes, 10–30 seconds; a JVM service with a large heap and a warm cache, 2–5 minutes. That spread is the difference between autoscaling being a real answer and being a slide.

The second number is the shape of the demand you expect. If your spike rises faster than your boot time, reactive scaling is not the answer and you need one of the alternatives below.

Where it breaks down

Spikes faster than boot time. Covered above. The mitigations are pre-warmed capacity held in reserve, scheduled scaling ahead of known events, predictive scaling from historical shape, or a queue that turns the spike into a backlog you drain — which is only acceptable if the work is asynchronous.

The dependency does not scale with you. Tripling your stateless tier triples the connection count on your database. Autoscaling a service without checking what it fans out into is a common way to convert a front-end capacity problem into a back-end outage.

Flapping. Scale up on a spike, scale down at the end of the cooldown, scale up again on the next one. Each cycle costs boot time and, on some pricing models, a full billing increment. Asymmetric thresholds — scale up eagerly, scale down reluctantly — are the standard fix.

Scaling on the wrong metric. CPU is the default and is wrong for I/O-bound services, where the instances are idle while the queue grows. Queue depth or concurrent requests per instance is usually the honest signal, and Little’s Law tells you how to convert between them: see Little’s Law.

Scale-to-zero. Delightful until the first request after idle pays the entire cold start, and that request belongs to a user.

What people get wrong

“We will autoscale” as a complete answer to a burst question. The follow-up is always “how fast”, and the answer is your boot time.

“Target 80% CPU.” At 80% average utilisation your latency is already well up the hockey stick, and the fleet you are scaling from is the one serving traffic during the whole dead time. Targets in the 50–70% range leave headroom for the boot window, and the reason is on the utilisation curve.

“Utilisation is the natural signal.” It saturates. Under real overload it cannot tell you the size of the deficit, which is exactly the moment you need the size of the deficit.

“Autoscaling is a cost optimisation.” It is a cost optimisation and a reliability mechanism, and the two want different settings. Optimising purely for cost drives you toward high targets and low floors, which is the configuration that fails during the incident.

In production

Kubernetes HPA samples every 15 seconds by default, with a stabilisation window on scale-down; the scaling formula is ceil(current * metric / target), which is the request-count shape when the metric is a rate and the saturating shape when it is a utilisation. Cluster Autoscaler adds node provisioning on top, which adds minutes. AWS Application Auto Scaling offers step scaling, target tracking, and predictive scaling from a fortnight of history. KEDA scales on queue depth, which is often the right metric and rarely the configured one.

Whatever the mechanism, the operationally important piece is the same: know your end-to-end boot time, hold enough headroom to survive it, and put a bound on the queue so that overload sheds rather than accumulates.

The follow-up questions

“Traffic goes 10× in thirty seconds. What happens?” — The fleet you had serves the spike; autoscaling arrives for the tail. Name the mitigations: pre-warming, scheduled scaling, queueing the work, shedding deliberately.

“What do you scale on?” — Not CPU by default. Concurrency or queue depth, with Little’s Law to justify the target.

“What breaks downstream when you triple the fleet?” — Connections, per-instance caches going cold, and any dependency with a fixed pool.

“How do you stop it flapping?” — Asymmetric thresholds, a cooldown longer than the boot time, and a minimum that reflects the floor you are willing to serve from.