Backpressure

The problem it solves

Something produces work faster than something else consumes it. That is not an exceptional condition; it is the normal condition, briefly, many times a day. The design question is what the system does about it, and there are exactly three answers: grow, drop, or push back.

The default answer, chosen implicitly by anyone who puts an unbounded queue between two components, is grow. It feels like the safe choice — nothing is refused, nothing is lost — and it is the one that produces the worst outages, because an unbounded queue does not absorb overload. It converts an error you would have noticed into a latency you will not, until the latency is thirty seconds and every client has timed out and retried and the queue now contains work whose requesters left minutes ago.

The mechanism

A queue’s purpose is to absorb variance, not mismatch. If the producer averages 800 messages a second and the consumer averages 800, a buffer smooths the moments when arrivals clump. If the producer averages 1,000 and the consumer 800, no buffer size in the universe helps: the backlog grows at 200 a second, forever, and the only variable is how long until something breaks.

Little’s Law tells you what growth costs. W = L / λ: at 800 processed per second, a backlog of 40,000 messages means the head of the queue is 50 seconds old. Nothing has failed; every message will be processed; and every one of them will be answered long after anybody cared. This is the sense in which an unbounded queue hides the truth — the metric that would have told you (errors) reads zero, and the metric that does tell you (queue depth) is one nobody alerts on.

Dropping bounds the backlog by discarding. For a metrics pipeline, a log stream, or a feed of position updates where only the latest matters, this is not merely acceptable, it is correct — a dropped sample is a small, known loss, and a delayed one is a useless answer.

Blocking — real backpressure — bounds the backlog by refusing to accept. The pressure travels upstream: the producer’s write blocks or its request is rejected, and the producer must then decide what to do, which is where the truth finally reaches somebody who can act on it. This is what TCP does with its receive window, what reactive-streams request(n) does, and what a bounded thread pool with a rejection policy does.

What the simulation shows

The default is a producer 25% faster than its consumer with an unbounded queue. Watch the queue-depth panel: a straight line up and to the right, 200 messages a second, no equilibrium. The staleness panel shows the consequence — the age of the head of the queue climbing without limit. Throughput looks fine. Errors are zero. Everything is broken.

Now switch to blocking. The queue depth flattens at the bound immediately. Rejections appear. The rejections are the point: they are the mismatch, made visible, at the moment it happens, to the component that can do something about it — slow down, shed, buy capacity, or tell a human. The backlog stops growing and the truth arrives on time.

Then switch to dropping. Depth is bounded again, and the loss is explicit and counted. Compare the staleness panel across all three: bounded queues keep the head of the queue young, which is another way of saying the answers you produce are still relevant.

The numbers worth carrying

Queue depth divided by drain rate is the age of the head of the queue. Put that number on a dashboard; it is more useful than depth alone, because depth without a rate has no units anyone feels.

Size a bounded queue from the latency you are willing to serve, not from available memory. If your SLA is 500 ms and the consumer does 800 a second, the queue can hold 400 items and not one more; a queue of 100,000 is a promise to answer two minutes late. This is the single most useful calculation on the page and the one least often done — most queue bounds are round numbers chosen because the memory fit.

Where it breaks down

Backpressure has to go somewhere. Pushing back on a synchronous caller is easy; pushing back on the internet is not. At the true edge, the only options are drop and shed, which is why a rate limiter or a load shedder belongs at the boundary — see rate limiting.

Blocking can deadlock. If a component blocks on a downstream queue while holding a resource the downstream needs, you have a cycle. This is common in thread-pool architectures where the same pool handles both halves of a request. Bounded queues plus separate pools — see bulkheads — is the standard resolution.

Rejection is load too. A rejected request that is retried immediately arrives again, and now the system spends its time rejecting. Backpressure only works if the upstream honours it: bounded retries, exponential backoff with jitter, and ideally a Retry-After. See retry storms.

Dropping the wrong thing. Head-drop and tail-drop are different decisions. For stale telemetry, dropping the oldest is right. For an ordered command stream, dropping anything is wrong and you needed blocking.

What people get wrong

“Add a queue so we do not lose anything.” An unbounded queue loses things too — it loses them to timeouts, after paying full processing cost for work nobody will read. It converts a visible loss into an invisible one.

“The queue is our buffer for spikes.” Only if the average is sustainable. Buffers absorb variance, not deficits. Divide the spike’s excess by the drain rate and you have how long the buffer lasts; if the answer is “until the spike ends”, fine, and if the deficit is permanent, no size works.

“Backpressure is a stream-processing thing.” It is TCP’s flow control, a bounded ArrayBlockingQueue, a semaphore around a database pool, an HTTP 503 with Retry-After, and a full disk. It is everywhere; most systems just implement the grow variant by accident.

“Queue depth is at zero, so we are fine.” Queue depth at zero means the consumer is keeping up or something upstream is refusing everything. Look at the accepted rate as well.

In production

Kafka’s consumer lag is exactly the staleness metric above, and lag in time is the version worth alerting on. RabbitMQ has queue length limits with drop-head or reject-publish overflow behaviour — pick deliberately. Java’s ThreadPoolExecutor has four rejection policies, of which CallerRunsPolicy is the sneaky backpressure one: the submitting thread executes the task, so producers are slowed by construction. gRPC, HTTP/2 and TCP all carry flow control that will apply backpressure for you if you do not defeat it by buffering in application code.

The design rule that comes out of all of it: every queue in the system has a bound, and every bound has a documented behaviour when it is reached. A queue whose bound is “whatever the heap allows” has both a bound and a behaviour, and neither was chosen.

The follow-up questions

“What happens when the queue is full?” — The question that distinguishes a design from a diagram. Have an answer per queue.

“Your consumer is 20% slower than your producer. How long until you have a problem?” — Deficit rate times time versus the bound; then the staleness calculation. If the queue is unbounded, until the heap dies.

“Is dropping ever acceptable?” — Yes, and say for what: telemetry, sampled traces, position updates where only the latest matters. The distinction is whether the item’s value decays.

“How do you signal backpressure to an external client?” — 429 or 503 with Retry-After, and a client that honours it. Then explain what happens if it does not.