Queues and ordering

The problem it solves

Work arrives faster than one worker can process it, so you add workers. Every one of them pulls from the same queue, throughput scales linearly, and the design looks finished.

Then a bug report: a user’s account shows a stale email address. The trace is clean — an update message and a create message for the same account, processed by two different workers, and the update finished first. Both succeeded. The final state is wrong. Nothing in the queue promised otherwise; the guarantee you assumed was never offered.

Competing consumers scale beautifully and reorder messages about the same entity. That trade — parallelism against ordering — is the whole subject.

The mechanism

A shared queue with competing consumers hands each message to whichever worker is free. Throughput scales with worker count, one slow worker delays only its own message, and there is no ordering guarantee whatsoever, because two messages can be in flight simultaneously on different machines.

A partitioned log — Kafka, Kinesis, Pulsar — assigns each message to a partition by a key, and each partition is consumed by exactly one consumer in a group, in order. Messages about the same entity share a key, so they share a partition, so they are processed in order. That is the guarantee you actually needed: order within a partition, not across the topic.

The cost is exact and comes in two parts.

First, parallelism is capped by partition count. Twelve consumers and six partitions means six idle consumers. Adding partitions is possible and changes the key-to-partition mapping, which breaks ordering across the change — so partition count is a decision made early and regretted later, exactly like a sharding key.

Second, and more painful: a stalled consumer blocks its whole partition, and no other consumer may help. Helping would mean processing out of order, which is the guarantee you asked for. The backlog for that partition grows while other consumers sit idle.

What the simulation shows

The default is a shared queue with twelve consumers. Watch the out-of-order panel: messages about the same entity applied in the wrong sequence, continuously, at a rate that scales with how parallel you are. Throughput, meanwhile, is excellent.

Now switch to a partitioned log. Out-of-order applications go to zero. Then watch what happens when a consumer stalls — the shaded window — and read the per-partition backlog panel. One partition’s backlog climbs alone; the other five are unaffected; and the eleven other consumers cannot touch it. That single partition’s lag is a user-visible delay for every entity that hashes to it, and the only remedies are to fix the consumer or to accept the wait.

Then the failure that surprises people. Turn on a hot key. One entity produces most of the messages, they all share a key, they all go to one partition, and that partition is capped at one consumer. Your effective throughput is now one consumer’s worth, however many you have deployed. It is precisely the hot-partition failure from the storage side, with the same lack of an easy answer: change the key, split the hot entity into sub-keys and give up ordering for it specifically, or process it differently.

The numbers worth carrying

Maximum parallelism equals partition count. Choose it from peak throughput divided by per-consumer throughput, then multiply by a healthy factor — 2 to 4 — because you cannot easily add partitions later without breaking key affinity. Over-provisioning partitions costs a little broker overhead; under-provisioning costs a migration.

Consumer lag is the metric, and lag in time is the version worth alerting on. Lag of a million messages is meaningless without a drain rate; lag of forty seconds is immediately interpretable and comparable across topics. This is backpressure arithmetic: depth divided by rate is the age of the head.

Rebalance cost: when a consumer joins or leaves, partitions are reassigned and processing pauses for the group. With a naive assignor that pause affects everybody, which is why cooperative or sticky assignment exists — and why deploying a consumer fleet during peak is a decision, not a routine.

Where it breaks down

Ordering guarantees are narrower than they sound. “Ordered” means within a partition, from the broker to one consumer. If your consumer processes messages concurrently after fetching them, you have thrown the guarantee away inside your own process — a genuinely common bug.

Retries reorder. A message that fails and is retried later arrives after messages that came behind it. Ordering plus retries is a contradiction unless the retry blocks the partition, which is what a dead-letter queue is for: move the poison message aside so the partition can continue, and accept that it is now out of order (or lost) for that entity.

At-least-once means duplicates, always, and ordering does not save you from them. Consumers must be idempotent — see idempotency.

Exactly-once is scoped. Kafka’s transactional exactly-once covers consume-process-produce within Kafka. The moment your side effect is an external database, you are back to at-least-once.

What people get wrong

“The queue preserves order.” Only within a partition, and only if your consumer is single-threaded per partition. A shared queue with competing consumers preserves nothing.

“Add consumers to catch up.” Only up to the partition count. Beyond that, extra consumers idle. This is the answer that shows you have operated one.

“Kafka is a message queue.” It is a partitioned, replayable log, and the differences matter: consumers track offsets rather than the broker tracking acknowledgements, messages are not deleted on read, and reprocessing from a past offset is a normal operation rather than a recovery procedure.

“We need global ordering.” Almost nobody does, and it costs you a single partition and therefore a single consumer. Per-entity ordering is what is actually required; say so and pick the key.

In production

Kafka and Pulsar are partitioned logs; SQS standard queues are unordered competing consumers; SQS FIFO queues offer ordering per message-group-id, which is precisely the partition-key idea. RabbitMQ preserves order per queue with a single consumer and loses it as soon as you add a second, which is a surprise to many teams.

The pattern that makes all of this manageable: choose the ordering key deliberately, make consumers idempotent, and monitor lag in seconds. The first decides your parallelism ceiling, the second makes redelivery safe, and the third is the metric that tells you which of your partitions is unhappy — because with partitioned consumption, the average is always fine and the worst partition is the incident.

The follow-up questions

“You have twelve consumers on one queue. What is the ordering guarantee?” — None. Then propose partitioning by entity key.

“One consumer is stuck. What happens to the others?” — With a shared queue, nothing. With partitions, its partition stops and nobody may help. Name the dead-letter queue as the escape.

“How many partitions?” — Peak throughput over per-consumer throughput, times a headroom factor, because changing it later breaks key affinity.

“A hot entity produces half your messages. Now what?” — One partition, one consumer, capped throughput. Sub-key it and give up ordering for that entity, or handle it out of band.