Asynchronous messaging and delivery guarantees — every question, written out
At-least-once as a guarantee rather than a defect, the dual-write problem, ordering against parallelism, and exactly-once effect.
What does at-least-once delivery actually promise?
Invariant identification
That nothing is lost, and that duplicates are guaranteed rather than rare.
Processing and acknowledging are separate steps and a process can die between them, so redelivery is part of the contract rather than an edge case. Designing for "usually once" is designing a bug that appears under load.
See it run — Ledger drift with no protection.
A broker advertises exactly-once. What is it really giving you?
Comparison
Deduplication inside its own boundary — your side effects are still at-least-once.
What a broker can provide is producer deduplication and transactional reads-and-writes within its own system. The moment your consumer charges a card or sends an email, that effect is outside the transaction — so the effect must be idempotent regardless.
See it run — Effect-level deduplication holding through a restart.
Where must an idempotency check live to survive a consumer restart?
Invariant identification
In the same transaction as the effect, as a unique constraint.
If the deduplication record and the effect commit together, the second delivery cannot apply — the insert violates the constraint and the transaction rolls back. Anything looser can be in one state while the effect is in the other, which is the bug you were trying to prevent.
See it run — A cache losing everything at the restart.
You write a row and then publish an event. What can go wrong?
Code diagnosis
The row exists and the event never happens, with nothing recording that.
Two writes to two systems with no transaction across them means a window where one succeeds and the other does not. Reversing the order inverts the failure into events for rows that do not exist; there is no ordering that is safe.
See it run — Rows with no event, counted.
How does the outbox pattern remove the dual-write window?
Invariant identification
The event row commits in the same transaction, and a relay publishes it after.
There is now exactly one atomic write. Either both rows are there or neither is, and everything after that point is retryable: if the relay dies the unsent rows remain, and if it publishes twice the consumer deduplicates.
See it run — The broker outage becomes a queue depth.
A partitioned log guarantees ordering. Scoped to what?
Trade-off & selection
One partition — never across the topic.
Route by entity key and every message about one entity is ordered, which is almost always the guarantee actually needed. The costs are that parallelism is capped by partition count, one slow consumer blocks its partition, and a hot key is a hot partition.
See it run — Out-of-order applications drop to zero.
One queue and twelve consumers. An update overwrites a later value. Why?
Code diagnosis
Two messages for one entity were handled concurrently by different workers.
Competing consumers is the cheapest scaling there is, and it gives up ordering by construction. Either partition by entity so one consumer owns each, or make the write conditional on a version so a stale update is rejected.
See it run — Messages applied after a newer one.
A message fails every time it is processed. What should happen?
Edge case reasoning
After a bounded number of attempts it goes to a dead-letter queue.
A poison message must not be able to block a partition forever. Bound the attempts, move it aside with its error, and alert on the dead-letter depth — an unmonitored dead-letter queue is a place where data goes to be forgotten.
Which metric tells you a consumer is falling behind?
Code diagnosis
The age of the oldest unprocessed message.
Queue depth is useful and age is better, because it is in the units the business cares about: an order that has been waiting four minutes is a customer experience, whereas ten thousand messages is a number needing interpretation.
See it run — Depth and age climbing together.
A user posts a comment, the page reloads, and it is not there. The write succeeded. Why?
Code diagnosis
The read model has not processed the event yet.
Splitting the read model creates a window in which a user cannot see their own write, and it widens exactly when traffic is heaviest or the projector is redeploying. The fix is to carry the write position on the read and wait for the projection to reach it.
See it run — The share of reads missing the reader’s own write.
Which fix for projection lag has the best cost profile?
Trade-off & selection
Return the write position and have the read wait for it, with a bounded fallback.
A position token costs one integer in the response and converts staleness into bounded latency, which is the trade you wanted. The bound matters: past it, fall back to the write model rather than waiting indefinitely.
See it run — Staleness gone, latency bounded.
A service cannot keep up, so a queue is added in front. What has changed?
Code diagnosis
Nothing about capacity — only where the excess waits.
A queue absorbs a burst; it cannot absorb a rate mismatch. If the consumer is slower on average, the backlog grows for as long as that lasts, and all the queue has done is convert a visible error into an invisible delay.
How many partitions should a topic have?
Trade-off & selection
At least as many as the consumers you will ever want, since it is hard to raise later.
Partition count caps consumer parallelism, and raising it later changes which partition a key lands in — so ordering is broken across the change. Over-provision moderately at the start, because the cost of extra partitions is small and the cost of changing the count is not.
See it run — Consumers beyond the partition count sit idle.
One partition has a growing backlog while the others are empty. What is happening?
Edge case reasoning
A hot key, or a stalled consumer on that partition.
Both causes look identical on the depth chart and differ in what fixes them: a stalled consumer needs restarting, while a hot key needs a different key. Check whether the messages in the backlog share a key — that is the distinguishing test.
See it run — One partition’s backlog against the rest.
A colleague says the payment consumer is safe because the broker guarantees exactly-once. Respond.
Explain it plainly
Exactly-once delivery is not something a broker can provide across a network, because the sender cannot tell a lost message from a lost acknowledgement — so it either resends, which is at-least-once, or it does not, which is at-most-once. What brokers do provide, and it is genuinely useful, is deduplication and transactional reads-and-writes inside their own system. The moment our consumer charges a card, that effect is outside the broker’s transaction, so it is at-least-once whatever the broker says. What we actually want is exactly-once effect: let the message arrive as many times as it likes and make the second application do nothing. Concretely, the producer generates an idempotency key, and the consumer inserts that key into a table with a unique constraint in the same transaction as the ledger entry. A redelivery violates the constraint, the transaction rolls back, and no money moves. The important part is the same transaction — an in-memory set of recently seen ids catches ordinary redeliveries and loses everything on restart, which is exactly when the redeliveries arrive in bulk.
The answer has to distinguish delivery from effect and land on the unique constraint.
You must replay six months of events to rebuild a projection. What is the risk?
Edge case reasoning
The replay competes with live traffic and its side effects fire again.
Two things go wrong: the backfill saturates the same consumers and stores serving live traffic, and any handler with a side effect re-sends six months of email. Replay on separate capacity, into a shadow projection, with side effects explicitly disabled.