Idempotency and exactly-once

The problem it solves

Exactly-once delivery does not exist. It is not a hard engineering problem awaiting a better broker; it is impossible over an unreliable network, for the same reason as the Two Generals problem. A sender that gets no acknowledgement cannot distinguish “the message was lost” from “the acknowledgement was lost”, and must choose between sending again — at-least-once, with duplicates — and not sending — at-most-once, with loss.

Every real system chooses at-least-once, because losing messages is worse than repeating them. Which means duplicates are not an edge case, they are a guarantee, and the design question is what your consumer does about it. If the consumer credits a ledger, a duplicate is money that nobody spent, appearing in your books with no error, no log line, and no alert.

What is achievable is exactly-once effect: the message may arrive any number of times, and the effect happens once. That is a property of the consumer, not of the broker.

The mechanism

Three levels of protection, in increasing order of actually working.

Nothing. Apply every message as it arrives. Duplicates apply twice. The failure is silent by construction — nothing errored, so nothing is logged.

A seen-cache. Keep the identifiers of recently processed messages in memory and skip repeats. This catches ordinary redeliveries and costs nothing. It has a specific, cruel failure mode: the cache is lost when the process restarts, and a consumer restart is precisely the event that causes a bulk redelivery. The protection disappears at the exact moment it is needed most, which makes it worse than useless — it is protection you will trust.

A unique constraint in the same transaction as the effect. Insert the idempotency key into a table with a primary key, and apply the effect, in one transaction. A duplicate violates the constraint, the transaction rolls back, and nothing is applied. The database’s atomicity is doing the work; there is no window, no cache to lose, and no restart to survive, because the record of what has been processed and the effect itself commit or fail together.

That last one is the entire answer, and its essential feature is the words same transaction. Checking a table and then writing in a separate transaction reintroduces the race: two concurrent deliveries both check, both find nothing, and both apply.

What the simulation shows

The scenario is a payment consumer: at-least-once delivery, a few percent of ordinary redeliveries, and one consumer restart that triggers a bulk redelivery of unacknowledged messages.

Run the default with no protection. The ledger panel shows money that does not exist — the actual balance against what was intended — and the gap grows steadily and then jumps at the restart. Nothing failed. Every message was processed successfully. The books are wrong.

Now switch to the in-memory seen-cache. The steady drift stops: ordinary duplicates are caught. Then the restart happens, and watch the drift jump anyway, because the cache went with the process and the redelivery burst arrived immediately after. The cache handled the easy case and failed the one that mattered.

Finally, the unique constraint. The drift is zero — through the ordinary duplicates and through the restart burst, because the check and the effect are one atomic act. The cost is visible in the latency panel and it is small: one indexed insert.

The numbers worth carrying

Duplicate rates in real systems are typically a fraction of a percent in steady state and much higher around consumer rebalances, restarts and deploys — which is to say, they cluster exactly when you are already having a bad day. Sizing an idempotency store from the steady-state rate underestimates the burst.

Retention is the parameter to decide deliberately: keys must be kept at least as long as the maximum possible redelivery delay. For a broker with a 7-day retention, that is 7 days of keys. At a million messages a day and 40 bytes a key, roughly 280 MB — cheap, and it needs to be an explicit decision rather than a table that grows forever or a TTL shorter than the redelivery window.

The cost per operation is one unique-index insert, a few milliseconds at most, on the write path only.

Where it breaks down

The effect must be in the same transactional domain as the key. If the effect is “charge a credit card” and the key lives in your database, the two cannot commit together, and you are back to a dual-write problem — see the outbox pattern. The standard resolution is to make the external call idempotent too, by passing your key through: Stripe’s Idempotency-Key header exists for precisely this, and it is the payment provider’s side of the same design.

Key choice. The key must identify the intent, not the delivery. A broker-assigned message id changes on redelivery in some systems and is useless; a hash of the payload collides for legitimately repeated actions (two identical $5 coffees are two purchases). The correct key is usually generated by the original client, at the moment of intent, and carried through unchanged.

Non-idempotent-by-nature operations. “Increment balance by 10” is not safe to repeat; “set balance to 60” is, and “apply transaction abc123” is. Reformulating operations as facts with identity rather than as deltas is often the cleanest fix, and it is what event-sourced systems get for free.

Concurrent duplicates. Two copies delivered simultaneously to two consumers both pass a check-then-act. Only the constraint — or a lock — is safe, and this is why “we check if it exists first” is the most common wrong answer here.

What people get wrong

“Our broker supports exactly-once.” Kafka’s exactly-once semantics are real and are scoped to Kafka-to-Kafka processing within a transaction: consume, process, produce, commit offsets, atomically. The moment your side effect is an external database or an HTTP call, you are back to at-least-once and the effect is your responsibility. Being able to state that scope precisely is a strong signal.

“We deduplicate in memory.” Until the restart. Which is when the duplicates come.

“We check before inserting.” Two concurrent deliveries, both check, both insert. The constraint is not an optimisation of the check; it is the only version that works.

“Retries are safe because our API is RESTful.” PUT and DELETE are idempotent by specification and by nothing else. Your PUT that appends to an audit log is not idempotent whatever the verb says.

In production

Stripe’s Idempotency-Key, AWS’s ClientToken on many mutating APIs, and Kafka’s producer enable.idempotence (which deduplicates producer retries at the partition level using a sequence number) are the three worth naming. On the consumer side, the pattern is the same everywhere: a processed_messages table with the key as primary key, written in the same transaction as the effect, plus a retention policy.

Store the response alongside the key as well. A duplicate should return the original result rather than an error, so the caller — who never learned whether the first attempt succeeded — gets a coherent answer instead of a conflict it has no way to interpret.

The follow-up questions

“Your payment service receives the same message twice. What happens?” — Constraint in the same transaction. Then say where the key comes from.

“Where does the idempotency key come from?” — The client, at the moment of intent, carried through every hop and every retry.

“How long do you keep the keys?” — At least the maximum redelivery window; give the storage arithmetic.

“Can the broker guarantee this for you?” — Only within its own transactional boundary. The moment an external effect is involved, it is yours.