1. Different rules from everything else

Requirements

A ledger is append-only and double-entry: every movement is two entries that sum to zero, and nothing is ever updated or deleted. Corrections are new entries, not edits. That constraint is not bureaucracy — it is what makes the system auditable and what makes reconstruction possible after any failure.

The balance is a derived value, not a stored one. Storing it invites the update-in-place bug where the balance and the entries disagree, and there is no way to tell which is right.

Ask what the consistency requirement really is. "Strong" is the reflex answer; the useful version is that a single account's balance must never be read as more than it is, which is a much narrower and cheaper requirement.

Append-only, double-entry, balance derived. No updates, ever.

2. Durable writes are expensive

Estimate

5,000 transactions a second at peak, each needing two entries and a durable commit. A durable write is bounded by fsync on the write-ahead log: a few thousand a second per primary with group commit, not the tens of thousands a cached read gets.

So this does not fit on one primary, which means partitioning — and the partition key is the question. By account is right. By time is catastrophic: the newest partition takes every write, which is the append-only hot shard problem.

5k durable writes/s needs partitioning. Partition by account, never by time.

3. The naive design and the money it invents

A first design

A payment service consuming from a queue and writing to the ledger. The queue is at-least-once, because every broker is, and the consumer has no duplicate protection.

Run it. A few percent of redeliveries plus one consumer restart, and the ledger contains money that nobody spent.

The consumer is redeployed. What happens to messages it had processed but not acknowledged?

They are redelivered and applied a second time — Processing and acknowledging are two separate steps, and a process can die between them. That is precisely what at-least-once means, and it is a guarantee rather than an edge case.

4. The drift

Watch it break

The number on screen is the difference between what the ledger says and what was actually intended. Nothing errored. No alert fired. The money exists, in the sense that the ledger says it does.

Note also the shape of it: a trickle of duplicates from ordinary redeliveries, and then a step at the restart, when everything unacknowledged comes back at once. The bulk of the damage happens in a moment that looks, from the outside, like a normal deploy.

Duplicates are a guarantee, not an edge case, and a restart delivers them in bulk.

5. Exactly-once effect, not exactly-once delivery

Find the bottleneck

Exactly-once delivery is not available. You cannot make a network stop losing acknowledgements, and any protocol claiming otherwise has moved the problem rather than solved it.

What is available is exactly-once EFFECT: let the message arrive as many times as it likes, and make the second application do nothing. That is a unique constraint on an idempotency key the producer chose, enforced in the same transaction as the entry.

An in-memory cache of recently-seen keys is not the same thing. It catches the ordinary redeliveries and loses everything at exactly the moment the redeliveries arrive in bulk — which is the restart.

Unique constraint in the same transaction as the effect. The drift goes to zero.

6. And the event you publish afterwards

Change one thing

The ledger entry is written. Now something downstream needs to know — the notification, the analytics, the partner webhook. That is a second write to a second system, and there is no transaction across the two.

The outbox pattern is the answer here for the same reason it was in the notification service: put the event row in the same transaction as the ledger entry, and let a relay publish it. Either both are there or neither is.

This is the piece that makes the whole thing composable. The ledger is exactly-once by constraint; the event is at-least-once by design; the consumer deduplicates. Every link in the chain has a stated guarantee, and none of them is "we hope".

Ledger entry and outbox row in one transaction. Everything downstream is retryable.

7. The partition question

What gets asked next

"A network partition splits your ledger cluster. What do you do?" Refuse writes on the minority side. This is the system where the CP answer is not a trade-off to be argued but the only defensible option: accepting writes on both sides means two balances for one account, and reconciling that means deciding whose money to delete.

Say what that costs, because an interviewer wants to hear that you know: a real share of users get an error for the duration of the partition. That is the correct outcome, and being able to state the cost plainly is what distinguishes a considered answer from a memorised one.

"How do you know the ledger is right?" Reconciliation, continuously: the sum of all entries for an account must equal its balance, and the sum of all entries everywhere must be zero. A double-entry ledger has an invariant you can check, and checking it on a schedule is how you find the bug you have not thought of.