Sagas and compensation

The problem it solves

A booking touches four services: reserve a flight, charge a card, reserve a hotel, confirm. Each owns its own database. There is no transaction across them, so there is no ROLLBACK — if the hotel step fails, the flight is already reserved and the card is already charged, and both of those are facts in other people’s systems.

A saga is the answer: a sequence of local transactions, each with a compensating action that semantically undoes it. When a step fails, the compensations for the completed steps run in reverse order. The flight is not un-booked; it is cancelled, which is a new, visible event in the real world, with a cancellation record, possibly a fee, and possibly an email to the customer.

That distinction is the whole conceptual content of the pattern. A saga is not a rollback. A rollback erases history. A saga adds to it.

The mechanism

Two coordination styles. Orchestration puts a coordinator in charge: it calls each step, tracks state, and triggers compensations on failure. State is explicit and debuggable, and the coordinator is a component you must build and keep available. Choreography has each service react to the previous one’s event with no central coordinator. Less machinery, and the flow exists only as an emergent property of the event graph — which is delightful at three steps and unmaintainable at eight, because no single artifact says what the process is.

Three properties every compensation must have.

Idempotent, because it will be retried. Commutative enough that arriving late is safe. And most importantly, it must eventually succeed — because there is nothing to compensate a compensation with. If the refund fails permanently, the saga is stuck in a state no code can resolve, and a human has to open the two systems and reconcile them by hand.

There is also a sequencing consequence that is easy to miss and cheap to exploit: order the steps so the most likely failure comes first. A step that fails before anything else has happened has nothing to compensate. Reordering is free at design time and saves compensation work forever.

What the simulation shows

Four steps, a forced failure, and compensations running backwards.

Run the default and watch the saga panel: the third step fails, so the second and first are undone in reverse order. The sequence view shows it as a real timeline — forward through the steps, then backwards through the compensations. Note that the flight is booked and then cancelled, visibly, in the outside world. A customer with a notification setting saw both.

Now the failure that defines the pattern’s limits. Make compensations fail 30% of the time and turn retries off. Watch the outcomes panel fill with sagas that nobody can resolve automatically — a charge with no booking, sitting in a state that requires a person. That counter is the reason compensations must be retried, and therefore the reason they must be idempotent.

Turn retries back on with the same failure rate and the stuck count collapses toward zero: the compensations eventually get through.

Finally, the design lesson. Move the failing step to the front and compare the compensation work. Identical failure rate, a fraction of the undo — because nothing had happened yet when it failed.

The numbers worth carrying

Expected compensation work is roughly failure_probability × steps_completed_before_failure. Putting the riskiest step first minimises the second factor directly, and it is the cheapest optimisation available in the whole pattern.

Stuck-saga rate is what to alert on. It should be very close to zero, and every instance is a manual reconciliation. Budget for that: a saga system needs an operations surface — list stuck sagas, inspect state, retry a compensation, mark resolved — and teams that skip building it discover they need it during an incident.

Latency is the sum of the steps, and a saga is not atomic in time: the system is in a partially-completed state for the whole duration, and anything reading during that window sees it. That must be acceptable, or the design is wrong.

Where it breaks down

No isolation. This is the property sagas give up, and it is the one people forget. Between step two and step three the money is gone and the hotel is not booked, and any query in that window sees exactly that. The standard countermeasures are semantic locks (mark the record “pending”), commutative updates, and re-reading values before acting — but there is no general fix, only per-case design.

Some steps cannot be compensated. An email has been sent. A message has been delivered. A physical package has shipped. For those, the design must ensure they come last, after everything that can fail has already succeeded — which is another argument for ordering by risk.

The coordinator needs durable state. An orchestrator that loses its state mid-saga leaves the saga stuck. Its state must be persisted transactionally with each step’s outcome — see the outbox pattern, which is how the coordinator publishes step commands without a dual write.

Compensation may arrive before the thing it compensates. In an asynchronous system, a cancel can overtake a booking. Compensations must handle “nothing to undo yet” — and ideally record the cancellation so the later booking sees it.

What people get wrong

“A saga is a distributed rollback.” It is a sequence of new transactions that semantically reverse earlier ones, and the difference is visible to users, to auditors, and to anyone reading a partially-completed state.

“We will just use two-phase commit.” 2PC blocks on coordinator failure — participants hold locks indefinitely — and it requires every participant to support XA, which most modern services do not. It buys atomicity at a real availability cost. Sagas trade isolation for availability, which is usually the right trade for long-running business processes.

“Compensations always work.” They fail, and the simulation counts what that costs. Retry them, make them idempotent, and build the manual escape hatch.

“Choreography is simpler.” For three steps. For eight, nobody can answer “what happens if step five fails” without reading five codebases.

In production

Temporal and AWS Step Functions are the mainstream orchestrators, and both persist workflow state durably so a coordinator crash resumes rather than strands. Camunda and Netflix Conductor occupy the same space. Choreographed sagas are what most event-driven microservice estates have by default, whether or not anyone called them that.

The operational advice that matters most: make the saga state queryable. “Show me every saga in a non-terminal state older than an hour” is the query that finds problems before customers do, and it is the first thing to build after the happy path.

The follow-up questions

“How do you do a transaction across three services?” — You do not; you do a saga. Then name compensations and their requirements.

“What if the compensation fails?” — Retry, idempotent, and eventually a human. Say the last part; it is the honest answer and it demonstrates operational experience.

“What does a saga give up compared to a transaction?” — Isolation. Partial states are visible. Say what a reader sees mid-saga.

“How would you order the steps?” — Riskiest first, irreversible last. This is the answer that shows design judgement rather than pattern recall.