Consensus and coordination — every question, written out
Quorums, leader election, distributed locks that cannot guarantee mutual exclusion, and what a clock cannot tell you.
Why does requiring a majority prevent two leaders?
Invariant identification
Two majorities of the same set must share at least one member.
Any two subsets of more than half a set intersect. That shared member voted at most once in the term, so at most one candidate can collect a majority — and the proof needs no communication between the two sides, which is what makes it hold during a partition.
See it run — One leader, whatever the split.
In Raft, what does "one vote per term" actually enforce?
Invariant identification
That the intersecting node cannot support two candidates at once.
The majority argument only works if the shared node’s vote cannot be counted twice. One vote per term is the line of code that makes the intersection meaningful, and it is why the term number appears on every message.
See it run — Vote messages and the term counter.
Why are consensus clusters built with odd node counts?
Complexity derivation
An even cluster tolerates no more failures and can split evenly.
Three and four both tolerate one failure, so the fourth node adds cost and no fault tolerance. Worse, an even cluster can split exactly in half, leaving neither side with a majority and the whole cluster unavailable.
See it run — Neither side leads.
What breaks in Raft if election timeouts are not randomised?
Invariant identification
Followers all campaign together, split the vote, and repeat forever.
Identical timeouts mean every follower becomes a candidate at the same instant. Each votes for itself, none reaches a majority, the term is wasted and the cycle repeats. Randomisation is not tuning — it is what makes the algorithm terminate.
See it run — Elections started against leaders elected.
How long is a Raft cluster unavailable for writes after its leader dies?
Trade-off & selection
About one election timeout, since nothing commits without a leader.
A follower waits out its election timeout, campaigns, and collects votes in a round trip. So the write outage is roughly the timeout plus one round trip — typically a few hundred milliseconds, and it is the number to quote when asked what failover costs.
See it run — The gap where no leader exists.
Why can a distributed lock with a lease not guarantee mutual exclusion?
Edge case reasoning
The holder can be paused past its lease and not know it.
A lease must expire or a crashed holder blocks the resource forever. So the guarantee becomes "one client believes it holds the lock" — and a stop-the-world garbage collection, a hypervisor migration or an overloaded machine breaks that belief while the process is none the wiser.
See it run — Clients that believe they hold the lock.
How does a fencing token fix what a longer lease cannot?
Invariant identification
The resource rejects any token below the highest it has accepted.
The lock hands out a monotonically increasing number and the client sends it with every write. Correctness moves from the lock to the resource, which is the only place it can live — a zombie holding token 33 is refused by a store that has already accepted 34.
See it run — Zombie writes rejected on their token.
A team doubles the lease duration after a lock-related incident. Is that a fix?
Code diagnosis
No — it only requires a longer pause, and pause length is not under your control.
Lengthening the lease moves the threshold a pause must exceed and cannot remove it, because you do not control how long a process can be descheduled. It also makes recovery from a genuine crash slower — worse on both axes.
A node stops sending heartbeats. What can you conclude?
Edge case reasoning
Only that you cannot reach it. The three causes look identical.
The impossibility of distinguishing a slow process from a dead one is the foundational result the whole field rests on. This is why quorum is the answer: it never requires knowing why a node is unreachable, only how many you can still reach.
Why is two-phase commit avoided across services?
Trade-off & selection
A coordinator failure after prepare leaves participants blocked holding locks.
After voting to prepare, a participant must hold its locks until it is told the outcome. If the coordinator dies in that window, it waits — and every participant’s availability now depends on every other participant and on the coordinator. That coupling is why sagas exist.
What does a saga give you instead of atomicity?
Comparison
Compensation: the steps really happened, and were really undone.
A saga runs forward and, on failure, runs compensating actions backwards. The flight was booked and then cancelled, visibly — that is not a rollback. Two properties follow: every compensation must be idempotent, and every one must eventually succeed.
See it run — Compensations running in reverse order.
How should the steps of a saga be ordered?
Trade-off & selection
Most likely to fail first, hardest to undo last.
A failure on the first step compensates nothing, so put the riskiest step there. And an irreversible step — sending an email, charging a card at a partner — should be last, because there is no compensation to run for it.
See it run — Compensation retries when the risky step is first.
A compensating action itself fails. What now?
Edge case reasoning
Retry until it succeeds, and escalate to a human if it does not.
Compensations must be retried until they succeed, which is why every one must be idempotent — the orchestrator cannot know whether the previous attempt took effect. When retries are exhausted the only honest answer is an alert and a human, and a design that never reaches that state is a design that has thought about it.
See it run — Sagas stuck with effects left behind.
Two events on different machines have timestamps 3ms apart. What can you conclude about their order?
Code diagnosis
Nothing — clock skew between machines routinely exceeds that.
Wall-clock timestamps from different machines are not a reliable ordering below the skew, which is a few milliseconds on a good day. Use logical clocks for causality, or a service with bounded and disclosed uncertainty that waits it out.
See it run — Concurrent writes resolved by comparing clocks.
A leader serves a read from local state without contacting followers. What is the risk?
Edge case reasoning
It may have been deposed already and not know it, returning stale data.
A partitioned leader continues believing it leads until it hears otherwise, and a stale local read violates linearizability. The fixes are a lease — serve local reads only within a bounded window since the last confirmed heartbeat — or a quorum read, which costs a round trip.
A colleague proposes using Redis as a distributed lock to prevent double-processing. Respond.
Explain it plainly
Redis will work as a lock most of the time, and I would want to be clear about what "most of the time" means before relying on it. The lock has to have a TTL, otherwise a client that crashes while holding it blocks the resource forever. But once it has a TTL, the guarantee is not "only one client holds the lock" — it is "only one client believes it does". If our worker is paused for longer than the lease, by a stop-the-world garbage collection or a hypervisor migration or just a very busy machine, the lease expires, another worker picks up the job, and then the first one wakes up and carries on as though nothing happened. Neither has crashed and both are writing. Making the lease longer does not fix that; it just requires a longer pause, and pause length is not something we control. So I would use the lock for what it is good at — reducing duplicate work, which is a performance concern — and put the correctness somewhere it can actually live. Concretely, a unique constraint on the job id in the database, or a conditional update that only applies if the record is still in the expected state. Then a double-processed job is rejected by the store rather than prevented by the lock, and it does not matter how long anybody was paused.
The answer has to separate the lock from the correctness guarantee, and land on where the guarantee has to live.