Split brain

The problem it solves

A cluster with a leader loses its network in the middle. Both halves are alive, both can see clients, and neither can see the other. Each half observes exactly what it would observe if the other half had died.

If your failover rule is “a node that cannot reach its peers promotes itself”, both halves promote. Now there are two leaders, both accepting writes, both correct by their own reasoning, and the two datasets diverge for as long as the partition lasts. When the network heals you have two authoritative histories of the same data and no automatic way to reconcile them — which makes split brain the failure that turns an availability incident into a data-loss incident.

The mechanism

The fix is a quorum: a node may lead only if it can reach a majority of the configured cluster. It is a fix because of one property of sets — any two majorities of the same set share at least one member — and that shared member cannot simultaneously be part of two different majorities on two sides of a partition. So at most one side can have one. The minority side, correctly, refuses to serve writes.

The property has an immediate corollary. Split a cluster of four into two and two, and neither half is a majority of four. Both refuse. Nothing diverges and nothing is writable — which is why quorum systems are built with odd node counts: an odd cluster cannot split evenly, so a partition always leaves exactly one side able to proceed.

Then there is the failure that actually happens in production. Quorum is computed against the configured cluster size, not the real one. Add two nodes to a three-node cluster and forget to update the configuration, and a node that can reach one peer computes “2 of 3 — I have a majority” while the actual cluster has five members. Now the minority of the real cluster is a majority of the remembered one, and it promotes. The quorum rule is intact and correct; the input to it is stale, and stale configuration defeats it entirely.

What the simulation shows

The leaders panel shows the number that must never exceed one.

Run with the quorum requirement off and watch the partition produce exactly the failure: two leaders, both accepting, two divergent datasets. Every write succeeds. Every client is served. The system is broken and nothing reports an error.

Now turn quorum on. Same partition, same timing. One leader — the majority side. The minority side refuses writes, and the refused-writes counter is the price. Both numbers on screen at once is the argument: you can have one leader, or you can have full availability, and during a partition you cannot have both. That is CAP, demonstrated with the specific mechanism.

Then split a four-node cluster evenly: zero leaders, everything refused, on both sides. Correct, safe, and completely unavailable — the argument for odd node counts, made concretely.

Finally, the realistic one. Quorum on, but the configured size still says three while the cluster has five. Two leaders again, with the safety mechanism enabled and believed in. This is what the incident report looks like.

The numbers worth carrying

Quorum is floor(n/2) + 1: 3 → 2, 4 → 3, 5 → 3, 7 → 4. Note that four and five both require three, so four tolerates only one failure while five tolerates two. Even counts are strictly worse than the odd number below them.

Detection time matters as much as the rule. A node must decide its peers are unreachable before acting, and that decision takes a failure-detection window — typically several missed heartbeats. Shorter windows fail over faster and produce more false positives; longer windows mean longer unavailability. This is the same trade as the election timeout on the Raft election page.

Where it breaks down

Quorum protects the cluster’s own decisions, not external resources. A node that has lost quorum and knows it will stop leading — but a request it issued before it noticed may still be in flight and land on a database afterwards. Bounding that requires either a lease with a bounded clock error or, better, a fencing token at the resource. See distributed locks.

Asymmetric and partial partitions. Node A can reach B, B can reach C, A cannot reach C. Majority arithmetic assumes a clean cut, and real networks provide much stranger topologies — including the one-way partition, where heartbeats flow in one direction only and the failure detector on each side reaches a different conclusion.

Witness and arbiter nodes. A two-datacentre deployment cannot have a majority in both, and putting the tie-breaker in one of them means that datacentre’s failure is a total outage. The standard answer is a lightweight third-site witness that participates in voting but holds no data.

Automatic failover is itself a risk. Some mature systems deliberately require human confirmation, because a false failover costs more than a few minutes of downtime. That is a legitimate engineering position and worth naming.

What people get wrong

“We have a quorum, so we are safe.” A quorum of what? The number in the configuration must equal the number of nodes in reality, and keeping those in step across scaling events is the actual operational work.

“Heartbeats detect the failure.” Heartbeats cannot distinguish a dead node from an unreachable one. That is not an implementation gap, it is the fundamental limitation the quorum rule exists to work around.

“Two datacentres, one leader each, active-active.” That is split brain as an architecture. Either one is primary, or you accept divergence and design a merge — see vector clocks.

“We will reconcile afterwards.” With what rule? Last-write-wins discards a user’s acknowledged write. If the answer is not a merge function, there is no answer.

In production

MongoDB’s replica-set elections require a majority and support arbiter members for the odd-count problem. Elasticsearch had exactly this failure historically — its minimum_master_nodes setting was the manual quorum configuration, and getting it wrong was the most common cause of split-brain data loss; version 7 made it automatic precisely because operators kept getting it wrong, which is the strongest possible argument that stale configuration is the real-world failure mode.

Traditional HA clusters (Pacemaker, and storage clusters generally) use STONITH — Shoot The Other Node In The Head — where the surviving side physically fences the other via a power controller or storage fence. It is crude and it is the only mechanism that gives a hard guarantee, because it does not require the other node to cooperate or even to be running correct software.

The follow-up questions

“Two datacentres, the link fails. What happens?” — With a majority in one, that one continues. With an even split, nobody does. If your answer is “both continue”, say what you will do about divergence.

“Why odd numbers?” — Even clusters can split evenly, leaving nobody with a majority; and n even tolerates the same failures as n − 1.

“How does quorum prevent two leaders?” — Two majorities intersect. One sentence.

“You added nodes last week. What else had to change?” — The configured cluster size, everywhere it is recorded. This is the answer that shows operational experience.