CAP in practice

The problem it solves

“Pick two of three” is the version of CAP everyone learns and it is wrong in a way that leads to bad designs. You do not pick partition tolerance. A partition is a network event — a switch fails, a link saturates, a rack loses connectivity, a cloud availability zone becomes unreachable — and it happens whether or not you selected it in a design review.

The correct statement is narrower and more useful: during a partition, you must choose between consistency and availability. Refuse writes on the minority side and stay consistent, or accept writes on both sides and diverge. There is no third option, and the choice is forced by the network rather than made by you.

The complement is equally important. Eric Brewer’s PACELC extends it: if Partitioned, choose Availability or Consistency; Else, choose Latency or Consistency. The else branch is the one that governs your system 99.9% of the time, because partitions are rare and the latency cost of a quorum on every write is permanent.

The mechanism

Consider five nodes cut into a side of three and a side of two, with clients on both sides.

CP. Writes require a quorum — a majority. The side of three has one and keeps serving. The side of two does not and refuses every write. Nothing diverges; a real share of users get errors for the duration of the partition. When the network heals, there is nothing to reconcile, because nothing conflicting was ever accepted.

AP. Both sides accept everything. Every user is served. Keys written on both sides now hold two different values, and when the partition heals, somebody has to decide which is right. That somebody is either your application or a default policy, and the default policy is usually worse than you think.

The reconciliation strategy is a decision inside the AP choice, and the simulation counts the cost of each. Last-write-wins compares timestamps and keeps the later. It is simple, it is the default in many systems, and it is not reconciliation — it is deletion. One user’s update is discarded, silently, and the only trace is the number on screen. Keep-both preserves siblings and asks the application to merge, which is correct and requires the application to have a merge function, which most applications do not.

What the simulation shows

The partition is a control rather than an accident, and both sides keep taking traffic.

Run CP and watch the refused-writes counter climb for the whole partition window. That counter is your availability cost, quantified. Divergence stays at zero throughout.

Now run AP. Refusals go to zero — every user is served — and the divergence counter climbs instead. Same partition, same traffic, opposite consequence. Neither is free; the panel puts a number on both, which is the entire pedagogical point of the page.

Then look at what reconciliation costs. With last-write-wins, the panel counts writes discarded at merge time — updates that a user made successfully, that the system acknowledged, and that no longer exist. Switch to keep-both and the discards go to zero, replaced by siblings that the application must handle.

Finally, split a four-node cluster evenly. Neither side has a majority of four. Both refuse every write. Zero availability, on both sides, from a partition that a five-node cluster would have survived on one side — which is why quorum systems are built with odd node counts.

The numbers worth carrying

A quorum of n nodes is floor(n/2) + 1. Five nodes tolerate two failures; four nodes also tolerate only one, which is why four is strictly worse than three for the same reason it is worse than five: you pay for a node that buys no additional fault tolerance.

Two majorities of the same set always intersect. That single fact is why quorums prevent split brain, and it is the sentence to say out loud — see split brain for the mechanism.

The PACELC latency term: a cross-region quorum write costs at least one cross-region round trip, so 100–200 ms is the floor for a globally consistent write and no engineering removes it. That number, not partition tolerance, is why most systems are AP in practice.

Where it breaks down

“CP” and “AP” are per-operation, not per-system. The same database can serve a strongly consistent read of an account balance and an eventually consistent read of a product description. Real systems mix, and the mature version of this answer names which data gets which guarantee, rather than labelling the whole architecture.

Partial partitions are worse than clean ones. A node that can reach half the cluster, or a link that drops 40% of packets, produces states that neither the CP nor the AP analysis models cleanly. Real incident reports are full of these.

“Consistency” in CAP is linearizability, which is stronger than what most databases mean by the word. A system advertising “strong consistency” may mean snapshot isolation, or read-your-writes, or something else. Ask what is actually guaranteed.

Availability in CAP means every non-failing node responds, which is stricter than a real availability target. A system that is 99.9% available in the operational sense is “not available” in CAP’s sense.

What people get wrong

“We chose CA.” There is no such choice; a partition will occur and the system will do something. Saying CA means the behaviour during a partition is undefined, which usually means it is AP with no reconciliation strategy — the worst combination available.

“NoSQL is AP, SQL is CP.” Cassandra’s consistency level is per-query and spans the range. MongoDB is CP by default and tunable. Postgres with synchronous replication is CP; with asynchronous replication it is AP-ish, and it is the same database.

“Last-write-wins is a reasonable default.” It is a data-loss policy with a friendly name, and it depends on clocks agreeing across machines that do not — see vector clocks.

“Partitions are rare, so this is theoretical.” They are rare per link and constant per fleet. Any system with hundreds of nodes has one somewhere most weeks, and the reason it is not an incident is that somebody made this choice deliberately.

In production

DynamoDB, Cassandra and Riak descend from the Dynamo paper and default toward AP with tunable quorums — Cassandra’s LOCAL_QUORUM versus ONE is precisely the PACELC else-branch, chosen per query. Etcd, ZooKeeper, Consul and anything built on Raft or Paxos are CP by construction, which is why they are used for configuration and leader election and not for the main data path. Spanner is famously “CP with very high availability”, using synchronised clocks and redundant networking to make partitions rare enough that the availability cost is small — an engineering answer to a theoretical constraint rather than an escape from it.

The follow-up questions

“CP or AP?” — For which data? Give a per-dataset answer: balances CP, session state AP, catalogue AP with a TTL.

“State CAP correctly.” — During a partition, choose consistency or availability. Partition tolerance is not optional. Then add PACELC, because the else branch is what you live with daily.

“You chose AP. Two conflicting writes. What now?” — A merge function, or last-write-wins with the loss acknowledged. Naming CRDTs or sibling resolution here is a strong signal.

“Why five nodes and not four?” — Same fault tolerance, higher cost, and an even split leaves nobody with a majority.