Replication and partitioning — every question, written out
Leaders, followers and quorums; range, hash and directory partitioning; and the hot key that no partitioning scheme splits.
What does partitioning give you that replication does not?
Comparison
Write capacity and dataset size beyond one machine.
Replication puts the same data on several machines, which buys durability and read throughput and nothing else. Partitioning puts different data on different machines, which is the only way past one machine’s write capacity or disk. They are orthogonal and most systems need both.
What makes a partition key a good one?
Trade-off & selection
High cardinality, even access, and it appears in most queries.
A good key spreads both data and traffic, and lets most queries be answered from one partition. Cardinality bounds how many partitions you can ever have; access evenness decides whether they are used; query alignment decides whether reads fan out.
See it run — Per-shard load against the fair share.
One tenant generates half your traffic. Does adding shards help?
Edge case reasoning
No — a single key lives on one shard however many there are.
You cannot split one key by adding capacity — you have to change what the key is. A compound key such as tenant plus bucket spreads one tenant across shards, at the cost of queries needing to fan out across those buckets.
See it run — The imbalance stays put as shards are added.
You range-partition by timestamp. What happens to write load?
Code diagnosis
Every write goes to the newest partition; the rest are idle.
A monotonic partition key under range partitioning means the newest partition is the only one being written to, forever. This is the single most common partitioning mistake, and it is why time-series stores partition by a hash of the series and only then by time.
See it run — One shard taking every write.
Adding a shard under naive hash partitioning moves how much data?
Complexity derivation
Nearly all of it, because every key’s modulo changes.
Under key mod n, changing n changes the answer for almost every key. Consistent hashing exists precisely to make membership changes cost one node’s share instead, and a directory achieves the same by making placement an explicit decision.
See it run — Ring against modulo, on the same keyspace.
A hash ring with one token per node is stable but unbalanced. Why?
Invariant identification
The arcs are random, so some nodes own far more of the circle.
With n random points on a circle, the gaps between them vary considerably — the largest is typically several times the smallest. Giving each node a hundred or more virtual nodes averages many gaps together, and the law of large numbers does the rest.
See it run — Share per node at one token, then at 150.
You add read replicas to relieve the primary. What have you introduced?
Trade-off & selection
A window in which a reader does not see a write that has committed.
Asynchronous replication means a committed write has not reached the replicas yet. Normally that window is milliseconds; during a restart or a bulk load it is seconds — and it is widest exactly when traffic is heaviest.
See it run — Stale reads while one replica is behind.
Fully synchronous replication to three replicas. What is the cost?
Trade-off & selection
Every write waits for the slowest replica, so one slow node stalls writes.
Waiting for all replicas makes availability the product of theirs rather than the sum: any one being slow makes every write slow. Semi-synchronous — waiting for one replica of several — is the usual compromise, bounding data loss without coupling to the slowest.
A primary fails and a replica is promoted. What can be lost?
Edge case reasoning
Any write acknowledged by the primary but not yet replicated.
The replication lag at the instant of failure is the data loss window. It is why semi-synchronous replication exists, and why the honest question about any failover design is how many seconds of acknowledged writes you are prepared to lose.
Multi-leader replication across two regions. What does it add?
Trade-off & selection
Local write latency, and write conflicts that must be resolved.
Multi-leader buys a local write path — no cross-ocean round trip — and pays for it with concurrent writes to the same key in different regions. That is only tenable when the data is mergeable or the application can present a conflict to a user.
Three replicas, W=2, R=2. What does that configuration buy?
Complexity derivation
A read always overlaps a write, and either can survive one failure.
R + W > N is the overlap condition, and 2 + 2 > 3 satisfies it, so a read set always shares a replica with the latest write set. It is the balanced point: both operations tolerate one node being unavailable.
When is a directory-based partitioning scheme worth its extra component?
Comparison
When placement must be a decision rather than a consequence.
A directory makes placement explicit, so you can move a noisy tenant onto its own shard or keep European data in Europe. The cost is a lookup on the request path and a component that is now critical — which is why it is usually cached everywhere and changed rarely.
A query needs data from every partition. What has gone wrong?
Edge case reasoning
The partition key does not match how the data is queried.
A scatter-gather waits for the slowest partition, so its p99 is the partition p99 amplified by the fan-out. Choose the key so common queries hit one partition; where a second access pattern is genuinely needed, that is what a secondary index or a second denormalised copy is for.
See it run — Fan-out latency against a single-partition read.
Replication lag climbs steadily on one replica while the others are fine. What is the likely cause?
Code diagnosis
That replica cannot apply changes as fast as they arrive.
Steadily growing lag on one replica means apply throughput below arrival rate, and single-threaded apply is the usual reason — a replica replaying serially cannot keep up with a primary writing in parallel. Long-running queries blocking the apply thread are the other common cause.
See it run — One replica falling behind while the others hold.
You must reshard a live system. What is the hard part?
Edge case reasoning
Serving reads and writes correctly while data is in two places.
During a move a key exists on both the old and new shard, and every read and write has to reach the right one at the right moment. This is why systems that expect to reshard use many more logical partitions than physical nodes — moving a whole partition is far easier than splitting one.
Explain how you would choose a partition key for a multi-tenant analytics product.
Explain it plainly
I would start from the queries, because a key that does not appear in the common query turns every read into a scatter-gather. For multi-tenant analytics almost everything is scoped to a tenant, so tenant id is the obvious candidate and it aligns reads to one partition. The problem is skew: tenants are Zipfian, so the largest customer may be a hundred times the median, and tenant id alone gives them their own permanently hot shard. So I would use a compound key — tenant id plus a bucket derived from something like the day or a hash of the record id — with the bucket count varying by tenant size. Small tenants get one bucket and their queries stay single-partition; large tenants get many, so their load spreads, and their queries fan out across a known small number of buckets rather than across the whole cluster. I would also keep far more logical partitions than physical nodes, so rebalancing is moving whole partitions rather than splitting them. And I would monitor per-partition traffic rather than per-partition size, because a partition can be small and still be the hottest thing in the system.
The answer has to reconcile query alignment against tenant skew, which pull in opposite directions.
What does read-your-writes cost when implemented by routing to the leader?
Trade-off & selection
A share of reads goes back to the leader you were relieving.
Pinning a session to the leader for a window after it writes is correct and puts read load back where you were trying to remove it. Keep the window short, or carry the write position instead and let a replica serve the read once it has caught up.
See it run — The share of reads served by the leader.