Replication lag

The problem it solves

Reads outnumber writes in most systems by an order of magnitude or more, so the first scaling move everyone makes is to add read replicas and send reads to them. It works, it is cheap, and it introduces a bug that the design does not mention: a user changes their profile, the page reloads, and the change is not there.

Nothing failed. The write committed on the leader. The read succeeded on a follower. The follower simply had not received the change yet — it was thirty milliseconds behind, and the user’s next request arrived in twenty. Eventual consistency is a promise about the limit, not about the next request, and the next request is the one users notice.

The mechanism

A leader accepts writes and streams them to followers. The lag is the time between a write committing on the leader and being applied on a follower: network transit, plus the follower’s own apply time, plus any queueing if the follower is behind. Under healthy conditions it is small — single-digit to tens of milliseconds. Under a replica stall — a long query holding a lock, a vacuum, a backup, a network hiccup, a restart — it can reach seconds or minutes, and that is when the stale-read rate stops being a rounding error.

Four strategies:

Read from any follower. Maximum throughput, maximum staleness. Fine for data where being a few hundred milliseconds behind is invisible: a product catalogue, an article, an aggregate count.

Read-your-writes. After a session writes, route that session’s reads to the leader for a short window — long enough to cover typical lag. Only the writing session pays, and only briefly. The window is a guess and needs to exceed real lag, which is why it must be paired with monitoring.

Monotonic reads. Pin a session to one replica. Staleness remains, but it never goes backwards: a user cannot see a new value and then an older one, which is the failure that reads to users as data loss and generates support tickets far out of proportion to its frequency.

Leader-only reads. Correct, and it gives up the entire reason the followers exist. Every read now lands on the one machine you were trying to protect.

What the simulation shows

The default routes reads to followers and counts, directly, the reads that fail to show a user their own write. It is a real, measurable share under normal lag — and then a replica stalls, the shaded region opens, and the rate climbs sharply. The per-follower lag panel shows one line diverging while the others stay flat, which is exactly what this looks like on a real dashboard.

Now switch to read-your-writes. Stale reads disappear. The where-reads-go panel shows the cost: a small share of reads now land on the leader — the ones from sessions that recently wrote, which is a minority of a minority.

Compare that against leader-only, where the same panel shows every read on the leader. Both configurations have zero stale reads. Only one of them still scales, and putting the two panels side by side is the fastest way to see why read-your-writes is worth the plumbing.

Then monotonic reads: staleness persists, but the going-backwards failure does not. Deciding which of those two properties you need is the actual design work.

The numbers worth carrying

Healthy same-region replication lag is typically 1–50 ms. Cross-region is bounded below by the speed of light: a same-continent round trip is a few tens of milliseconds and a cross-continent one is around 150 ms, so a cross-region follower cannot be fresher than that no matter what you buy.

Human reaction time after a write — the click, the redirect, the render — is often 50–500 ms, which is the same order of magnitude as lag. That coincidence is why stale reads are common rather than rare: the user’s next request arrives inside the replication window by default.

Size the read-your-writes window from your p99 lag, not your mean, and alert on lag exceeding it. A 500 ms window against a p99 lag of 2 seconds is a config that appears to work and does not.

Where it breaks down

Sticky sessions are not enough. Read-your-writes needs to know that this session wrote recently, which means a token, a cookie, or a timestamp travelling with the session — not merely affinity to a server.

Cross-entity causality. You write a comment and then read the post’s comment count. Read-your-writes on the comment does not cover the count if it is computed elsewhere. Genuinely causal consistency needs the write position to travel with the read, which is the same mechanism as CQRS projection lag.

Lag measured wrong. Byte-position lag on a replication stream is not time lag, and a replica that has received everything but is applying slowly reports zero on some metrics and seconds on others. Measure by writing a heartbeat row on the leader and reading its age on the follower.

Failover. Promoting a follower that is behind loses the writes it had not received. This is the durability side of the same number, and it is the reason semi-synchronous replication — wait for at least one follower before acknowledging — exists.

What people get wrong

“Eventually consistent means a few seconds.” It means no bound at all, unless your system provides one and you monitor it. Most give you a metric, not a guarantee.

“Stale reads are rare.” They occur at roughly the rate at which users read shortly after writing — and users read shortly after writing constantly, because that is what a form submission and redirect is.

“We use sticky sessions, so we are fine.” Sticky to a follower means consistently stale. Monotonic, not read-your-writes.

“Add more replicas to reduce lag.” More replicas is more load on the leader’s replication stream, not less lag. It helps read throughput and does nothing for freshness.

In production

Postgres exposes pg_stat_replication and lets you request synchronous_commit per transaction, which is a per-write choice between latency and durability — a good pattern to name, because it means the strong guarantee can be reserved for the writes that need it. MySQL has semi-synchronous replication. AWS Aurora replicas are typically tens of milliseconds behind; DynamoDB offers strongly consistent reads at double the read-capacity cost and only within a region.

The application-level pattern that generalises best is a write position token: the write returns an LSN, a version, or a timestamp; the client carries it; the read either waits for the replica to reach it or routes to the leader. That is read-your-writes without a guessed window, and it is the same mechanism recommended on the CQRS page.

The follow-up questions

“You add read replicas. What breaks?” — Read-your-writes. Say it before being asked; it is the follow-up every interviewer has queued up.

“How do you fix it without sending everything to the leader?” — Route only recently-writing sessions, for a bounded window, sized from p99 lag. Or carry a position token.

“A replica falls 30 seconds behind. What do users see?” — Their own writes missing, values apparently going backwards if reads move between replicas, and any read-your-writes window shorter than 30 seconds silently failing.

“What happens on failover?” — You lose the writes the promoted replica had not received. Then discuss semi-synchronous replication as the trade.