CQRS projection lag

The problem it solves

Reads and writes want different shapes. Writes want normalised tables, constraints and transactions; reads want denormalised documents, precomputed aggregates and whatever index makes the query fast. CQRS separates them: commands go to a write model, a projector consumes the resulting events and maintains a read model, and queries hit the read model.

The benefits are real — each side is optimised and scaled independently, and the read side can be a search index, a cache, or three different stores for three different views. The cost is one sentence long and is usually left out of the proposal: there is now a window in which a user cannot see their own write.

Worse, the window widens exactly when the system is busiest, because the projector is competing for the same resources as everything else. The failure is smallest in development, where you tested it, and largest in production during peak.

The mechanism

A write commits to the write model and emits an event. A projector reads the event stream and applies it to the read model. Between those two moments, a query returns the old state. That interval is projection lag, and it is the sum of event publication, projector queueing, and projection work.

Four responses:

Do nothing. Correct for data where a few hundred milliseconds of staleness is invisible — a dashboard, a search result, an aggregate count.

Read-your-writes via a position token. The write returns the stream position it produced. The client carries it on the next read. The read side either waits for the projection to reach that position or falls back. This converts staleness — an unbounded correctness problem — into latency, which is bounded and measurable. It is the best answer available and it is real plumbing: the token must survive the round trip through your API and your client.

Read from the write model after a write. Correct, and it puts load straight back on the store you split CQRS to protect. Acceptable in small doses for the minority of reads that immediately follow a write.

Optimistic rendering. Patch the change into the UI without waiting. Free, and it lies: when a write is rejected downstream, the user watches their change appear and then disappear, which is worse than having waited.

What the simulation shows

The default has no mitigation. Read the panel counting reads that do not show the user their own write: a real, steady share under normal lag. Then the projector stalls — a redeploy, a slow batch, a backlog — the shaded window opens, and the rate climbs sharply while the lag panel shows seconds rather than milliseconds. That combination, a normally-fine rate that explodes during a routine operational event, is exactly why this bites in production and not in testing.

Now turn on the position token. Stale reads go to zero — including through the stall — and the cost moves to the latency panel: post-write reads now wait for the projection to catch up. During the stall those waits are long, and they are bounded by the maximum-wait setting, after which the read falls back rather than hanging. Staleness became latency, and latency has a ceiling.

Compare with reading the write model: also zero stale reads, and the load lands on the store CQRS exists to protect. And optimistic rendering, where the stale-read counter looks wonderful and a separate counter tracks the writes that were rejected after the UI had already shown them as done — the lie, quantified.

The numbers worth carrying

Healthy projection lag is tens of milliseconds. During a projector deploy, a backfill, or a burst, it is seconds — and the ratio between those two numbers is what your read-your-writes window must cover.

Human round-trip time after a write is 50–500 ms: the click, the redirect, the render. That is the same order of magnitude as lag, which is why stale reads are common rather than rare — the same coincidence that makes replication lag visible to users.

Projector throughput must exceed peak event rate with headroom, and catch-up is the number that matters: if the projector runs at 400 events a second and 3,000 events accumulated during a stall, catching up takes 7.5 seconds after the stall ends, during which lag is still elevated. Recovery is always longer than the incident.

Where it breaks down

The window must be honest. A read-your-writes window shorter than actual p99 lag silently fails, and it fails only under load, which is the worst possible failure schedule. Waiting for a position rather than guessing a duration avoids this entirely, which is why the token is worth the plumbing.

Cross-aggregate reads. A token covers one stream. A page assembled from three projections needs three positions, or a global position, and a global position couples what CQRS was separating.

Projector bugs need rebuilds. The upside of event sourcing is that a broken projection can be rebuilt from the event log. The downside is that rebuilding takes as long as it takes, and the read model is stale or absent for the duration. Rebuild into a shadow table and swap.

Ordering. Projections applied out of order corrupt the read model — see queues and ordering. Partition by aggregate id, and make projections idempotent so replay is safe.

What people get wrong

“CQRS means event sourcing.” They are separate. CQRS is a read/write split and can be implemented with a materialised view over the same database. Event sourcing is storing state as an event log. They combine well and are often confused, and separating them out loud is a clear signal.

“The lag is milliseconds, so it does not matter.” It is milliseconds until a deploy, a backfill, or a burst. Design for the p99, not the median.

“We will use CQRS for consistency.” It is the opposite: it introduces an inconsistency window you did not previously have, in exchange for read scalability and shape.

“Optimistic UI solves it.” It hides it, until the write fails, and then it is the worst experience of the four.

In production

Elasticsearch fed from a database is CQRS whether or not anyone calls it that, and its refresh interval — one second by default — is the projection lag, configurable and frequently the cause of “I just saved it and search cannot find it”. Materialised views in Postgres are the same pattern with the same window. Kafka Streams and Flink build read models from event streams at scale, and their consumer lag metric is the lag panel above.

The most valuable operational habit is to expose the projection position in both directions: return it on writes, accept it on reads, and publish it as a metric. Once the position is a first-class value, read-your-writes is a small amount of code rather than a guessed timeout.

The follow-up questions

“A user updates their profile and reloads. What do they see?” — Stale, unless you carry a position. Say the mechanism.

“How stale can the read model get?” — Tens of milliseconds normally; seconds during a stall; and give the catch-up arithmetic.

“How do you rebuild a projection?” — Replay from the log into a shadow table, then swap. Say how long it takes and what serves reads meanwhile.

“Is CQRS worth it here?” — Frequently no. The mature answer names the read pattern that justifies it, and otherwise proposes a read replica or a cache, which cost far less.