Cache hit ratio

The problem it solves

A cache trades memory for latency and for load on whatever sits behind it. The design questions are always the same two: how big, and how much does that buy? Both are usually answered by feel, and both have real answers that are cheap to compute — because the shape of the answer is not a line, it is a curve, and the curve is steep at the start and nearly flat afterwards.

The underlying fact is about popularity, not about caches. Real key access follows a heavily skewed distribution — Zipf-like, where the k-th most popular item gets traffic proportional to 1/k. A small number of keys account for a large fraction of requests. Cache those and you have captured most of the value; cache the rest and you are buying memory for the long tail, one request at a time.

The mechanism

Under a Zipf distribution with exponent near 1, the cumulative share of traffic captured by the top m of N keys grows like ln(m) / ln(N). That logarithm is the whole story. Going from caching 10 keys to 100 buys the same increment as going from 100 to 1,000, and from 1,000 to 10,000 — each tenfold increase in memory buys the same fixed number of percentage points. Which means the first 1% of your keyspace does most of the work and the next 39% does comparatively little.

The simulation does not assert this. It sweeps nine cache sizes over a synthetic-but-realistic Zipfian workload, runs each to a steady hit ratio, and plots the measured points. The curve you see is observed behaviour on a real (simulated) request stream, not a formula redrawn.

What the simulation shows

Read the swept curve first. Caching 1% of the keyspace already delivers most of the hit ratio that caching 40% would. That is the number to carry into a design conversation, because it converts “we need a big cache” into “we need a small cache and a way to keep the popular keys in it”.

Then break it. Flatten the popularity distribution to uniform and the curve becomes a straight line through the origin: caching 1% of a uniform keyspace buys a 1% hit ratio, because there is no popularity to exploit. Caching works because traffic is skewed. If your access pattern is genuinely uniform — random UUID lookups with no repeat structure, a scan over a table — a cache buys you approximately nothing and you should spend the money elsewhere.

Finally, look at the origin-load panel rather than the hit-ratio panel. This is the reframing that changes decisions. A hit ratio of 95% sounds like a solved problem. Expressed as origin load, it means the database still sees one request in twenty — and at 90%, twice that. Going from 90% to 95% does not improve things by five percent; it halves the load on the database. Going from 99% to 99.5% halves it again. Hit ratio flatters; origin load tells the truth.

The numbers worth carrying

• Hit ratio 90% → origin load 10%. 95% → 5%. 99% → 1%. The interesting metric is the complement, and the complement halves at every step up the ladder.
• Cache 1% of a Zipfian keyspace and you are most of the way to what 40% delivers.
• Every tenfold increase in cache size buys roughly a constant number of hit-ratio points, not a proportional one.
• Latency arithmetic: at a 95% hit ratio with a 0.4 ms cache and a 25 ms origin, the mean is 0.95 × 0.4 + 0.05 × 25 ≈ 1.6 ms. But the p99 is 25 ms, because the p99 is a miss by definition. The latency histogram shows two humps, one per outcome, and averaging across them is how people convince themselves a cache fixed a tail it did not touch.

Where it breaks down

The working set moves. Zipf describes a snapshot. In reality the popular set churns — news, product launches, a viral item — and a cache that was perfectly sized yesterday spends today’s mornings cold. The relevant number is not the keyspace size but the size of the working set over your TTL.

Cold start. A cache that has just been deployed, restarted, or invalidated wholesale has a 0% hit ratio and passes 100% of traffic to an origin sized for 5%. That is the same shape of failure as cache stampede, and it is why cache restarts are dangerous operations that deserve a plan.

Consistency. Everything above is about hit ratio; none of it is about correctness. Once you cache, you own a second copy of the data and the question of how it becomes wrong — see write strategies — and the TTL you choose is a statement about how stale you are willing to be.

Small keyspaces, big values. The curve is drawn in entries. If entries vary by three orders of magnitude in size, entry count is the wrong axis and you should be sweeping bytes.

What people get wrong

“We got the hit ratio to 85%, that is good.” For a database sized to handle 15% of read traffic, yes. Otherwise it is a number without a denominator. Always convert to origin load.

“Cache everything.” The tail of a Zipf distribution is enormous and each of its keys will be read approximately once. Caching them costs memory, costs eviction pressure on the keys that matter, and buys almost nothing.

“The cache will handle the spike.” Only for keys already in it. A spike concentrated on a new item is 100% miss, and that is precisely the shape of a viral event.

“Mean latency improved 15×, so users are 15× happier.” The p99 barely moved. Look at the two-humped histogram.

In production

The layers stack, and each has its own curve: a CPU cache, a per-process in-memory cache, a shared Redis or Memcached tier, a CDN. Each absorbs some fraction of what the layer above misses, and the arithmetic composes — a 90% local hit ratio in front of a 90% shared hit ratio leaves 1% of traffic for the origin.

Two operational habits matter more than the sizing. Report hit_ratio and origin_qps on the same dashboard, so the conversation about a two-point improvement is a conversation about halving database load. And measure hit ratio per key class rather than globally: a global 92% frequently hides one class at 99% and another at 20%, and the 20% is where the incident will come from.

For which entries survive when the cache is full, see eviction policies; for what happens the instant a popular key expires, see cache stampede.

The follow-up questions

“What hit ratio do you need?” — Answer with origin load. “The database can serve 5,000 QPS and we expect 100,000, so we need 95%, and here is the cache size that gets us there.”

“How big should the cache be?” — Working set over your TTL, not keyspace. Then point at the curve and note that ten times the memory is not ten times the benefit.

“What happens when the cache restarts?” — 100% of traffic to the origin. Say how you avoid it: staged restarts, warming, request coalescing, or a second cache layer that does not restart at the same time.

“Your access pattern is uniform. Now what?” — Do not cache. Say so; it is a stronger answer than sizing something that cannot work.