Caching strategies — every question, written out
Hit ratio as a curve, invalidation as the real problem, and the three write strategies with what each one loses.
Doubling a cache from 1% to 2% of a Zipfian keyspace does what to the hit ratio?
Complexity derivation
Raises it by a few points — the curve is already flattening.
Popularity is Zipfian, so the hundredth most popular key gets a hundredth of the traffic of the first. The first slice of cache buys an enormous share of requests and each subsequent slice buys less — which is why "just add memory" stops working long before the hit ratio does.
See it run — Hit ratio plotted against cache size.
Hit ratio falls from 99% to 98%. What happens to origin traffic?
Code diagnosis
It doubles — the origin sees the miss rate, which went from 1% to 2%.
Always read a hit ratio as its complement. 99% to 98% is one miss in a hundred becoming two — a doubling of everything downstream. This is why a small regression in hit ratio is a large incident, and why the metric to alert on is origin load.
See it run — Origin requests per second as the cache shrinks.
Your cache holds 1% of the keyspace and the hit ratio is 1%. What does that tell you?
Edge case reasoning
Access is close to uniform, so caching cannot help much.
A hit ratio equal to the fraction of the keyspace cached is the signature of uniform access. The cache is working perfectly and there is no skew to exploit — which means the answer is to change the access pattern or drop the cache, not to buy memory.
See it run — The curve becomes a straight line.
Hit ratio collapses every night at 2am and recovers by 6am. What is likely happening?
Code diagnosis
A batch job scans a large table and evicts the working set.
One pass over a table larger than the cache evicts every useful entry, in exactly the order it will next be needed. LRU is the worst possible policy for this and the default nearly everywhere — which is why scan resistance is a feature databases advertise.
See it run — LRU against LFU while the scan runs.
On ordinary Zipfian traffic, how much does the eviction policy matter?
Comparison
Very little — all the common policies land within a few points.
On ordinary traffic LRU, LFU, FIFO, CLOCK and random cluster within about ten points, and size matters far more than policy. Policy becomes decisive exactly when the workload stops being ordinary — a scan, or a looping working set slightly larger than the cache.
See it run — The bars sit close together.
A hot key’s TTL expires under heavy traffic. What does the origin see?
Code diagnosis
Thousands of identical queries for the same row at once.
Every request arriving between the expiry and the first successful repopulation is its own miss. The size of the herd is the request rate times the fetch duration, which is why expensive queries — exactly the ones worth caching — produce the largest stampedes.
See it run — Queries per second at each expiry.
What does request coalescing guarantee?
Invariant identification
At most one in-flight fetch per key, however many clients want it.
The first miss takes a lock and fetches; everyone else waits for that result. Thousands of database calls become one, and the mechanism is a few lines of code in any real client library — usually called single-flight.
See it run — The origin spike disappears.
Coalescing fixes one hot key. What problem does TTL jitter fix that coalescing does not?
Edge case reasoning
A whole warmed keyspace expiring in the same second.
A cache warmed by a deploy has every entry created within the same second, so every entry expires within the same second. Coalescing still allows one fetch per key, and a thousand keys is a thousand fetches at once. Jitter spreads them over a window.
See it run — The expiry cliff flattens.
What does write-back buy, and what does it cost?
Comparison
Memory-speed writes, at the price of everything since the last flush.
Write-back acknowledges from memory and flushes later, so writes are hundreds of times faster and a crash loses everything buffered. The exposure is bounded by the flush interval, which makes it a dial rather than a gamble — the question is what number you can defend.
See it run — Unflushed writes at the moment of the crash.
Write-around invalidates rather than updating the cache. Why would you choose it?
Trade-off & selection
Because written data is often not read soon, so populating wastes the cache.
Write-through fills the cache with everything written, whether or not anyone reads it, evicting things people are actually reading. Write-around is right for write-heavy data with a low read-after-write rate — and its cost is that the next reader takes a miss.
See it run — Read latency compared with write-through.
TTL expiry or explicit invalidation?
Trade-off & selection
TTL is simple and bounded-stale; explicit is exact and easy to miss a path.
A TTL gives a bounded staleness with no coordination. Explicit invalidation gives exactness and requires every write path to remember — and across a hundred cache nodes it is itself a fan-out with its own delivery guarantees. Most systems use both: explicit for the common paths, a TTL as the backstop.
Requests for keys that do not exist are hammering your origin. What do you do?
Edge case reasoning
Cache the absence too, with a short TTL, or put a Bloom filter in front.
A cache that only stores hits lets every miss through, so a workload of nonexistent keys bypasses it entirely. Negative caching with a short TTL fixes it; a Bloom filter fixes it with far less memory, at the cost of a small false-positive rate that costs one wasted lookup.
See it run — Lookups removed before they reach the store.
One of ten cache nodes dies. What is the effect on the origin?
Edge case reasoning
A step increase in load as that node’s share becomes misses.
With consistent hashing, one node in ten is roughly a tenth of the keyspace becoming misses instantly. If the origin was sized for a 5% miss rate, it now sees three times that. Size for node loss, and warm a replacement rather than letting the ring rebalance onto cold nodes.
See it run — The share of keys that move when a node leaves.
You deploy a change that restarts every cache node at once. What happens?
Edge case reasoning
The origin takes full traffic until the cache refills, which it is not sized for.
A cold cache means the origin briefly sees the traffic the cache was built to absorb — often ten to twenty times its normal load. Restart nodes in waves, and consider warming a node before it takes traffic.
An interviewer asks: "How much cache do we need?" Explain how you would answer.
Explain it plainly
I would work backwards from the origin rather than forwards from a hit-ratio target. Start with what the origin can serve — say a database that comfortably handles three thousand reads a second at the utilisation we want. If we are offering a hundred thousand reads a second, then the origin can take at most three per cent of them, so we need a hit ratio of ninety-seven per cent or better. Then I would work out what cache size buys that, and the honest answer is that it depends on the popularity distribution and I would measure it rather than guess — but for typical Zipfian traffic, something in the low single-digit percentage of the keyspace usually gets into the nineties, and the last few points cost disproportionately more. Two things I would add. First, size for a node failure: if we lose a tenth of the cache, our miss rate roughly doubles, so the origin has to survive that. Second, the number to alert on is origin requests per second, not hit ratio — because a one-point drop in hit ratio at ninety-nine per cent is a doubling downstream, and a percentage point looks like nothing on a dashboard.
The answer has to invert the question: hit ratio is chosen from what the origin can take, not from a target.
Client, CDN, application or database cache — what decides which?
Comparison
How far you can move the answer toward the user without it going stale.
Every layer closer to the user saves more work and makes invalidation harder — a browser cache cannot be invalidated at all. The rule is to cache as far out as the freshness requirement allows, and to be able to state what that requirement is.