1. What is being cached decides everything

Requirements

Ask what the values are. Immutable content needs no invalidation and can be cached forever. Derived data needs a TTL. Anything that can be edited needs invalidation, which is the hard part and the reason "there are only two hard things in computer science" is a joke about this.

Ask whether a stale read is acceptable and for how long, because that number is the difference between a cache and a replica.

Invalidation is the design; storage is the easy part.

2. Memory and the hit ratio you need

Estimate

A terabyte of cache across a hundred nodes is ten gigabytes each, which is unremarkable. The question is what hit ratio that buys, and that depends entirely on the popularity distribution.

For Zipfian traffic, caching around one per cent of the keyspace delivers most of the achievable hit ratio. Getting from 90% to 99% costs far more than getting from 0% to 90% did — and the number that matters is not the hit ratio but its complement: at 99% the origin sees one request in a hundred, at 90% it sees ten times that.

Read the hit ratio as origin load. 99% to 90% is a tenfold increase in origin traffic.

3. Where do the keys live

A first design

The obvious answer is hash the key and take it modulo the node count. It distributes perfectly and it has a catastrophic property: change the node count and almost every key moves.

A hundred-node cache loses a node regularly. If that means the entire cache is invalidated, then every node failure is an origin outage.

A node in a hundred-node cluster dies. Under modulo hashing, how much of the cache is lost?

Nearly all of it — Every key's owner is its hash modulo the node count. Change the count from 100 to 99 and the modulo of almost every key changes. The data is still on disk somewhere; it is simply no longer where anyone will look for it.

4. The ring, side by side with the alternative

Watch it break

Both schemes are computed over the same keyspace at the same time. Watch what each does when a node is added and when one is removed. The ring moves about one node's share; modulo hashing moves nearly everything.

Then turn the virtual node count down to one and watch the ring's own weakness: the arcs are random, so a few nodes own far more of the circle than their share. Virtual nodes are what fix that, and the fact that production libraries default to a hundred or more is not an accident.

The ring moves 1/n of the keys. Virtual nodes make the shares even.

5. The other failure: everyone misses at once

Find the bottleneck

A hot key expires and every request for it misses simultaneously. Each one goes to the origin, which is sized for the small share of traffic that normally reaches it. It slows down under the load, so the queries take longer, so more pile up behind the same expired key.

This is the cache stampede, and it is the most common cache-related outage. The fix is request coalescing: the first miss takes the lock and fetches, everyone else waits for that one answer.

Coalesce identical misses. Thousands of database calls become one.

6. And what to evict

Change one thing

On ordinary traffic the eviction policy barely matters — LRU, LFU, CLOCK and random land within a few points of each other, and the size of the cache matters far more than the policy.

Then a nightly analytics job scans a large table. LRU, the default everywhere, collapses to a near-zero hit ratio, because a single pass evicts every useful entry in exactly the order it will next be needed. Run it and watch the lines separate.

The answer is scan resistance: LFU, or a segmented LRU, or simply not letting the batch job share the cache.

One scan destroys an LRU cache. Policy matters exactly when the workload is not ordinary.

7. The questions after that

What gets asked next

"How do you invalidate?" Either a TTL, which is simple and stale, or explicit invalidation, which is exact and easy to get wrong. Explicit invalidation across a hundred nodes is a fan-out with its own delivery guarantees, and if it is best-effort then so is your consistency.

"What about a node that is slow rather than dead?" Worse than dead, because the ring still routes to it and every request there waits out a timeout. Health checks and ejection matter more than failover.

"Would you replicate cache entries?" Usually not — a cache miss is cheap and a replica doubles the memory. The exception is the hot key, where losing one node loses the entry that half your traffic wants, and replicating just the hot set is a real technique.