1. What is the limit protecting

Requirements

A rate limiter exists to protect something. Ask what: the origin, a downstream partner, a billing tier, or fairness between tenants. The answer changes whether you want a hard cap, a burst allowance, or a fair-share scheduler.

Ask also whether the limit is per-key or global, whether exceeding it should reject or delay, and — the question people forget — what happens when the limiter itself is unavailable. Fail open and you have no limiter; fail closed and the limiter is now a single point of failure for the whole API.

A limiter that fails closed is a new single point of failure.

2. The limiter is on the hot path

Estimate

Every request checks the limit, so the limiter takes the full 10,000 a second — more, because rejected requests are checked too, and clients that are being limited tend to retry.

If the counter lives in a shared store, that is 10,000 round trips a second added to every request's latency. A same-datacentre round trip is about half a millisecond, so the limiter adds 0.5ms to a p50 and rather more to a p99.

The limiter sees more traffic than the service it protects.

3. The obvious implementation

A first design

A counter per key, reset every second. Increment on each request; reject above the limit. It is four lines of code and it is what most people write first.

It is also wrong in a specific, measurable way, and the way it is wrong is invisible from inside the limiter.

A burst arrives straddling a window boundary. How much gets through in that one second?

Up to 2x the limit — The end of one window can be emptied and the start of the next can be emptied immediately afterwards. Both windows are within their limit; the second that spans them is not.

4. Twice the limit, and the counter never went over

Watch it break

The chart to watch is the rolling one-second count, not the limiter's own counter. The limiter's counter stays inside its limit the entire time. The actual number of requests admitted in a one-second window goes to nearly twice it.

This is why the bug survives: every metric the limiter emits says it is working. The only way to see it is to measure what got through, from outside.

A fixed window admits up to 2x its limit across a boundary.

5. Three better options, and what each costs

Find the bottleneck

A sliding window counter weights the previous window by how much of it is still in view. It is an approximation, it is close enough, and it costs one extra number per key.

A token bucket gives you burst tolerance deliberately rather than accidentally: allowance accrues while you are quiet and is spent on the burst, then settles to the refill rate. This is usually what you want for a public API, because clients are bursty and punishing that is unfriendly.

A leaky bucket smooths output to a fixed rate, which is what you want when the thing you are protecting cannot absorb bursts at all.

Token bucket for public APIs; leaky bucket when the downstream cannot burst.

6. And the distributed part

Change one thing

Everything above assumed one counter. With many limiter instances, either they share a store — adding a network round trip to every request — or each holds a local share of the limit, which is cheaper and admits up to the instance count times the limit in the worst case.

The usual answer is local buckets with periodic reconciliation: each instance takes a slice of the budget, reports usage, and adjusts. It is approximate, and approximate is fine, because the limit was a business decision with a round number in it rather than a physical constraint.

The failure to plan for is the shared store going away. Fail open for a public API — a brief period of unlimited traffic is survivable, and an outage of your entire API because the limiter is down is not.

Local buckets plus reconciliation; fail open.

7. The follow-up that catches people

What gets asked next

"Your limiter starts rejecting. What do the clients do?" They retry. Immediately, all of them, because that is what a client library does by default. Now the limiter is taking several times the traffic, and if it shares infrastructure with anything else, so is that.

Return a Retry-After header, and mean it. Reject cheaply — a rejection that costs as much as a success has not protected anything. And put a retry budget on your own clients, because backoff configured by somebody else is not a control you have.