1. The unusual property of this system

Requirements

A metrics pipeline is one of the few systems where dropping data is not a failure but a design choice. A monitoring system that falls over under load is worse than useless: it goes down exactly when you need it, and it takes the ability to diagnose the outage with it.

Ask what the queries look like, because they decide the storage. Dashboards read recent data at low resolution over many series; alerts read the last few minutes of a few series constantly; investigations read one series at full resolution over hours.

Dropping is a correct answer here. Falling over is not.

2. Points, and how small they can be

Estimate

10,000 servers times 500 metrics is 5 million points every 10 seconds — 500,000 points a second, continuously.

A raw point is a series id, a timestamp and a value: 16 bytes or so. That is 8 MB a second, 250 TB a year. Compressed with delta-of-delta timestamps and XOR-encoded values, a point is closer to 2 bytes, which is 30 TB a year. That compression ratio is the reason storing metrics is affordable at all, and quoting the raw number is the most common mistake in this question.

500k points/s. 2 bytes each compressed, 16 raw. The ratio is the whole story.

3. Ingest, queue, store

A first design

Agents push to an ingest tier, which writes to a queue, which workers drain into a time-series store. Standard, and correct in shape.

The question is what the queue does when the store cannot keep up — and the default answer, an unbounded queue, is the one that turns a slow store into a memory exhaustion incident.

The store slows to 80% of the ingest rate. With an unbounded queue, what happens?

The queue grows forever and the data gets older and older — A 20% mismatch is arithmetic, not a burst. The queue grows at 200 a second for as long as the mismatch lasts, and the age of the head of the queue grows with it. Nothing has failed and nobody has been told.

4. The backlog nobody is told about

Watch it break

Watch the queue depth chart and, more importantly, the age of the oldest message. Within a minute the head of the queue is a minute old. Your dashboards are showing data from a minute ago and there is no indication anywhere that they are.

For a monitoring system this is the worst possible failure: it is lying, confidently, at the moment you most need it to be honest.

An unbounded queue converts an error you would notice into a lag you would not.

5. Bound it, and choose what to lose

Find the bottleneck

Bound the queue. Then choose: drop the newest, which keeps history intact and loses the present, or drop the oldest, which keeps the present and loses history. For metrics, dropping the OLDEST is almost always right — a dashboard showing the last thirty seconds with a gap ten minutes ago is far more useful than a dashboard showing ten minutes ago.

And emit a metric counting what you dropped. A pipeline that silently discards data is indistinguishable from a pipeline that is working.

Drop the oldest, count what you dropped, alert on the counter.

6. The store underneath

Change one thing

A time-series store is append-heavy and read-recent, which is exactly the workload an LSM tree is built for: writes are sequential appends into memory, flushed as immutable files.

The bill arrives at compaction. Those files must be merged, repeatedly, and every byte written is rewritten several times over its life — and the merge competes with your reads for the same disk. Run the storage engine concept and watch the read p99 spike every time a compaction runs.

This is why time-series stores are so aggressive about downsampling: rolling up old data into coarser buckets reduces both the storage and the compaction work, and nobody queries last month at one-second resolution.

LSM for the write path; compaction owns the read tail; downsample aggressively.

7. What gets asked

What gets asked next

"How do you handle a metrics storm from one bad deploy?" Per-tenant and per-series limits at ingest, applied before the queue. One service emitting a million new series because someone put a request id in a label is the most common metrics outage there is, and it is called cardinality explosion.

"How do alerts stay fast when the pipeline is backed up?" Give them a separate path. An alert evaluating on the last two minutes must not queue behind a dashboard backfill, and the isolation is worth the duplication.

"What about exactly-once?" You do not need it. Metrics are idempotent by construction if you key on series and timestamp — writing the same point twice produces the same result — which is a rare and pleasant property worth pointing out.