Observability, security and cost — every question, written out
The three questions an interviewer asks after the design is drawn, and the ones candidates most often have no answer for.
Metrics, logs and traces — what is each actually for?
Comparison
Metrics say something is wrong, traces say where, logs say why.
Metrics are cheap, aggregated and low-cardinality, so they carry alerts. Traces follow one request across services and answer which hop was slow. Logs carry the detail for a specific event. Using one for another’s job is how observability bills get out of control.
Your metrics bill triples after a deploy. What is the most likely cause?
Code diagnosis
A high-cardinality label — a user id or request id — on a metric.
Every distinct label combination is a separate time series, so putting a user id in a label creates one series per user. This is cardinality explosion and it is the most common metrics incident there is — which is why ingest should enforce a per-service series limit.
See it run — Bounded ingest, with the drops counted.
What does an error budget give you that an uptime target does not?
Trade-off & selection
A quantity to spend, which turns reliability into a decision.
A 99.9% target is 43 minutes of unavailability a month, and that is a budget rather than a failure allowance. Budget remaining means you can ship faster; budget exhausted means reliability work takes priority. It converts an argument about risk appetite into arithmetic.
What should page a human at three in the morning?
Trade-off & selection
Symptoms users feel, not causes a dashboard can show.
Page on symptoms — error rate, latency, saturation approaching a cliff — because those are what users experience and what needs a human now. Causes belong on a dashboard for whoever is already awake and looking.
You sample one per cent of traces. What have you given up?
Trade-off & selection
The specific slow request, unless sampling is biased toward slow ones.
Head-based sampling decides before knowing the outcome, so the interesting traces are usually discarded. Tail-based sampling decides after the request finishes and can keep everything slow or failed — more expensive to run and far more useful.
What is the difference between authentication and authorisation?
Comparison
Who you are, against what you are allowed to do.
Authentication establishes identity; authorisation decides what that identity may do. They are usually implemented in different places — identity at the edge, permissions near the resource — and confusing them produces systems that check who you are and then let you do anything.
Signed tokens validated locally, or opaque tokens checked with the issuer?
Trade-off & selection
Local validation is fast and cannot be revoked before expiry.
A locally validated token needs no network call, which is why it is used at scale — and it remains valid until it expires, so revoking access takes as long as the token lifetime. Short lifetimes plus refresh is the usual compromise, and the lifetime is the revocation delay you are accepting.
Where should a service get its database password?
Invariant identification
From a secrets manager at startup, with rotation supported.
The requirement is not only that the secret is hidden but that it can be changed without a deploy. A secrets manager gives short-lived credentials fetched at startup and refreshed, so a leaked credential expires on its own — which is the property that matters after an incident.
Data is encrypted at rest and in transit. What is still exposed?
Edge case reasoning
Data in use, and anything the application itself is tricked into returning.
Encryption at rest protects a stolen disk; in transit protects a tapped network. Neither protects against an injection flaw, a broken authorisation check or a leaked credential — and those are how data actually escapes. Encryption is table stakes rather than a security posture.
Which cost do candidates most often omit from a design?
Complexity derivation
Egress bandwidth, which is billed and is often the largest line.
Storing a terabyte costs a few tens of dollars a month; serving it out repeatedly can cost far more. This is why CDNs pay for themselves, why cross-region replication is expensive, and why "just replicate everything everywhere" gets a raised eyebrow.
You add a third replica in another region. What multiplies?
Complexity derivation
Storage, plus continuous cross-region transfer for every write.
The storage is a third more; the transfer is your entire write volume, forever, across a billed boundary. For a write-heavy system that recurring cost usually exceeds the storage, which is why cross-region replicas are chosen for a stated reason rather than by default.
Finance asks why the fleet runs at 40% utilisation. What is the answer?
Trade-off & selection
Headroom for peaks and for losing capacity, and latency is non-linear in utilisation.
Average utilisation hides the peak, and the peak is what the fleet must serve. Losing an availability zone at 40% takes you to 60%; the same loss at 70% takes you to 105%, and past saturation the queue never drains. The saving is real and smaller than it looks.
See it run — What the last stretch of capacity costs in latency.
What makes a deploy safe to roll back?
Invariant identification
Every change being backward compatible with the version it replaces.
Rollback is only possible if the old code can handle the new data. That means expand-and-contract for schemas: add the column, write to both, migrate, read from the new one, and only then remove the old — with each step independently reversible.
When is active-active multi-region worth its complexity?
Trade-off & selection
When you need regional failover or local write latency, and can handle conflicts.
Active-active means accepting writes in more than one region, which means concurrent writes to the same key and a conflict resolution strategy you must be able to defend. It is worth it when a regional outage is unacceptable or cross-ocean write latency is — and it is a large step, not an incremental one.
Which number tells you a system is close to a cliff rather than merely busy?
Code diagnosis
Queue depth trending up while throughput is flat.
Throughput flat while queue depth climbs means arrivals exceed service rate, and past that point latency grows without bound. It is the earliest reliable signal and it leads both latency and errors — which is why it is worth a panel of its own.
An interviewer asks how you would know your design is working in production. Answer.
Explain it plainly
I would start from what users experience and work inward. At the edge: request rate, error rate and latency percentiles per endpoint, and I would alert on the p99 and the error rate against an SLO rather than on any component being unusual. Then the leading indicators specific to this design. The queue depth and the age of the oldest message, because those move before latency does and tell me a mismatch is accumulating. The cache hit ratio, and specifically the origin request rate rather than the ratio itself, because a one-point drop at ninety-nine per cent is a doubling downstream. Utilisation on the database, since it is the tightest component and the queueing curve means the last stretch costs disproportionately. For debugging: distributed tracing with tail-based sampling so we keep the slow and failed traces rather than a random one per cent, and structured logs correlated by the same trace id. On paging: symptoms only. Error rate breaching the budget, latency past the SLO, queue age past what the business tolerates. Everything else goes on a dashboard for whoever is already looking. And I would want a runbook entry for each alert, because an alert nobody knows how to action is an alert that gets silenced.
The answer has to name specific signals tied to the design just drawn, not a generic list of tools.
Which parts of a system should degrade first under load?
Trade-off & selection
Whatever is optional, decided in advance and made explicit in the design.
Graceful degradation is a design decision made before the incident: recommendations disappear, personalisation falls back to a default, search narrows — while checkout keeps working. Deciding at three in the morning means not deciding.