1. This is not a request/response system

Requirements

Chat breaks the assumption every other design in this section rests on. Connections are long-lived, so the constraint is memory and file descriptors rather than throughput. Delivery is pushed rather than pulled, so a message published on one server must reach subscribers on every other. And ordering matters within a conversation but not across them.

Ask whether messages must be delivered exactly once (they cannot be — ask instead whether duplicates are visible), whether read receipts and presence are in scope (presence is far more expensive than messaging), and how long history is kept.

Sized by connections and fan-out, not by request rate.

2. Connections first

Estimate

If ten per cent of fifty million daily users are connected at peak, that is five million concurrent connections. A well-tuned server holds around a hundred thousand idle WebSocket connections — memory, not CPU, is the limit — so fifty servers just to hold the sockets, before any message is sent.

Messages: say twenty per active user per day. A billion messages a day, about 12,000 a second average, 35,000 at peak. Each goes to a conversation with a handful of members, so delivered messages are perhaps five times that.

The number that decides the architecture is neither of those. It is that a message published on one of fifty servers may have recipients on any of the other forty-nine.

5M concurrent connections, ~50 servers, and a cross-server delivery problem.

3. The first design

A first design

Clients connect to a gateway, which holds the socket. A chat service handles the message, writes it to the store, and publishes it to a backplane that every gateway subscribes to. Each gateway delivers to whichever of its connections are in that conversation.

The flaw is on the write path: the message store is synchronous, so delivery is hostage to storage latency.

The message store slows down by eight times for fifteen seconds. What do users see?

Message sending stalls for everyone — The store is on the send path, so its latency is the send latency. Every chat service thread is occupied waiting on it, and once they are all occupied nobody can send at all — including people whose messages would never have touched the slow part.

4. Watch the slow store take the send path with it

Watch it break

The fault is injected at twenty-five seconds. Watch the chat service utilisation climb as its threads sit waiting, and the error rate follow. This is a bulkhead failure: one dependency, one shared pool, and every endpoint that shares it.

It is worth being precise about the mechanism, because "the database was slow" is not a root cause. The root cause is that a slow dependency held threads that other work needed, and that is a property of the design rather than of the database.

A slow dependency on the send path stops sending, not just storing.

5. Two problems, two fixes

Find the bottleneck

First: take storage off the delivery path. Deliver the message immediately and persist it through a queue. History becomes eventually consistent, by a second or two, which nobody notices — and delivery stops depending on the store at all.

Second: the backplane. A broadcast backplane sends every message to every gateway whether or not it holds a recipient, so its traffic grows with the server count. At fifty servers that is fifty times the message rate crossing the network for no reason. Shard the conversations across gateways instead.

And third, the one that actually pages someone at night: the slow consumer. A client on a bad connection stops reading, its outbound queue grows in the gateway's memory, and with enough of them the gateway runs out of memory and drops every connection it holds — including the healthy ones.

Bound the per-connection queue and disconnect slow consumers, or lose the whole server.

6. Deliver first, persist after

Change one thing

The chat service now publishes to the backplane and enqueues the write. Delivery latency is the network plus a few milliseconds; persistence happens behind it. Re-run the same fault: the store is still slow, and sending is unaffected.

The queue depth chart is now the thing to watch. If the store stays slow for long enough, the backlog grows — and the honest answer to "what then" is that you shed, or you accept unbounded lag, and you should say which before you are asked.

The same fault now shows up as queue depth rather than as an outage.

7. What gets asked next

What gets asked next

"How do you guarantee a message is not lost?" You do not, with an at-most-once send. You acknowledge to the sender only after the message is durable, and you accept that delivery to other members is at-least-once — which means the client deduplicates on a message id the sender generated.

"How do you order messages in a conversation?" Partition the queue by conversation id. Order within a partition is guaranteed; order across conversations is not, and was never needed.

"What about presence?" Presence is far more expensive than messaging: every user's status change fans out to everyone who can see them, continuously. Most systems degrade it deliberately — batched, delayed, and approximate — and saying so unprompted is a strong signal.