WebSocket fan-out

The problem it solves

“Add real-time updates” is sized as though it were request/response: how many messages per second, times some cost each. That arithmetic is wrong by two to four orders of magnitude, because the unit of load in a fan-out system is not the publish, it is the send — and one publish into a room of ten thousand subscribers is ten thousand sends.

Long-lived connections also change the resource model completely. A request holds resources for milliseconds. A WebSocket holds a socket, a TLS session, a read buffer, a write buffer and application state for hours. A hundred thousand of them is a memory and file-descriptor problem before it is a CPU problem, and the failure it produces is not gradual.

The mechanism

Connections land on servers via a load balancer, and a client subscribed to a room may be on any server. So a publish must reach every server holding a subscriber, which is what the backplane is for.

A broadcast backplane sends every message to every server, and each server filters for its own subscribers. Simple, and its traffic grows with the server count — add servers to handle more connections and you add backplane load on every existing server. That term is what stops you scaling horizontally: eventually each server spends its capacity receiving messages for rooms it does not hold.

A sharded backplane routes by room, so a server receives only messages for rooms it holds subscribers for. It requires a routing layer that knows which servers hold which rooms, and it removes the quadratic term.

Then the failure that takes servers down. Each connection has an outbound buffer. A client that stops reading — a phone that went into a tunnel, a laptop that slept, a browser tab throttled in the background — has messages queued for it that it never takes. The buffer grows. With a fraction of a percent of clients doing this and a busy room, the memory held for clients that are not reading grows until the process hits its limit — and then every connection on that server dies, including the healthy ones.

The fix is a bounded per-connection queue and a disconnect for anyone who exceeds it. It feels harsh. It is the difference between dropping a handful of slow clients, who will reconnect, and dropping all hundred thousand.

What the simulation shows

Run the default and watch the memory panel. A slow-client share of 0.4% is enough: memory held for clients that are not reading climbs steadily, and when it crosses the limit the server takes every connection with it. The buffered-messages panel shows which servers are in trouble before they fail, which is the signal you would want on a dashboard.

Now bound the per-connection queue and disconnect anyone over it. The slow clients are dropped, they reconnect, memory stays flat, and the server never falls over. Same load, same slow-client share, no outage. This is backpressure applied to a connection buffer, and it is the same lesson: an unbounded queue does not absorb the mismatch, it postpones and amplifies it.

Then the scaling term. Compare a broadcast backplane at eighteen servers with a sharded one. Broadcast backplane traffic grows with server count; sharded does not. The fan-out panel makes the headline number visible throughout: the publish rate is not the load, the multiplier is.

The numbers worth carrying

Sends per second = publish rate × mean subscribers per room. 1,500 publishes a second into rooms averaging 30 subscribers is 45,000 sends a second — and with skewed room sizes, the busy rooms dominate and the mean understates the peak badly.

Memory per connection is the sizing constraint: 10–50 KB is typical for socket buffers, TLS state and application context. At 24 KB, a hundred thousand connections is 2.4 GB before a single message is buffered. Buffered messages are on top of that, which is why the slow-client failure arrives so fast.

Connections per server: tens of thousands is routine for a tuned Node, Go or Erlang process; a hundred thousand-plus is achievable with care (file descriptor limits, ephemeral port ranges, kernel buffer tuning). Choose the number, then divide, and remember every server needs headroom for a reconnect storm.

Reconnect storms are the real capacity event. If a server holding 100,000 connections dies, all of them reconnect at once, usually within seconds, onto the remaining servers — which must have both the capacity and the authentication throughput to accept them. Jittered reconnect backoff on the client is mandatory, and it is the same lesson as retry storms.

Where it breaks down

Room size skew. Most rooms are small; a few are enormous. The enormous ones set your architecture — a room with a million subscribers cannot be served by fanning out individually, and needs a tree, a pub-sub tier, or a CDN-style edge fan-out.

Ordering and delivery guarantees. WebSocket gives you ordered delivery on a connection. Across a reconnect, ordering and completeness are gone. If the client needs to know what it missed, it needs a sequence number and a catch-up query, which is an entirely separate design.

Sticky routing. Connections are stateful; a load balancer must keep a client on its server for the connection’s life, and the backplane must know where rooms live. Server failure means mass reconnection, which is why the capacity headroom above is not optional.

Presence is harder than messaging. “Who is online” requires a consistent view across servers, and it is the feature that most often forces a coordination service into an otherwise simple design.

What people get wrong

“We can handle 1,000 messages a second.” Publishes or sends? The answer differs by the fan-out multiplier, which is frequently a factor of a thousand.

“Slow clients just get slow.” They consume server memory until the server dies, taking every healthy connection with it. Bounded buffers and disconnection are the only fix.

“We will use Redis pub/sub for the backplane.” Fine at small scale, and it is a broadcast backplane: every server receives everything. Know the term you are signing up for, and know that Redis pub/sub is fire-and-forget, so a momentary disconnect loses messages silently.

“WebSockets are always better than polling.” For low-frequency updates, long-polling or server-sent events cost far less and are much simpler operationally. SSE in particular gets you unidirectional push with plain HTTP semantics, and most “real-time” features are unidirectional.

In production

Socket.IO with the Redis adapter is the common starting point and is a broadcast backplane. Phoenix Channels (Elixir) shard by topic across the cluster and are the reference implementation for doing this well. Centrifugo, Ably, Pusher and AWS API Gateway WebSockets sell the problem as a service, which is frequently the right call — the operational surface here is larger than it looks.

Whatever the stack, the three settings that matter are the per-connection buffer bound, the slow-client disconnect policy, and the client’s reconnect backoff with jitter. All three are usually left at defaults, and all three are what decide whether a slow-client incident is a metric or an outage.

The follow-up questions

“A million connected users, one message to a room of 10,000. What is the load?” — 10,000 sends, plus backplane traffic to every server holding a subscriber. Do the multiplication out loud.

“A client stops reading. What happens?” — Server memory grows until the process dies. Then the bounded-buffer fix.

“A server dies. What happens?” — Every one of its connections reconnects at once. Say how the remaining fleet absorbs that, and mention jittered backoff.

“Do you need WebSockets?” — Often SSE or polling is enough. Choosing the simpler transport when it fits is a stronger answer than scaling the complex one.