1. Someone else owns half your system

Requirements

A notification service is mostly an integration problem. The push gateways, the email provider and the SMS carrier are all third parties with their own rate limits, their own outages and their own opinions about retries. You control none of them.

Ask what the delivery guarantee is per channel — a duplicate push is an annoyance, a duplicate SMS costs money, a duplicate payment receipt is a support ticket. Ask what the priority classes are, because a password reset and a marketing blast must not share a queue.

Duplicates cost different amounts per channel. That decides the design.

2. The fan-out and the burst

Estimate

500 million a day is about 6,000 a second average. That is unremarkable, and it is not the number that matters.

The number that matters is the burst: a marketing campaign to twenty million users is twenty million notifications in whatever window the campaign is scheduled for. If that window is a minute, it is 330,000 a second, fifty times the average, and it arrives on a system sized for 6,000.

Average 6k/s. A campaign is 330k/s. Size for the burst or shed it deliberately.

3. The first design, and the dual write in it

A first design

A service that records the notification and publishes it to a queue for the senders. Two writes, two systems, no transaction across them — which is the dual-write problem, and it will silently lose notifications every time a process dies between the two.

Run the outbox concept and watch the count of records that exist with nothing downstream ever having heard about them.

The broker is unreachable for eight seconds. What happens to notifications created in that window?

They exist in the database and are never sent — The database write already committed. The publish failed. Nothing anywhere records that these two facts disagree, so no retry will ever find them — only a reconciliation job that somebody has to think to write.

4. Notifications that exist and were never sent

Watch it break

The count on screen is notifications that were created and will never be delivered. No error was raised, no alert fired, and the only symptom is a user who did not get their password reset email.

Reversing the order does not fix it; it inverts it into events for notifications that do not exist. There is no ordering of two writes to two systems that is safe.

No ordering of two writes to two systems is safe.

5. One write, then a relay

Find the bottleneck

Put the outbound event in the same database transaction as the notification record. Either both are there or neither is, and there is no window. A relay then reads the outbox and publishes, marking rows sent.

If the relay dies, unsent rows are still there. If it publishes twice, the consumer deduplicates. The system is now at-least-once by design rather than lossy by accident, and at-least-once is a guarantee you can build on.

One transaction, then a relay. Everything after it is retryable.

6. And now the duplicates

Change one thing

At-least-once means duplicates, guaranteed rather than occasional. For push that is an annoyance. For SMS it is a bill. The provider is charging per message and you are about to send some of them twice.

The fix is a unique constraint on an idempotency key, applied in the same transaction as the send record — not a cache of recently-seen ids, which is fast and loses everything exactly when the redeliveries arrive in bulk, which is on restart.

Run the idempotency concept with each of the three strategies and watch the ledger drift.

Unique constraint in the same transaction as the effect. Nothing weaker survives a restart.

7. The provider is down. Now what?

What gets asked next

"The SMS carrier starts failing. What does your system do?" If the answer is "retry", ask yourself what happens when the carrier comes back and finds your entire backlog arriving at once — that is a retry storm aimed at somebody else's system, and they will rate-limit you for it.

Circuit-break per provider, back off with jitter, and hold the backlog in the queue rather than in flight. Then decide, in advance, what the queue depth limit is and what happens when it is reached, because "unbounded" is a decision to deliver a password reset four hours late.

And separate the priority classes into separate queues with separate workers. A marketing campaign must never be able to delay a password reset, and one shared queue guarantees that it can.