Design a notification system

Preference-aware, deduplicated, rate-limited delivery across email, push, SMS, and in-app: data model, delivery semantics, and failure modes.

Notification System Design

A notification system turns product events into user-visible messages across email, push, SMS, in-app inboxes, webhooks, and realtime badges. Sending one message is easy: it is a provider SDK call. The hard part is preference-aware, deduplicated, rate-limited, observable delivery across unreliable providers , at a fanout that must never block the product write path.

What we are building

Transactional notifications: login alerts, mentions, payment receipts, reminders, workflow updates. Five channels: in-app inbox, mobile push (APNs and FCM), email, SMS, and outbound webhooks for tenant integrations. The system must:

respect user preferences, quiet hours, tenant policy, locale, and legal unsubscribe rules (CAN-SPAM and GDPR for email, TCPA for SMS), guarantee that retries and event replays never produce a second visible message, record delivery state and provider responses so support can answer "why did I not get this", absorb a tenant-wide announcement to 2 million users without delaying a password reset, keep product services unaware of channels and providers.

Out of scope: marketing campaign authoring, audience segmentation, subject-line experiments. Those share the delivery tier below but have their own creation path with different rules (a campaign is scheduled and cancellable; a login alert is neither).

Assume a consumer product with 50 million monthly and 10 million daily active users. The numbers that matter are intents per second, deliveries per day, the worst single fanout, and the SMS bill.

Quantity Assumption Result --- --- --- Notifiable events 20 per DAU per day 200 million events per day Recipients per event 1.5 on average 300 million intents per day Average intent rate 300 million over 86,400 s about 3,500 intents per s Peak factor 5x (morning wave plus one hot broadcast) about 17,500 intents per s Suppression 60 percent removed by preferences, dedupe, digests 120 million deliveries per day Channel mix 55 push, 30 in-app, 14 email, 1 SMS (percent) 1.2 million SMS per day SMS cost 0.0075 dollars per message about 9,000 dollars per day Intent row 300 bytes 90 GB per day Delivery row 500 bytes 60 GB per day Raw state per year 150 GB per day times 365 about 55 TB, or 13.5 TB at 90 day hot retention

Two derived facts shape the design. First, the average is not the problem: 3,500 intents per second is one modest queue. The problem is the single event with 2 million recipients, which at 17,500 per second would occupy the whole system for two minutes if it shared a lane with everything else. Second, one percent of the volume produces most of the provider cost, so SMS gets its own policy gate and its own budget counter.

Product services never call a provider. They commit their own transaction plus an outbox row, and a relay publishes the fact to the event bus (see wiki/event-bus-for-product-events). The router is the only component that knows about users, preferences, and channels. It turns one event into zero or more intents, and each intent is a durable row before it is a queue message. Per-channel lanes feed workers that talk to providers, and a receipt consumer folds provider callbacks back into intent state. Creation (the router) and delivery (the workers) are separate services with separate on-call, because they fail differently.

Entity Key Partition key Notes --- --- --- --- notification event event id event id hash Immutable fact from the bus, kept 30 days for replay and audit. notification intent intent id, a hash of event id, user id, channel user id The decision to notify. Unique on event id, user id, channel. notification delivery delivery id user id One attempt through one provider: provider message id, error class, timestamps. notification preference user id, tenant id user id Channel opt-ins, quiet hours, digest cadence, unsubscribe tokens. device token user id, token user id Push tokens with platform and last-seen, invalidated by provider feedback. notification template template key, version replicated everywhere Localized content, immutable per version. inbox item user id, created at desc, event id user id The in-app list. Same uniqueness as intent.

Everything hot is partitioned by user id. The product reads per user (inbox, badge, preference screen, support lookups), and the uniqueness constraint that enforces at-most-once visible delivery lives on (event id, user id, channel), so both the read and the guard sit inside one partition. Events partition by their own id because nobody reads them per user; they are the replay source.

The product service commits its change and an outbox row in one transaction. If the notification were published before the commit, a rollback would leave a message about something that never happened; the outbox removes that class of ghost. The relay publishes the event with event id, occurred at, actor, subject ids, and a trace id. The router's consumer group reads it and first checks a short-lived processed-event set; a replayed event stops here. The router loads preferences and tenant policy for each candidate recipient, evaluates quiet hours in the recipient's timezone, and applies the per-user rate limit (a mention storm on one thread should collapse into one push, not 40). It inserts one intent per surviving recipient and channel with the deterministic intent id. A duplicate key error is not an error; it means a previous run got here first, and the router moves on. Only after the intent commits does the router enqueue a message per channel carrying the intent id. If the process dies between steps 4 and 5, a sweeper enqueues any intent older than 30 seconds still in the created state. The intent row is the truth; the queue is a hint.

The template version is chosen at step 4 and stored on the intent. Workers render with that version, so a template edit does not change a message already decided.

Three reads exist, and none of them scan.

The inbox is a keyset-paginated read of inbox item by user id, newest first. The badge count is a separate projection in Redis, a sorted set per user of unseen event ids, cleared when the user opens the inbox (see wiki/newly-unread-indicator for why this is an acknowledgement, not read state). Delivery status for support tooling reads intent plus deliveries by user id and event id.