◐ system-design/design-notification-system.md

43. Design a Notification System

Tests: fan-out, delivery via multiple channels (push/email/SMS), retries, idempotency, rate limiting, user preferences.

~4 min read·updated 5/29/2026

43. Design a Notification System

Tests: fan-out, delivery via multiple channels (push/email/SMS), retries, idempotency, rate limiting, user preferences.

43.1 Requirements

Functional

  • Send notifications via push (APNS/FCM), email, SMS, in-app.
  • Support templates with personalization.
  • Respect user preferences (channels, do-not-disturb hours).
  • Track delivery status.
  • Batched / scheduled notifications.

Non-functional

  • High throughput: 100M+ notifications/day; spikes during marketing campaigns.
  • Best-effort but reliable: at-least-once delivery to provider.
  • Idempotent: don't spam users with duplicates.
  • Low latency for transactional (login codes < 5 sec).

43.2 Estimates

  • 100M notifications/day = ~1200/sec average; peak ~10K/sec (campaign spike).
  • Per notification: small (KBs).
  • Storage: notification log retained for analytics + audit (~30 days hot, archive longer).

43.3 Architecture

[ Producer services ]──→[ Notification API ]──→Kafka──→[ Dispatcher pool ]
                                                                │
                                                ┌───────────────┼───────────────┐
                                                ↓               ↓               ↓
                                          [ Push svc ]   [ Email svc ]   [ SMS svc ]
                                                ↓               ↓               ↓
                                          [ APNS/FCM ]    [ SES / SG ]    [ Twilio ]
                                                │               │               │
                                                └───────→[ Status callbacks ]←──┘
                                                                │
                                                       [ Notification DB ]
[ User prefs DB ]                                       [ Analytics / dashboards ]
[ Templates DB ]

43.4 Notification API

POST /notifications/send
{
  "user_id": "...",
  "type": "comment_reply",
  "channel": ["push", "email"]  // or "auto"
  "template": "comment_reply_v2",
  "params": { "commenter": "Alice", "post_id": "..." },
  "idempotency_key": "...",
  "send_at": "2026-05-04T10:00:00Z"  // optional
}

Steps:

  • Validate, idempotency dedup.
  • Fetch user prefs (channels, DND, language).
  • Render template (per channel).
  • Persist intent (Notification DB).
  • Push to Kafka.

43.5 Dispatcher

Consumes Kafka; routes per channel:

  • Determine channel(s) per user prefs.
  • Look up provider config.
  • Per-channel rate limiting (provider-imposed).
  • Send via provider API.
  • Capture provider message_id.
  • Update status (queued, sent).

43.6 Channels

Push (mobile)

  • APNS (Apple): HTTP/2, requires device token + certificate.
  • FCM (Google): for Android; also HTTP/2.
  • Webpush (browsers): VAPID-signed messages to push services.

Device token registration: client registers; server stores (user_id → device_tokens[]).

Email

  • Transactional ESP: SendGrid, AWS SES, Mailgun, Postmark.
  • DKIM/SPF/DMARC for deliverability.
  • Bounces / complaints fed back via webhooks; suppress.

SMS

  • Twilio, AWS SNS, Vonage.
  • Region-specific: 10DLC in US, sender ID in EU/India.
  • Deliverability is unreliable; track failures.

In-app

  • Stored as a row in inbox per user; client polls or WS-pushed.

43.7 User preferences

user_preferences (
  user_id,
  channel_settings (per type, per channel),
  do_not_disturb_hours,
  language,
  unsubscribed_categories
)

Resolved per notification:

  • Type=marketing? Skip if unsubscribed.
  • DND active? Defer or drop.
  • Channel preference: push first, fall back to email if no token.

43.8 Templates

Versioned templates per channel:

template_id, channel, subject, body, params_schema

Render with parameters; localization via per-language variant.

A/B test on subject lines: send 5% with variant A, 5% with B; pick winner.

43.9 Idempotency

Producer sends idempotency_key per logical event ("user X replied to post Y" → key = reply:Y:X).

Notification API dedups: if key seen in last N hours, return original notification ID without queuing again.

Critical for retries from upstream services.

43.10 Status tracking

Each notification has lifecycle:

  • queued
  • sent (handed to provider)
  • delivered (provider confirmed)
  • failed
  • bounced (email)
  • clicked (engagement tracking)

Provider webhooks update status; deliveries unique per notification + provider message id.

43.11 Retries

Provider call fails:

  • 4xx (bad request): don't retry; log.
  • 5xx / network: retry with exponential backoff + jitter.
  • After N retries: dead-letter queue + alert.

43.12 Scheduling

For send_at in future:

  • Persist; do not push to dispatcher Kafka yet.
  • Scheduler service polls "due" notifications; pushes to dispatcher.
  • Use a sorted-by-time index for efficient "due now" queries.

For batched campaigns (e.g., "send to 10M users now"):

  • Materialize the user list (pre-resolve).
  • Push in chunks to Kafka.
  • Throttle to provider rate limits.

43.13 Rate limiting

Multiple layers:

  • Provider limits (Twilio: X messages/sec; APNS: connection limits).
  • Per-user (don't spam: max N notifications/day).
  • Per-tenant (isolation in multi-tenant).
  • Global (cap total spend on third-party).

43.14 Quiet hours / DND

User-defined or default. Notifications during DND:

  • Drop (low importance).
  • Defer to end of DND (medium).
  • Always send (high: 2FA codes, security alerts).

Importance level set by sender.

43.15 Personalization

Beyond template params:

  • ML-driven (best time to send for this user).
  • Smart bundling (group multiple notifications into one digest).
  • Smart channel selection (which channel does user engage with most?).

43.16 Failure modes

  • Provider down: queue grows; alert; fall back to alternate channel where allowed.
  • Kafka down: API queues to local disk briefly; alerts; refuses if backlog huge.
  • DB down: degraded mode (status not tracked); messages still flow.
  • Token rotation: old tokens fail → mark device as inactive.

43.17 Sketch

Producers──→[Notif API]──dedup, prefs──→Kafka(notif)
                                              │
                                       [Dispatcher pool]
                                              │
                                ┌─────────────┼─────────────┐
                            Push svc       Email svc      SMS svc
                                │              │              │
                            APNS/FCM       SES/SG        Twilio
                                │              │              │
                                └────→ webhooks ←────┘
                                              │
                                       [Status DB]
                                       [Analytics]
[Scheduler]──polls due──→[Notif API]
[Pref svc]                    
[Template svc]

43.18 What an interviewer wants

  • Async event-driven design (Kafka).
  • Per-channel dispatcher with retry + DLQ.
  • Idempotency keys end-to-end.
  • User prefs and DND handled.
  • Provider rate limit awareness.
  • Status webhooks for delivery / bounce / click tracking.

43.19 Common pitfalls

  • Sync send via provider in API: latency + provider outage = API outage.
  • No idempotency: retried events spam user.
  • No bounce / complaint handling: deliverability tanks.
  • No DND: user disables app.
  • No rate limit per user: campaign sends 100 emails to one person.

Key takeaways

  • Notification = async fan-out via Kafka to per-channel dispatchers.
  • Idempotency keys end-to-end prevent dupes.
  • User preferences + DND + rate limits enforce respect for the user.
  • Provider webhooks give status (delivered/bounced/clicked).
  • Dead-letter queue + monitoring handle stuck messages.

// 2 views

●main
UTF-8·typescript