Push Notification Service

0Problem RestatementProblem

Design a scalable push notification service that can send notifications to millions of users across multiple platforms (iOS, Android, Web) with high throughput, low latency, and guaranteed delivery. Core challenges include handling massive fan-out (one message to millions of devices), respecting rate limits from platform providers (APNs, FCM), and ensuring reliability despite failures.

1RequirementsRequirements

1.1 Functional

• Send push notifications to users on iOS, Android, and Web.
• Support multiple notification types: transactional (order updates), promotional (offers), social (friend requests).
• Target individual users, user segments, or broadcast to all users.
• Schedule notifications for future delivery.
• Track delivery status (sent, delivered, clicked, failed).
• Support rich notifications (images, actions, deep links).
• Allow users to manage notification preferences.
• Provide analytics (delivery rate, click-through rate).

1.2 Non-Functional

• High Throughput: Send millions of notifications per second.
• Low Latency: Deliver critical notifications within seconds.
• Reliability: Guaranteed delivery with retries and fallbacks.
• Scalability: Handle 1 billion+ registered devices.
• Rate Limiting: Respect provider limits (APNs, FCM).
• Availability: 99.9% uptime.
• Deduplication: Prevent duplicate notifications.
• Prioritization: Critical alerts delivered before promotional content.

1.3 Scale Estimates

• Registered devices: 1 billion devices (500M iOS, 400M Android, 100M Web).
• Average throughput: 10B/day ÷ 86,400 s ≈ 116K notifications/second.
• Peak throughput: ~500K notifications/second (broadcasts, major events, flash sales — ~4× average).
• Provider rate limits:
• Apple and Google don't publish fixed throughput numbers. The figures below (~500 notifications/sec per APNs HTTP/2 connection, thousands of connections) are planning assumptions — measure real throughput and back off on errors.
• FCM: No published per-sender limit, but it throttles (HTTP 429) when you send too fast.
• Storage: 10B notifications/day × 1 KB × 30 days retention = ~300 TB retained at any time (rolling window).

1.4 API Design

The core APIs required for the service:

2High-Level ArchitectureArchitecture

2.1 Overview

• Client Apps → API Gateway → Notification Service → Fan-Out Service → Provider Workers (APNs, FCM) → Platform Providers → User Devices.
• Key components: Fan-out for segment/broadcast, rate limiter per provider, retry queue, analytics pipeline.

2.2 Architecture Diagram

%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart TB
    %% Sender initiates notification
    Sender["Notification Sender<br/>(App Backend)"] -->|"1. POST /notify {userId, message}"| AG["API Gateway"]
    
    %% Notification Service
    AG -->|"2. Validate & Enqueue"| NS["Notification Service"]
    NS -->|"3. Lookup Devices"| DeviceDB[(Device Registry DB)]
    DeviceDB -->|"4. Return Tokens"| NS
    
    %% Fan-Out for Segments/Broadcast
    NS -->|"5a. Single User"| Queue["Notification Queue<br/>(Kafka/SQS)"]
    NS -->|"5b. Broadcast Request"| FanOut["Fan-Out Service"]
    FanOut -->|"6. Query Segment"| UserDB[(User Segment DB)]
    FanOut -->|"7. Batch Enqueue"| Queue
    
    %% Provider Workers
    Queue -->|"8. Consume"| iOS["iOS Worker Pool"]
    Queue -->|"8. Consume"| Android["Android Worker Pool"]
    Queue -->|"8. Consume"| Web["Web Worker Pool"]
    
    %% Rate Limiters
    iOS -->|"9. Rate Limit & Send"| RateLimiter1["Rate Limiter<br/>(APNs)"]
    Android -->|"9. Rate Limit & Send"| RateLimiter2["Rate Limiter<br/>(FCM)"]
    Web -->|"9. Rate Limit & Send"| RateLimiter3["Rate Limiter<br/>(Web Push)"]
    
    %% Platform Providers
    RateLimiter1 -->|"10. HTTP/2 Connection"| APNs["Apple Push<br/>Notification Service"]
    RateLimiter2 -->|"10. HTTP/2 Connection"| FCM["Firebase Cloud<br/>Messaging"]
    RateLimiter3 -->|"10. HTTP/2 Connection"| WebPush["Web Push Protocol"]
    
    %% Delivery to Devices
    APNs -->|"11. Push to Device"| iOSDevice["iOS Devices"]
    FCM -->|"11. Push to Device"| AndroidDevice["Android Devices"]
    WebPush -->|"11. Push to Device"| WebBrowser["Web Browsers"]
    
    %% Failure Handling
    RateLimiter1 -->|"12a. Retriable failure"| RetryQ["Retry Queue<br/>(delayed)"]
    RateLimiter2 -->|"12a. Retriable failure"| RetryQ
    RateLimiter3 -->|"12a. Retriable failure"| RetryQ
    RetryQ -->|"13. Retry with Backoff"| Queue
    RetryQ -->|"14. Max retries exceeded"| DeadLetter["Dead Letter Queue<br/>(inspect / alert)"]
    
    %% Analytics Pipeline
    iOS -->|"12b. Log Status"| Analytics["Analytics Service"]
    Android -->|"12b. Log Status"| Analytics
    Web -->|"12b. Log Status"| Analytics
    Analytics -->|"Store Metrics"| MetricsDB[(Metrics DB<br/>ClickHouse)]
    
    %% Scheduled Notifications
    Scheduler["Scheduler Service"] -->|"Trigger at Scheduled Time"| NS
    
    %% Styling
    classDef sender fill:#e3f2fd,stroke:#1976d2,stroke-width:2px;
    classDef core fill:#c8e6c9,stroke:#388e3c,stroke-width:2px;
    classDef worker fill:#fff9c4,stroke:#f57f17,stroke-width:2px;
    classDef provider fill:#ffe0b2,stroke:#e65100,stroke-width:2px;
    classDef storage fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px;
    
    class Sender,AG sender;
    class NS,FanOut,Scheduler core;
    class iOS,Android,Web,RateLimiter1,RateLimiter2,RateLimiter3 worker;
    class APNs,FCM,WebPush,iOSDevice,AndroidDevice,WebBrowser provider;
    class DeviceDB,UserDB,Queue,DeadLetter,Analytics,MetricsDB storage;

3Components (what & why)

API Gateway

• Accept notification requests from app backends.
• Validate payload, authentication, and authorization.
• Rate limiting to prevent abuse.

Notification Service

• Responsibilities:
• Parse notification payload.
• Look up device tokens from Device Registry.
• Determine targeting (single user, segment, broadcast).
• Enqueue to notification queue.
• Validation: Check payload size, required fields, deep link validity.

Device Registry DB

• Store mapping of users to device tokens.
• Schema: (user_id, device_token, platform, created_at, last_active)
• Updates: Tokens registered/updated when app opens.
• Cleanup: Expire inactive tokens (e.g., 90 days).

Fan-Out Service

• Purpose: Handle segment and broadcast notifications.
• Strategy:
• Query User Segment DB to get user IDs matching criteria (e.g., "premium users in US").
• Batch fetch device tokens (e.g., 10K at a time).
• Enqueue individual notification tasks to queue.
• Optimization: Parallelize fan-out across multiple workers.

Notification Queue (Kafka/SQS)

• Purpose: Decouple notification ingestion from delivery.
• Partitioning: Separate topics by platform (iOS, Android, Web) for independent scaling, and by priority (transactional vs promotional) so a 1B-device marketing blast can never delay an OTP or order update.
• Collapse keys: For updates that replace each other (e.g. "3 new messages" → "4 new messages"), set the provider's collapse key so the device shows only the latest.
• Ordering: FIFO not required (notifications are independent).

Provider Workers (iOS/Android/Web)

• Responsibilities:
• Consume from queue.
• Send notifications to platform providers (APNs, FCM, Web Push).
• Handle provider responses (success, failure, token expired).
• Log delivery status.
• Connection Pooling: Maintain persistent HTTP/2 connections to providers.
• Horizontal Scaling: Scale workers based on queue depth.

Rate Limiter

• Purpose: Respect provider rate limits to avoid throttling.
• Implementation: Token bucket algorithm per connection.
• APNs Limits: Assume ~500 notifications/sec per connection (not published by Apple — tune from measurements); use multiple connections.
• FCM Limits: Adaptive rate limiting based on error responses.

Retry Queue and Dead Letter Queue (DLQ)

• Retry Queue: Holds retriable failures (429, 5xx, timeouts) until their next attempt. Implement with delay queues (SQS delay, or one Kafka topic per backoff tier: retry-1s, retry-30s, retry-5m…).
• Retry Strategy: Exponential backoff with jitter (1s, 5s, 30s, 5min, 1hour).
• DLQ: After 5 failed attempts, the message goes to the dead letter queue — a terminal holding area that's monitored and inspected, not automatically retried.

Analytics Service

• Metrics Tracked:
• Sent, delivered, failed, clicked.
• Delivery latency (time from enqueue to delivery).
• Error rates per provider.
• Storage: Time-series DB (ClickHouse, TimescaleDB) for fast aggregation.

Scheduler Service

• Purpose: Send notifications at scheduled future times.
• Implementation: Cron-like scheduler or distributed task queue (Celery, Airflow).
• Storage: Scheduled notifications stored in DB, triggered at specified time.

4Data ModelData model

Device Token

DeviceToken(
  token_id,
  user_id,
  device_token,  -- unique token from APNs/FCM
  platform,      -- iOS, Android, Web
  app_version,
  created_at,
  last_active_at
)
-- Index on user_id for fast lookup
-- Index on device_token for deduplication

Notification

Notification(
  notification_id,
  user_id,
  title,
  body,
  payload,       -- JSON (deep link, image URL, actions)
  platform,
  priority,      -- HIGH, NORMAL, LOW
  scheduled_at,  -- NULL for immediate
  status,        -- PENDING, SENT, DELIVERED, FAILED, CLICKED
  created_at,
  sent_at,
  delivered_at
)

User Segment

UserSegment(
  segment_id,
  name,          -- e.g., "Premium users in California"
  criteria,      -- JSON filter (e.g., {subscription: "premium", location: "CA"})
  user_count,    -- cached count
  last_updated
)

Analytics Event

AnalyticsEvent(
  event_id,
  notification_id,
  event_type,    -- SENT, DELIVERED, CLICKED, FAILED
  device_token,
  timestamp,
  error_code     -- NULL if success
)

5Key FlowsFlows

5.1 Single User Notification Flow

1. App backend calls: POST /notify {user_id: "123", title: "Order shipped", body: "..."}.
2. Notification Service validates payload.
3. Service queries Device Registry for user's device tokens (may have multiple devices).
4. For each device token, enqueue task to Notification Queue (partitioned by platform).
5. iOS Worker consumes task, sends to APNs via HTTP/2.
6. APNs delivers to device, returns success/failure.
7. Worker logs status to Analytics Service.

5.2 Broadcast Notification Flow

1. App backend calls: POST /broadcast {title: "Flash sale!", body: "50% off"}.
2. Notification Service triggers Fan-Out Service.
3. Fan-Out Service:

• Queries all active users (or predefined segment).
• Fetches device tokens in batches (e.g., 100K users → 150K tokens).
• Enqueues individual notification tasks to queue.

1. Challenge: 1 broadcast → 1 billion devices = massive queue depth.

5.3 Segment Notification Flow

1. Marketing team calls: POST /notify-segment {segment_id: "premium_users", ...}.
2. Fan-Out Service queries User Segment DB for matching users.
3. Batch fetch device tokens and enqueue.
4. Delivery proceeds as normal.

5.4 Scheduled Notification Flow

1. App backend calls: POST /notify {user_id: "123", ..., scheduled_at: "2023-10-20T10:00:00Z"}.
2. Notification Service stores in DB with status PENDING.
3. Scheduler Service polls DB every minute for notifications due.
4. At scheduled time, Scheduler enqueues notification to queue.
5. Delivery proceeds as normal.

5.5 Failure & Retry Flow

1. Worker sends notification to APNs → receives "InvalidToken" error.
2. Worker marks device token as inactive in Device Registry.
3. Worker logs failure to Analytics.
4. If retriable error (e.g., rate limit, network timeout):

• Enqueue to the Retry Queue with retry count.
• Retry with exponential backoff.

6Deep Dive A: Fan-Out for Broadcast Notifications (~10 mins)Deep dive

Problem

Sending a broadcast notification to 1 billion devices creates 1 billion tasks. Naively enqueuing all tasks at once can overwhelm the queue and delay processing.

Challenges

• Queue Depth: 1 billion messages in queue → high memory, slow dequeue.
• Latency: Users should receive notification within minutes, not hours.
• Throttling: Provider rate limits constrain throughput.

Solution: Multi-Stage Fan-Out

Stage 1: User Aggregation

• Goal: Fetch all users matching criteria.
• Strategy: Query User DB in batches (e.g., 100K users at a time).
• Parallelization: Multiple Fan-Out Workers fetch batches concurrently.

Stage 2: Device Token Lookup

• Goal: Map users to device tokens (1 user may have 1-3 devices).
• Strategy: Batch lookup from Device Registry (100K users → ~150K tokens).
• Caching: Cache active tokens in Redis to reduce DB load.

Stage 3: Batched Enqueue

• Goal: Enqueue tasks to Notification Queue.
• Strategy: Batch enqueue (e.g., 1000 tasks per queue write).
• Partitioning: Distribute across queue partitions (iOS, Android, Web).

Stage 4: Rate-Limited Delivery

• Goal: Send to providers without hitting rate limits.
• Strategy: Workers pull from queue and send at controlled rate.
• Dynamic Scaling: Auto-scale workers based on queue depth.

Fan-Out Architecture

%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart TD
    Broadcast["Broadcast Request<br/>(1B users)"] --> Coordinator["Fan-Out Coordinator"]
    
    Coordinator -->|"distribute work"| Worker1["Fan-Out Worker 1<br/>(users 1-100K)"]
    Coordinator -->|"distribute work"| Worker2["Fan-Out Worker 2<br/>(users 100K-200K)"]
    Coordinator -->|"distribute work"| WorkerN["Fan-Out Worker N<br/>(users 999.9M-1B)"]
    
    Worker1 -->|"batch lookup"| DeviceDB[(Device Registry)]
    Worker2 -->|"batch lookup"| DeviceDB
    WorkerN -->|"batch lookup"| DeviceDB
    
    Worker1 -->|"batch enqueue<br/>(1000 tasks)"| Queue["Notification Queue"]
    Worker2 -->|"batch enqueue"| Queue
    WorkerN -->|"batch enqueue"| Queue
    
    Queue -->|"consume at rate limit"| Senders["Provider Workers"]
    
    classDef fanout fill:#c8e6c9,stroke:#388e3c,stroke-width:2px;
    classDef storage fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px;
    
    class Coordinator,Worker1,Worker2,WorkerN fanout;
    class DeviceDB,Queue,Senders storage;

Time Estimation

• 1 billion users, 10K fan-out workers:
• Each worker handles 100K users.
• Device lookup: ~5 seconds per batch.
• Enqueue: ~2 seconds per batch.
• Total fan-out time: ~7 seconds.
• Delivery time (rate-limited):
• APNs: 500 notifs/sec/connection × 1000 connections = 500K notifs/sec.
• 1 billion / 500K = 2000 seconds (~33 minutes).

7Deep Dive B: Rate Limiting & Throttling (~8 mins)Deep dive

Problem

Platform providers (APNs, FCM) enforce rate limits. Exceeding limits results in throttling, errors, or IP bans.

APNs Rate Limits (Planning Assumptions)

Apple doesn't publish hard numbers; these are assumptions to size the system, to be replaced by measured values.

• Connections: Thousands of persistent HTTP/2 connections (e.g. ~5000).
• Throughput: ~500 notifications/sec per connection.
• Total Throughput: 5000 × 500 = 2.5M notifs/sec (theoretical max).
• APNs also throttles per device: bursts of notifications to the *same* device get delayed or dropped, so rate-limit per device as well as per connection.

FCM Rate Limits

• No Published Limit: But adaptive rate limiting exists.
• Strategy: Monitor error rate; if high, back off.

Solution: Token Bucket Rate Limiter

Implementation

• Bucket per Connection: Each HTTP/2 connection has a token bucket.
• Capacity: 500 tokens (500 notifs/sec).
• Refill Rate: 500 tokens/second.
• Consumption: Each notification consumes 1 token.
• Blocking: If bucket empty, wait until refill.

Algorithm

class TokenBucket:
    def __init__(self, capacity=500, refill_rate=500):
        self.capacity = capacity
        self.tokens = capacity
        self.refill_rate = refill_rate
        self.last_refill = time.now()
    
    def consume(self, count=1):
        self.refill()
        if self.tokens >= count:
            self.tokens -= count
            return True  # Success
        else:
            return False  # Rate limit exceeded, wait
    
    def refill(self):
        elapsed = time.now() - self.last_refill
        new_tokens = elapsed * self.refill_rate
        self.tokens = min(self.capacity, self.tokens + new_tokens)
        self.last_refill = time.now()

Connection Pooling

• Worker Pool: 1000 iOS workers, each with 5 HTTP/2 connections = 5000 total connections.
• Load Balancing: Distribute tasks evenly across workers.

Adaptive Rate Limiting for FCM

• Monitor Error Rate: Track 429 (Too Many Requests) errors.
• Backoff: If error rate > 5%, reduce send rate by 50%.
• Recovery: Gradually increase rate if errors decrease.

Rate Limiter Architecture

%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
    Queue["Notification Queue"] --> Worker["iOS Worker"]
    Worker --> Conn1["HTTP/2 Conn 1<br/>(Token Bucket)"]
    Worker --> Conn2["HTTP/2 Conn 2<br/>(Token Bucket)"]
    Worker --> Conn5["HTTP/2 Conn 5<br/>(Token Bucket)"]
    
    Conn1 -->|"rate limited"| APNs["APNs"]
    Conn2 -->|"rate limited"| APNs
    Conn5 -->|"rate limited"| APNs
    
    APNs -->|"success/error"| Worker
    Worker -->|"log metrics"| Metrics["Metrics DB"]
    Worker -->|"429 error → retry"| DLQ["Retry Queue (delayed)"]
    
    classDef worker fill:#c8e6c9,stroke:#388e3c,stroke-width:2px;
    classDef provider fill:#ffe0b2,stroke:#e65100,stroke-width:2px;
    
    class Worker,Conn1,Conn2,Conn5 worker;
    class APNs provider;

8Deep Dive C: Reliability & Failure Handling (~7 mins)Deep dive

Problem

Notifications must be delivered reliably despite failures (network issues, token expiry, provider downtime).

Failure Modes

1. Invalid/Expired Token

• Cause: User uninstalled app or token expired.
• Detection: APNs/FCM returns "InvalidToken" or "Unregistered".
• Action: Mark token as inactive in Device Registry, do not retry.

2. Rate Limit Exceeded

• Cause: Exceeded provider rate limit.
• Detection: 429 error from provider.
• Action: Retry with exponential backoff, reduce send rate.

3. Network Timeout

• Cause: Temporary network issue.
• Detection: No response within timeout (5 seconds).
• Action: Retry up to 3 times.

4. Provider Downtime

• Cause: APNs/FCM service outage.
• Detection: High error rate across all workers.
• Action: Circuit breaker pattern (pause sends for 5 minutes, then retry).

Retry Strategy

Exponential Backoff

Attempt 1: Wait 1 second
Attempt 2: Wait 5 seconds
Attempt 3: Wait 30 seconds
Attempt 4: Wait 5 minutes
Attempt 5: Wait 1 hour
After 5 attempts: Mark as permanent failure

Retry Queue vs Dead Letter Queue

• Retriable failures go to a retry queue with a delay; a retry consumer re-enqueues them when due.
• Track the retry count on the message to prevent infinite loops.
• After the last attempt, the message moves to the dead letter queue, which is terminal: it triggers alerts and is inspected, not retried automatically.

Idempotency

• Problem: If worker crashes after sending but before acking queue, notification may be sent twice.
• Solution: Use idempotency key (notification_id + device_token) and track sent notifications in cache (Redis).
• Check: Before sending, check if already sent within last 5 minutes.

Circuit Breaker Pattern

• Open: After 100 consecutive failures, stop sending for 5 minutes.
• Half-Open: After cooldown, try 10 test requests.
• Closed: If test requests succeed, resume normal operation.

Reliability Architecture

%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
stateDiagram-v2
    [*] --> Enqueued: New notification
    Enqueued --> Sending: Worker picks up
    Sending --> Success: Provider responds OK
    Sending --> Failed: Provider error
    
    Failed --> RetryQueue: Retriable error
    RetryQueue --> Retry: Wait (exponential backoff)
    Retry --> Sending: Re-attempt
    
    Failed --> PermanentFail: Non-retriable (InvalidToken)
    Failed --> DLQ: Max retries exceeded
    DLQ --> PermanentFail: Inspected / alerted
    
    Success --> [*]
    PermanentFail --> [*]

9Scaling & Performance (~5 mins)Scale

Horizontal Scaling

• Workers: Auto-scale based on queue depth (e.g., if depth > 10K, add workers).
• Fan-Out Service: Scale to handle large broadcasts.
• Database: Shard Device Registry by user_id.

Performance Metrics

• Throughput: ~120K notifs/sec average, 500K notifs/sec peak.
• Latency: P99 delivery within 10 seconds.
• Delivery Rate: > 98% delivered.
• Error Rate: < 2%.

Bottlenecks & Mitigations

• Queue Depth: Use Kafka with high partition count (100+ partitions).
• Database Lookups: Cache active device tokens in Redis.
• Provider Rate Limits: Use max allowed connections, batch requests.

Cost Optimization

• Prioritization: Send high-priority (transactional) first, batch low-priority (promotional).
• Deduplication: Avoid sending duplicate notifications to same device.

10Failure Modes & Recovery

Queue Failure (Kafka/SQS Down)

• Impact: Cannot enqueue new notifications.
• Mitigation: Use replicated queue, fallback to secondary queue.

Database Failure

• Impact: Cannot lookup device tokens.
• Mitigation: Cache tokens in Redis, use read replicas.

Provider Outage (APNs/FCM Down)

• Impact: Cannot deliver notifications.
• Mitigation: Circuit breaker (stop sending), queue for retry when provider recovers.

Worker Crash

• Impact: In-flight notifications lost.
• Mitigation: Queue ack only after successful delivery (at-least-once semantics).

11Trade-offs & AlternativesTrade-offs

At-Least-Once vs Exactly-Once Delivery

• At-Least-Once: Simpler, but may send duplicates.
• Exactly-Once: Requires idempotency tracking (complex).
• Choice: At-least-once with idempotency checks.

Push vs Pull (Polling)

• Push: Low latency, but requires persistent connection.
• Pull: Simpler, but higher latency.
• Choice: Push (standard for mobile).

Self-Hosted vs Third-Party (SNS, OneSignal)

• Self-Hosted: Full control, lower cost at scale.
• Third-Party: Faster to integrate, managed service.
• Choice: Self-hosted for cost efficiency at billion-scale.

12Security & Compliance

Authentication

• API keys or JWT tokens for app backends.

Device Token Security

• Encrypt device tokens at rest.
• Tokens are issued by APNs/FCM, not by us, so we can't rotate them. Accept the new token whenever the app re-registers, and delete tokens the provider reports as invalid.
• Rotate *our* provider credentials (APNs signing key, FCM service account) on a schedule.

User Preferences

• Honor OS-level permission (the user can disable notifications entirely) and per-category opt-outs in our own preferences store, checked before every send.
• Get consent for marketing notifications where the law requires it (e.g. GDPR in the EU).
• Respect quiet hours in the user's time zone for non-urgent notifications, and cap marketing notifications per user per day.

Rate Limiting

• Prevent abuse from rogue clients.

13Interview Time Allocation (45 min)

• 5 min: Requirements & scope (functional, non-functional, scale).
• 10 min: HLD & architecture diagram (components, data flow).
• 5 min: Data model & key flows (single user, broadcast, scheduled).
• 10 min: Deep dive on fan-out for broadcast notifications.
• 8 min: Deep dive on rate limiting & throttling.
• 5 min: Reliability, failure handling, and scaling.
• 2 min: Trade-offs, security, wrap-up.

14SummaryWrap-up

• Core Challenges: Massive fan-out (1:1B), provider rate limits, reliable delivery at scale.
• Key Components:
• Fan-Out Service: Parallelize broadcast to 1B users in ~33 minutes.
• Rate Limiter: Token bucket per connection to respect APNs/FCM limits.
• Retry Logic: Exponential backoff via a delayed retry queue; dead letter queue after max retries.
• Analytics: Track delivery metrics for optimization.
• Scaling Strategy: Horizontal worker scaling, partitioned queue, cached device tokens, multi-provider connections.
• Reliability: Idempotency checks, circuit breaker, at-least-once delivery with deduplication.

This design handles 10 billion notifications/day with 500K/sec peak throughput while ensuring 98%+ delivery success.