Messenger App

0Problem RestatementProblem

Design a real-time messaging system (like WhatsApp / Messenger) that supports 1:1 and group chat, message delivery guarantees, presence, typing indicators, push notifications, and scales to hundreds of millions of users. Focus on architecture, delivery semantics, storage, and scaling.

1RequirementsRequirements

1.1 Functional

• 1:1 and group text messaging (send/receive).
• Delivery receipts: sent, delivered, read.
• Presence (online/offline) and typing indicators.
• Push notifications for offline devices.
• Message history sync across devices.
• Support attachments (images, video) — store in object store.

1.2 Non-Functional

• Low latency: real-time <200–500ms for active sessions.
• High throughput: hundreds of millions of users, billions of messages/day.
• Durable storage for history.
• Strong privacy (optionally E2EE).
• High availability, region-distributed.

2High-Level ArchitectureArchitecture

2.1 Overview (what components connect)

• Clients (mobile/web) ↔ Gateway (Auth, TLS) ↔ Connection Layer (WebSocket / MQTT)
• Connection Layer ↔ Routing / Session Service ↔ Delivery Workers
• Delivery Workers ↔ Persistent Message Store (Primary DB + Cold store)
• Delivery Workers ↔ Fanout / Push Service (APNs / FCM)
• Presence Service (Redis) for online status
• Attachment Service ↔ Object Store (S3) + CDN
• Control Plane: Auth Service, User/Profile Service, Group Service, ACLs

2.2 Flow Diagram

%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
    %% Clients hold a persistent connection; REST is only for history sync and media
    U[Sender - Mobile/Web] <-->|"WebSocket: send / ACK"| GW[Connection Gateway<br/>WebSocket / MQTT]
    R[Recipient - Mobile/Web] <-->|"WebSocket: deliver / receipts"| GW2[Connection Gateway<br/>node holding recipient]

    GW -->|"message + client_msg_id"| RT[Routing / Session Service]
    RT -->|"lookup user → gateway node"| SR[(Session Registry<br/>Redis)]
    RT -->|"append, keyed by conversation_id"| K[(Kafka<br/>per-conversation order)]
    K -->|"consume"| DW[Delivery Workers]
    DW -->|"persist"| DB[(Message Store<br/>Cassandra)]
    DW -->|"recipient online"| GW2
    DW -->|"recipient offline"| PUSH[Push Service<br/>APNs / FCM]
    PUSH -.->|"notification"| R

    U -->|"REST: history sync, signed upload URL"| API[API Gateway]
    API -->|"GET messages since seq_no"| DB
    API -->|"signed URL"| OBJ[(Object Store + CDN)]

    classDef userFlow fill:#f0f8ff,stroke:#333,stroke-width:1px;
    classDef coreService fill:#e0ffe0,stroke:#333,stroke-width:1px;
    classDef asyncFlow fill:#fff0f0,stroke:#333,stroke-width:1px;
    class U,R,API userFlow;
    class GW,GW2,RT,SR,DW,DB,OBJ coreService;
    class K,PUSH asyncFlow;

3Core Components (what & why)

Client

• Maintains persistent connection (WebSocket or MQTT) to receive messages in real-time.
• Falls back to push notifications when offline.
• Syncs message history and state (read receipts, deletions).

API Gateway

• TLS termination, authentication (OAuth/JWT), rate limiting, routing to connection brokers or REST endpoints.

Connection Layer (WebSocket / MQTT Brokers)

• Handles millions of concurrent TCP connections.
• Routes frames to Routing/Session Service; maintains per-connection metadata (node, heartbeat).
• Use clustered brokers (e.g., MQTT brokers, custom WS gateway) with sticky sessions.

Routing / Session Service

• Knows which gateway node holds which user session(s).
• Accepts incoming messages and routes to Delivery Workers or directly to target session node.
• Maintains session registry (user_id → gateway_node(s)) in a fast store (Redis / Consul).

Delivery Worker / Router

• Decouples ingest from persistence/delivery.
• Publishes incoming messages to primary store, adds to topic keyed by conversation_id, and fans out to subscribers.
• Handles retries, ordering, deduplication, and acknowledgment tracking.

Message Queue / Stream (Kafka)

• Durably buffer messages; keyed by conversation_id for per-conversation ordering.
• Enables replays, backpressure, multiple consumers (archiving, analytics).

Primary DB / Message Store

• Store message metadata and pointers. Options:
• Cassandra / DynamoDB / Scylla for multi-region, high-write throughput.
• Relational DB for small-scale / metadata.
• Store message: {message_id, conversation_id, sender_id, ts, body_ref or body, seq_no}. Delivery/read state is tracked per recipient (see Data Model), not on the message.

Presence Service

• Hot store (Redis) to track online/offline status and last-seen.
• Publishes presence changes for subscribers.

Typing Indicator Service

• Lightweight ephemeral events broadcasted to conversation participants via same routing infra.

Push Service

• Converts server-side events to APNs / FCM for offline devices.
• Manages tokens, platform differences, payload size, and rate-limits.

Attachment Service & Object Store

• Upload endpoint (signed URL) → object store (S3) → thumbnail generation → CDN for delivery.
• Messages reference attachment URLs.

Control Plane (Auth, Groups)

• Manages user profiles, group membership, ACLs, and access tokens.

4Data Model (minimal)Data model

Message

Message {
  message_id STRING PRIMARY KEY,
  conversation_id STRING,
  sender_id STRING,
  ts_ms BIGINT,
  body_ref STRING,   -- small text inline, larger stored in blob store
  attachments [url...],
  seq_no BIGINT       -- per-conversation sequence number
}

Receipt Cursors (per member, per conversation)

ReceiptCursor {
  conversation_id STRING,
  user_id STRING,
  delivered_up_to_seq BIGINT,
  read_up_to_seq BIGINT,
  PRIMARY KEY (conversation_id, user_id)
}

• A single status field on the message can't say "delivered to Ann, read by Bob, not yet seen by Cara" in a group. Per-member cursors can: message *n* is read by everyone whose read_up_to_seq ≥ n.
• One write per member per receipt, instead of one per message per member.

Conversation / Thread

Conversation {
  conversation_id STRING,
  type ENUM('one_to_one','group'),
  participants [user_id...],
  created_at TIMESTAMP,
  last_activity_ts
}

Presence (Redis)

• Key: presence:{user_id} → {node_id, last_seen, status}

Delivery State (optional)

• Key: conv:{conversation_id}:cursor → last_seq_no persisted / per-device cursor

5Key FlowsFlows

5.1 Send Message (Active path)

1. Client sends message over WS with local client_msg_id and seq_no.
2. Gateway authenticates and forwards to Routing/Session Service.
3. Routing service forwards to Delivery Worker; Delivery Worker:

• Assigns server-side message_id, timestamp.
• Persists message to Primary DB (or logs to Kafka first and then persists).
• Publishes to conversation topic on Kafka (keyed by conversation_id).

1. Recipients' clients ACK reception; server marks delivered.
2. If recipient offline, Push Service triggers APNs/FCM.

5.2 Delivery Receipts & Read Receipts

• Client sends receipt events (delivered/read) back to server.
• Server advances that member's delivered_up_to_seq / read_up_to_seq cursor (receipts are cumulative: "read up to #57") and notifies the sender.

5.3 Message Sync (new device / reconnect)

• Client requests conversation history since last_seq_no.
• Server returns persisted messages from Primary DB or cold store.
• For long histories, provide pagination.

5.4 Attachments

• Client requests signed upload URL from Attachment Service.
• Uploads to S3; Attachment Service creates thumbnails, stores metadata.
• Message references attachment URL.

5.5 Presence & Typing

• Presence: client heartbeats to Presence Service; presence changes broadcast to friends.
• Typing: ephemeral events routed to active participants (no DB persist).

6Deep Dive A: Message Delivery Guarantees & Ordering (~8–10 mins)Deep dive

Delivery Semantics Options

• At-most-once: simplest, may lose messages.
• At-least-once: ensure message delivered, may cause duplicates — requires dedup logic.
• Exactly-once (per recipient): desirable but complex; achieve via idempotency + dedup.

Recommended Approach

• Producer-side idempotency: client attaches client_msg_id. Server persist operation uses idempotency to avoid duplicate writes.
• Kafka stream + consumer offsets: Use per-conversation partitioning (key = conversation_id) to preserve ordering within a conversation.
• Deduplication and idempotency at consumer: track seen client_msg_id or message_id in a small cache (Redis) to drop duplicates.
• Per-device sequencing: maintain per-device cursors to resume from last_seq_no reliably.

Ordering

• Ordering guaranteed per-conversation by using the same Kafka partition (keyed by conversation_id). For group chat, a single partition may become a hotspot — consider sharding by conversation_id + time windows or virtual partitions with ordering guarantees at application level.

Acks & End-to-End

• Use at-least-once delivery with idempotent writes and ACK semantics:
• Server sends ACK to sender when persisted.
• Server marks delivered when recipient ACKs.
• Sender sees sent/delivered/read states updated.

7Deep Dive B: Scaling, Sharding & Fanout (~8–10 mins)Deep dive

Scaling Writes & Reads

• Partitioning: partition message streams by conversation_id. Use consistent hashing to distribute partitions across Kafka brokers and delivery workers.
• Hot conversation mitigation: large groups can overload a single partition. Strategies:
• Use group-specific optimizations: tiered delivery (batch updates), ephemeral sampling, or multiple partitions per large group + merge ordering at consumers.
• Offload to push-based summary updates for very large groups.

Connection & Socket Scaling

• Use many WebSocket gateway nodes behind LB. Keep sticky sessions. Store mapping user_id → node_id in Redis (session registry).
• Cross-node fanout: if recipient on different node, the delivery worker forwards message to that node (via internal RPC/queue) which then pushes to client.

Fanout Strategies

• Direct push: push to each active session (works for small groups / 1:1).
• Interest topics: consumers subscribe to conversation topics (WS gateways subscribe to Kafka topics for conversations they have sessions for). This reduces central fanout work.
• Batching & compression: group messages into frames for same recipient; compress payload for large bursts.

Storage Scaling

• Use wide-column DB (Cassandra / DynamoDB) for high-write throughput, partitioned by conversation_id and time.
• Use TTL or cold archiving for older messages (move to object store).

8Offline & Push Notification Handling (~4 mins)

• When recipient offline, queue message for push.
• Push payload should be minimal (summary) to avoid leaking content; for privacy, fetch server-side or use encrypted payloads.
• Respect platform rate-limits and backoff when push failures occur.

9Security & Privacy (~3 mins)

• TLS for transport.
• Auth tokens (short-lived).
• Optionally End-to-End Encryption (E2EE):
• Server routes only encrypted blobs; cannot read contents.
• Key management: Signal protocol for 1:1; for groups, Sender Keys (each member encrypts once for the group) or the newer MLS protocol — pairwise encryption to every member doesn't scale to large groups.
• Trade-offs: server-side features (search, spam detection, indexing) limited with E2EE.
• Content moderation via metadata, client-side reporting, and optional server-side scanning for non-E2EE deployments.

10Failure Modes & Recovery (~3 mins)

• Gateway node crash: sessions reconnect; session registry updated; undelivered messages replayed from Kafka or DB.
• Delivery worker crash: Kafka ensures messages retained; consumers resume from offsets.
• DB outage: degrade to readonly for history, continue in-memory buffering and accept messages to Kafka with later persistence.
• Network partitions: favor availability for messaging delivery; reconcile using sequence numbers and reconciliation jobs.

11Observability & SLOs (~2 mins)

• Metrics: message ingests/s, delivery latency p50/p95/p99, queue backlog, active connections, ack rates, push success rates.
• Tracing: propagate trace ids through send → persist → fanout → ack.
• SLO examples: p95 in-app delivery <500ms, message durability 99.999% (depends on storage SLAs).

12Trade-offs & Alternatives (~2 mins)Trade-offs

• E2EE vs server-side features: E2EE maximizes privacy but limits server functionality.
• Relational vs NoSQL store: NoSQL (Cassandra/DynamoDB) favored for writes and horizontal scaling.
• Push vs pull for large groups: pull/chunked sync for huge groups reduces server cost.

13Interview Time Allocation (45 min)

• 5 min: Clarify requirements & scope
• 10 min: High-level architecture drawing & walkthrough
• 10 min: Key flows (send/receive/sync/push)
• 15 min: Deep dives (Delivery guarantees & Scaling/Fanout)
• 5 min: Failure modes, security, summary

14SummaryWrap-up

• Core pieces: Connection layer (WS/MQTT), Routing/Session service, Delivery workers, Message stream (Kafka), Primary message store (Cassandra/DynamoDB), Presence (Redis), Push for offline.
• Use per-conversation partitioning for ordering; idempotency + deduplication for correctness; and efficient fanout patterns for scale.
• Consider E2EE for privacy with trade-offs. Monitor key metrics and design for graceful degradation during outages.