Instagram

0Problem RestatementProblem

Design a social media platform like Instagram where users can post photos/videos, follow other users, like posts, search content, and view a personalized news feed. Core challenges include handling massive media uploads, efficiently generating news feeds for millions of users, storing and serving media at scale, and managing the social graph.

1RequirementsRequirements

1.1 Functional

• Post photos and videos: Users can upload and share media.
• Follow and unfollow users: Users can follow/unfollow other users.
• Like posts: Users can like/unlike posts.
• Search photos and videos: Search by captions, hashtags, and location.
• Generate news feed: View recent posts from followed users (start with reverse-chronological; ranking comes later, see Trade-offs).
• Comment on posts: Users can comment on posts.
• Share posts: Users can share posts with others.
• View user profiles: See user's posts, followers, following count.
• Notifications: Notify users of likes, comments, new followers.

1.2 Non-Functional

• Scalability: Support 1 billion+ users, 500M daily active users.
• Availability: 99.99% uptime.
• Low Latency: News feed load < 500ms, image load < 200ms.
• Storage: Petabytes of photos and videos.
• Consistency: Eventual consistency acceptable for likes/follows.
• Reliability: No data loss for uploaded media.

1.3 Scale Estimates

• Users: 1 billion total, 500M daily active users (DAU).
• Storage per day: Assume 90% photos, 10% videos → 90M × 2 MB + 10M × 20 MB = 180 TB + 200 TB ≈ 380 TB/day of originals. Resized variants and transcodes add roughly 50% → ~570 TB/day.
• Total storage: ~570 TB/day × 365 days × 5 years ≈ 1 EB, before replication. This is why media goes to object storage with cold tiers, not a database.
• News feed requests: 500M DAU × 10 feeds/day = 5B feed requests/day (~60K/sec).

•API Specifications

Post Management APIs

• POST /api/posts - Create new post with photo/video and caption
• GET /api/posts/{post_id} - Get post details including media, caption, likes count
• DELETE /api/posts/{post_id} - Delete a post
• PUT /api/posts/{post_id} - Edit post caption or location

Social Interaction APIs

• POST /api/follow - Follow a user
• DELETE /api/follow/{user_id} - Unfollow a user
• POST /api/likes - Like a post
• DELETE /api/likes/{post_id} - Unlike a post
• POST /api/comments - Comment on a post
• GET /api/comments/{post_id} - Get comments for a post

Feed & Discovery APIs

• GET /api/feed - Get personalized news feed (posts from followed users)
• GET /api/explore - Get trending and recommended posts
• GET /api/search - Search posts by caption, hashtags, or location
• GET /api/hashtags/{tag} - Get posts with specific hashtag

User Profile APIs

• GET /api/users/{user_id} - Get user profile with stats (followers, following, posts count)
• GET /api/users/{user_id}/posts - Get all posts by a user
• GET /api/users/{user_id}/followers - Get list of followers
• GET /api/users/{user_id}/following - Get list of users being followed

Media APIs

• POST /api/upload - Upload photo or video
• GET /api/media/{media_id} - Get media file (returns CDN URL)

Notification APIs

• GET /api/notifications - Get user notifications (likes, comments, new followers)
• PUT /api/notifications/{notif_id}/read - Mark notification as read

2High-Level ArchitectureArchitecture

2.1 Overview

• Upload Pipeline: User → Upload Service → Object Storage (S3) → CDN.
• Feed Pipeline: User → Feed Service → Follow Graph → Posts DB → Ranked Feed.
• Social Graph: Follow Service manages followers/following relationships.
• Key components: Media storage (S3 + CDN), feed generation (fan-out), search indexing, notification system.

2.2 Architecture Diagram

%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart TB
    %% User and Frontend
    User["User - Mobile/Web"] -->|"1. upload media"| AG["API Gateway"]
    User -->|"13. view feed"| AG
    User -->|"17. follow/like"| AG
    
    %% Upload Flow
    AG -->|"2. POST /upload"| Upload["Upload Service"]
    Upload -->|"3. store original"| S3["Object Storage<br/>(S3)"]
    Upload -->|"4. trigger processing"| Queue["Message Queue<br/>(Kafka)"]
    Queue -->|"5. consume"| MediaProc["Media Processing<br/>(Resize, Thumbnail)"]
    MediaProc -->|"6. store variants"| S3
    MediaProc -->|"7. update metadata"| PostDB[(Posts DB)]
    S3 -->|"serve media"| CDN["CDN<br/>(CloudFront)"]
    CDN -->|"return to user"| User
    
    %% Post Creation
    Upload -->|"8. create post"| PostService["Post Service"]
    PostService -->|"9. INSERT post"| PostDB
    PostService -->|"10. fan-out to followers"| FeedGen["Feed Generator"]
    
    %% Follow Graph
    AG -->|"18. POST /follow"| FollowService["Follow Service"]
    FollowService -->|"19. update graph"| GraphDB[(Follow Graph DB<br/>Neo4j/Cassandra)]
    
    %% Feed Generation
    AG -->|"14. GET /feed"| FeedService["Feed Service"]
    FeedService -->|"15. get following list"| GraphDB
    GraphDB -->|"16. user IDs"| FeedService
    FeedService -->|"fetch recent posts"| PostDB
    FeedService -->|"check cache"| FeedCache["Feed Cache<br/>(Redis)"]
    FeedCache -->|"cached feed"| User
    
    %% Fan-out on Write
    FeedGen -->|"11. get followers"| GraphDB
    FeedGen -->|"12. write to feeds"| FeedCache
    
    %% Likes
    AG -->|"20. POST /like"| LikeService["Like Service"]
    LikeService -->|"21. record like"| LikeDB[(Likes DB)]
    LikeService -->|"22. batched count update (async)"| PostDB
    
    %% Search
    AG -->|"23. GET /search"| SearchService["Search Service"]
    SearchService -->|"24. query index"| ElasticSearch["Elasticsearch<br/>(Posts Index)"]
    PostDB -->|"index posts"| ElasticSearch
    
    %% Notifications
    LikeService -->|"25. emit event"| NotifQueue["Notification Queue"]
    FollowService -.->|"emit event"| NotifQueue
    NotifQueue -.->|"consume"| NotifService["Notification Service"]
    NotifService -.->|"push notification"| User
    
    %% Styling
    classDef user fill:#e3f2fd,stroke:#1976d2,stroke-width:2px;
    classDef core fill:#c8e6c9,stroke:#388e3c,stroke-width:2px;
    classDef storage fill:#fff9c4,stroke:#f57f17,stroke-width:2px;
    classDef cache fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px;
    
    class User,AG user;
    class Upload,PostService,FollowService,FeedService,LikeService,SearchService,MediaProc,FeedGen core;
    class S3,CDN,PostDB,GraphDB,LikeDB,ElasticSearch storage;
    class Queue,FeedCache,NotifQueue,NotifService cache;

3Components (what & why)

API Gateway

• Route requests to appropriate microservices.
• Authentication (JWT), rate limiting, SSL termination.

Upload Service

• Accept media uploads from clients.
• Generate unique post ID.
• Upload original media to S3.
• Trigger async media processing.

Media Processing Service

• Generate multiple image sizes (thumbnail, medium, full).
• Transcode videos to different resolutions.
• Extract metadata (EXIF, duration, resolution).
• Tools: FFmpeg for video, ImageMagick for images.

Object Storage (S3)

• Store original and processed media.
• Structure: /media/{user_id}/{post_id}/{variant}.jpg
• Redundancy: Multi-region replication.

CDN (Content Delivery Network)

• Serve media with low latency globally.
• Cache frequently accessed images/videos.
• Origin: S3.

Post Service

• Create, update, delete posts.
• Store post metadata (caption, hashtags, location, timestamp).
• Trigger feed fan-out for followers.

Posts DB

• Store post metadata: {post_id, user_id, media_url, caption, hashtags, location, likes_count, created_at}
• DB Choice: Cassandra (write-heavy, time-series data).
• Partitioning: By user_id or post_id.

Follow Service

• Handle follow/unfollow requests.
• Maintain bidirectional follow graph (followers and following).
• Validate follow limits (prevent spam).

Follow Graph DB

• Store follow relationships: (user_id, follows_user_id, timestamp)
• Graph DB: Neo4j or adjacency list in Cassandra.
• Queries: Get followers, get following, mutual followers.

Feed Service

• Generate personalized news feed for users.
• Two approaches:
• Fan-out on write: Pre-compute feeds when post is created (faster read).
• Fan-out on read: Compute feed on demand (slower read, less storage).
• Hybrid: Fan-out for regular users, on-read for celebrities.

Feed Cache (Redis)

• Cache pre-computed news feeds per user.
• Structure: {user_id: [post_id_1, post_id_2, ...]} (sorted by timestamp), capped at the newest ~500 entries per user.
• Not a short-TTL cache: with fan-out on write, this list *is* the precomputed feed — posts are pushed into it as they're created. If it expired after 30 minutes, those pushes would be lost. Keep it until it's trimmed, and only drop feeds for users inactive for weeks (rebuild them with fan-out on read when they return).

Like Service

• Record likes/unlikes.
• Increment/decrement like count on posts.
• Prevent duplicate likes (idempotency).

Likes DB

• Store like records: {user_id, post_id, created_at}
• Composite key: (user_id, post_id).

Search Service

• Index posts by caption, hashtags, location.
• Full-text search using Elasticsearch.
• Rank by relevance, recency, popularity.

Notification Service

• Send push notifications for likes, comments, new followers.
• Async processing via message queue.
• Channels: Push (mobile), email, in-app.

4Data ModelData model

User

User(
  user_id,
  username,
  email,
  profile_pic_url,
  bio,
  follower_count,
  following_count,
  created_at
)

Post

Post(
  post_id,
  user_id,
  media_url,
  media_type,  -- photo, video
  caption,
  hashtags[],
  location,
  likes_count,
  comments_count,
  created_at
)

Follow

Follow(
  follower_user_id,
  following_user_id,
  created_at
)
-- Index on follower_user_id for "get following"
-- Index on following_user_id for "get followers"

Like

Like(
  user_id,
  post_id,
  created_at
)
-- Composite primary key: (user_id, post_id)

News Feed (Cache)

{
  "user_id": "123",
  "feed": [
    {"post_id": "p1", "timestamp": "2023-10-15T10:00:00Z"},
    {"post_id": "p2", "timestamp": "2023-10-15T09:30:00Z"},
    ...
  ]
}

5Key FlowsFlows

5.1 Upload Photo/Video Flow

1. User selects photo/video and adds caption.
2. Client compresses media (optional) and calls POST /upload.
3. Upload Service generates unique post_id.
4. Upload Service uploads media to S3 and returns URL.
5. Upload Service enqueues media processing job (resize, thumbnail).
6. Media Processor generates variants and stores in S3.
7. Post Service creates post record in Posts DB.
8. Feed Generator fans out post to all followers' feeds (async).

5.2 Follow User Flow

1. User A clicks "Follow" on User B's profile.
2. Client calls POST /follow {user_id: B}.
3. Follow Service inserts record: (follower: A, following: B).
4. Update follower_count for B, following_count for A.
5. Notification Service sends notification to User B.

5.3 Like Post Flow

1. User clicks "Like" on a post.
2. Client calls POST /like {post_id}.
3. Like Service checks if already liked (idempotency).
4. If not liked, insert record in Likes DB.
5. Increment likes_count asynchronously: emit a like event, aggregate in a counter service (e.g., sharded Redis counters), and flush totals to the Posts DB every few seconds. Writing likes_count on the post row for every like would turn one viral post into a single hot row at thousands of writes/sec.
6. Notification Service notifies post owner.

5.4 Read News Feed Flow (Hybrid)

1. User requests news feed: GET /feed.
2. Feed Service reads the user's precomputed feed from Redis (filled by fan-out on write).
3. Feed Service fetches recent posts from the celebrities the user follows (fan-out on read — celebrity posts are never pushed; see Deep Dive A).
4. Merge both lists and sort by timestamp (or rank).
5. Feed missing (new or long-inactive user, or Redis lost the key): rebuild it once by querying recent posts from everyone they follow, store it, then continue as above.
6. Fetch post metadata (media URL, caption, likes count) for the top 20 posts.
7. Return to client.

5.5 Search Posts Flow

1. User searches: "sunset beach #travel".
2. Client calls GET /search?q=sunset+beach+%23travel.
3. Search Service queries Elasticsearch index.
4. Elasticsearch returns matching post IDs ranked by relevance.
5. Fetch post metadata from Posts DB.
6. Return results to client.

6Deep Dive A: News Feed Generation (Fan-out on Write vs Read) (~10 mins)Deep dive

Problem

Generating news feed efficiently for 500M users, some following thousands of accounts.

Approach 1: Fan-out on Write (Push Model)

How It Works

• When post is created: Push post to all followers' feeds immediately.
• Feed lookup: Simple read from pre-computed feed cache.

Implementation

1. User A posts a photo.
2. Feed Generator fetches User A's followers (e.g., 1000 followers).
3. For each follower, append post_id to their feed cache in Redis: LPUSH user:{follower_id}:feed post_id.
4. When follower requests feed, read directly from cache.

Pros

• Fast reads: Feed already computed, just fetch from cache.
• Simple read logic: No complex queries.

Cons

• Slow writes: If user has 10M followers (celebrity), fan-out takes time.
• High storage: Duplicate post_id stored in millions of caches.
• Inactive users: Waste resources computing feeds for inactive users.

Approach 2: Fan-out on Read (Pull Model)

How It Works

• When post is created: Do nothing.
• Feed lookup: Compute feed on-demand by querying recent posts from followed users.

Implementation

1. User requests feed.
2. Fetch list of followed users from Follow Graph.
3. Query Posts DB for recent posts from those users (e.g., last 100 posts each).
4. Merge and sort by timestamp.
5. Cache result for short TTL.

Pros

• Fast writes: No fan-out overhead.
• No wasted computation: Only compute for active users.

Cons

• Slow reads: Complex query aggregating posts from many users.
• Hotspot users: Users following 1000s create expensive queries.

Approach 3: Hybrid (Best of Both)

Strategy

• Regular users (< 1M followers): Fan-out on write.
• Celebrities (> 1M followers): Fan-out on read.
• Detection: Flag users with > 1M followers in database.

Implementation

1. Post creation:

• If user has < 1M followers: Fan-out to all followers (push).
• If user has > 1M followers: Skip fan-out.

• Fetch pre-computed feed from cache (fan-out on write).
• Fetch recent posts from followed celebrities (fan-out on read).
• Merge both and sort.

Pros

• Fast reads for most users.
• Avoids fan-out bottleneck for celebrities.

Feed Generation Architecture

%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart TD
    NewPost["User Creates Post"] -->|"regular user"| FanOut["Fan-out Service"]
    NewPost -->|"celebrity user"| Skip["Skip Fan-out"]
    
    FanOut -->|"get followers"| GraphDB["Follow Graph DB"]
    GraphDB -->|"1000 followers"| FanOut
    FanOut -->|"write to each feed"| Redis["Redis Feed Cache"]
    
    UserReq["User Requests Feed"] --> Check["Check Cache"]
    Check -->|"cache hit"| Cached["Return Cached Feed"]
    Check -->|"cache miss"| Compute["Compute Feed"]
    
    Compute -->|"get following (regular users)"| Redis2["Redis"]
    Compute -->|"get following (celebrities)"| Query["Query Posts DB"]
    
    Redis2 -->|"pre-computed feed"| Merge["Merge & Sort"]
    Query -->|"celebrity posts"| Merge
    Merge --> Cache["Cache Result"]
    Cache --> Return["Return Feed"]
    
    classDef fanout fill:#c8e6c9,stroke:#388e3c,stroke-width:2px;
    classDef compute fill:#e3f2fd,stroke:#1976d2,stroke-width:2px;
    
    class FanOut,GraphDB fanout;
    class Compute,Query,Merge compute;

7Deep Dive B: Media Storage & CDN (~8 mins)Deep dive

Problem

Store and serve petabytes of images/videos with low latency globally.

Storage Architecture

Object Storage (S3)

• Structure: s3://instagram-media/{user_id}/{post_id}/{variant}.jpg
• Variants:
• Original (uploaded)
• Thumbnail (150×150)
• Medium (640×640)
• Full (1080×1080)
• Videos: Multiple resolutions (360p, 720p, 1080p).

Compression

• Images: JPEG with 85% quality, WebP for newer clients.
• Videos: H.264 codec, adaptive bitrate.

Replication

• Multi-region: Store in 3 regions for redundancy.
• Cross-region replication: Async replication to other regions.

CDN Strategy

Edge Caching

• CDN caches images/videos at edge locations near users.
• Cache hit: Serve from edge (low latency, < 50ms).
• Cache miss: Fetch from S3 origin, cache at edge.

Cache TTL

• Images: 1 year (immutable).
• Videos: 1 month.

Pre-warming

• Popular posts (trending, viral) pre-loaded to CDN edge.

Upload Flow with Processing

1. User uploads 10 MB photo.
2. Upload Service stores in S3 as original.jpg.
3. Triggers async processing job.
4. Media Processor:

• Generates thumbnail (150×150, 20 KB).
• Generates medium (640×640, 200 KB).
• Generates full (1080×1080, 1 MB).

1. Updates Posts DB with URLs.

Serving Flow

1. User scrolls feed, client requests: https://cdn.instagram.com/media/user123/post456/medium.jpg.
2. CDN checks edge cache:

• Hit: Return image (50ms).
• Miss: Fetch from S3 (200ms), cache at edge, return.

Media Storage Architecture

%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
    User["User Upload<br/>(10 MB)"] --> Upload["Upload Service"]
    Upload --> S3Origin["S3 (Origin)<br/>original.jpg"]
    
    S3Origin --> Processor["Media Processor"]
    Processor --> S3Thumb["S3<br/>thumbnail.jpg (20 KB)"]
    Processor --> S3Med["S3<br/>medium.jpg (200 KB)"]
    Processor --> S3Full["S3<br/>full.jpg (1 MB)"]
    
    S3Med --> CDN["CDN Edge<br/>(CloudFront)"]
    CDN -->|"cache hit"| UserView["User Views Feed"]
    CDN -->|"cache miss"| S3Med
    
    classDef storage fill:#fff9c4,stroke:#f57f17,stroke-width:2px;
    classDef cdn fill:#ffe0b2,stroke:#e65100,stroke-width:2px;
    
    class S3Origin,S3Thumb,S3Med,S3Full storage;
    class CDN cdn;

8Deep Dive C: Follow Graph & Scalability (~7 mins)Deep dive

Problem

Manage follow relationships for 1B users, support queries like "get followers", "get following", "mutual friends".

Data Structure

Adjacency List (Cassandra)

User A:
  following: [B, C, D, E]
  followers: [F, G, H]

Graph Database (Neo4j)

(UserA)-[:FOLLOWS]->(UserB)
(UserC)-[:FOLLOWS]->(UserA)

Choice: Cassandra (Adjacency List)

• Reason: Better horizontal scaling for simple queries.
• Trade-off: Complex graph queries (mutual friends) harder.

Schema

FollowingList(
  user_id,
  following_user_id,
  created_at
)
-- Partition key: user_id
-- Clustering key: following_user_id

FollowersList(
  user_id,
  follower_user_id,
  created_at
)
-- Partition key: user_id
-- Clustering key: follower_user_id

Queries

Get Following

SELECT following_user_id FROM FollowingList WHERE user_id = 'A'

Get Followers

SELECT follower_user_id FROM FollowersList WHERE user_id = 'A'

Denormalization

• Store both FollowingList and FollowersList for fast lookups.
• Write amplification: Each follow writes to 2 tables.
• Benefit: Fast reads for both "who I follow" and "who follows me".

Handling Celebrities

• Problem: Celebrity with 100M followers creates hot partition.
• Solution:
• Shard followers list (e.g., 1M followers per shard).
• Store in blob storage for very large lists.

Follow Graph Architecture

%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart TD
    UserA["User A Follows User B"] --> FollowSvc["Follow Service"]
    
    FollowSvc -->|"INSERT"| Following["FollowingList DB<br/>(user_id: A, following: B)"]
    FollowSvc -->|"INSERT"| Followers["FollowersList DB<br/>(user_id: B, follower: A)"]
    
    FollowSvc -->|"increment"| CountA["User A: following_count++"]
    FollowSvc -->|"increment"| CountB["User B: follower_count++"]
    
    Query1["Get A's Following"] --> ReadFollowing["Read FollowingList<br/>(user_id: A)"]
    Query2["Get B's Followers"] --> ReadFollowers["Read FollowersList<br/>(user_id: B)"]
    
    classDef write fill:#c8e6c9,stroke:#388e3c,stroke-width:2px;
    classDef read fill:#e3f2fd,stroke:#1976d2,stroke-width:2px;
    
    class FollowSvc,Following,Followers write;
    class Query1,Query2,ReadFollowing,ReadFollowers read;

9Scaling & Performance (~5 mins)Scale

Horizontal Scaling

• Upload Service: Stateless, scale with load balancer.
• Feed Service: Stateless, cache-heavy.
• Databases: Shard by user_id or post_id.

Database Sharding

• Posts DB: Shard by user_id (all posts from same user in one shard).
• Follow Graph: Shard by user_id.
• Likes DB: Shard by post_id.

Caching Layers

• Feed Store: Redis, precomputed feeds capped per user (not TTL-expired).
• Rendered Feed Page Cache: Short TTL (~1 min) for the hydrated first page.
• CDN: Edge caching for media.
• Database Query Cache: Memcached for user profiles.

Performance Metrics

• Feed load: < 500ms (P99).
• Image load: < 200ms from CDN.
• Upload: < 5 seconds for 10 MB photo.
• Throughput: 60K feed requests/sec.

10Failure Modes & Recovery

Upload Failure

• Retry: Client retries with exponential backoff.
• Partial Upload: Use multipart upload (S3).

Feed Cache Failure (Redis Down)

• Fallback: Compute feed on-read from Posts DB.
• Graceful Degradation: Slower but functional.

Database Failure

• Replication: Use read replicas for failover.
• Multi-region: Route to healthy region.

CDN Failure

• Multi-CDN: Use multiple CDN providers.
• Fallback to Origin: Serve directly from S3.

11Trade-offs & AlternativesTrade-offs

Fan-out on Write vs Read

• Write: Faster reads, slower writes, more storage.
• Read: Faster writes, slower reads, less storage.
• Choice: Hybrid (both based on follower count).

SQL vs NoSQL

• SQL: ACID transactions, complex queries.
• NoSQL: Horizontal scaling, high write throughput.
• Choice: NoSQL (Cassandra) for posts, follows, likes.

Chronological vs Algorithmic Feed

• Chronological: Simple, fair, but may miss important content.
• Algorithmic: Personalized, better engagement, but less transparent.
• Instagram: Moved to algorithmic for higher engagement.

12Security & Privacy

Authentication

• JWT tokens, OAuth for third-party apps.

Authorization

• Users can only access their own feed and followed users' posts.
• Private accounts: Require follow approval.

Content Moderation

• ML models detect inappropriate content (nudity, violence).
• User reporting and manual review.

Data Privacy

• GDPR compliance: Users can download/delete their data.

13Interview Time Allocation (45 min)

• 5 min: Requirements & scope (functional, non-functional, scale).
• 10 min: HLD & architecture diagram (upload, feed, social graph).
• 5 min: Data model & key flows (upload, follow, like, feed).
• 10 min: Deep dive on news feed generation (fan-out strategies).
• 8 min: Deep dive on media storage & CDN.
• 5 min: Follow graph, scaling, failure handling.
• 2 min: Trade-offs, security, wrap-up.

14SummaryWrap-up

• Core Challenges: News feed generation at scale, petabyte media storage, managing social graph for billions.
• Key Components:
• Upload Pipeline: S3 storage + async processing + CDN delivery.
• Feed Generation: Hybrid fan-out (write for regular users, read for celebrities).
• Social Graph: Cassandra adjacency lists for follow relationships.
• Media Storage: S3 origin + CloudFront CDN with edge caching.
• Scaling Strategy: Database sharding by user_id, Redis feed cache, CDN for media, horizontal service scaling.
• Performance: 500ms feed load, 200ms image load, 60K requests/sec.

This design supports 1B users posting 100M photos/videos daily with personalized feeds and global media delivery.