News Aggregator (Google News)

0Problem RestatementProblem

Design a news aggregation service like Google News. It collects articles from thousands of publishers, groups articles about the same story, and shows each user a fresh feed. Users can browse by topic (Sports, Tech), open an article (which takes them to the publisher's site), and get a personalized "For you" feed.

This problem was asked many times at Rippling, in several versions:

• Focus on pulling articles on a schedule, storing them, and reading by topic.
• Make sure a feed never shows duplicates, even when each publisher gives each article a unique URL.
• Design the frontend feed (React, pagination, virtualization, offline).

1RequirementsRequirements

1.1 Functional

• Fetch new articles from publisher feeds (RSS/APIs) regularly.
• Store article metadata: URL, title, summary, image, publisher, topic, publish time.
• Group articles about the same story, and show one story with "5 more sources".
• Serve topic feeds and a personalized feed, newest or most relevant first, paginated.

1.2 Non-Functional

• Fresh: breaking news appears within a few minutes.
• Fast reads: feed loads in under ~200 ms.
• Read-heavy: many more reads than writes.
• No duplicates within a user's feed.

1.3 Scale Estimates

• 50,000 sources, ~2M new articles/day ≈ 25 articles/sec (small).
• 50M daily users × 10 feed loads = 500M reads/day ≈ 6K/sec, with spikes during big news.
• Storage: 2M × 2 KB = 4 GB/day. We only need recent articles hot (e.g., 30 days).

1.4 API Design

2High-Level ArchitectureArchitecture

2.1 Overview

• Fetch Scheduler: decides when to poll each source.
• Fetchers: download feeds, parse new items and push them to a queue.
• Ingest pipeline: normalizes URLs, deduplicates, classifies the topic, and clusters articles into stories.
• Article store: articles and stories (SQL or a document DB).
• Feed builder: precomputes topic feeds (sorted lists of story IDs) in Redis.
• Feed API: reads the precomputed lists, applies personalization and filters out seen items, then hydrates details from cache.

2.2 Architecture Diagram

%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
    SCH["Fetch Scheduler"] -->|"due sources"| FQ[("Fetch queue")]
    FQ --> F["Fetchers"]
    F --> PUB["Publisher RSS / APIs"]
    F --> IQ[("Kafka - new articles")]
    IQ --> P["Ingest: normalize, dedupe, topic, cluster"]
    P --> DB[("Articles + Stories DB")]
    P --> FB["Feed builder"]
    FB --> R[("Redis - topic feeds")]
    U["Users"] --> API["Feed API"]
    API --> R
    API --> C[("Article cache")]
    C --> DB

3Data ModelData model

sources:   source_id, feed_url, avg_publish_interval, last_fetched_at, next_fetch_at, etag
articles:  article_id, canonical_url (unique), url_hash, title, summary, image_url,
           source_id, topic_id, published_at, story_id, content_fingerprint
stories:   story_id, headline_article_id, topic_id, first_seen_at, last_updated_at, article_count
Redis:     feed:topic:{topic_id} → sorted set of story_id by score (recency/importance)

4Key FlowsFlows

4.1 Scheduling pulls

• Each source has next_fetch_at. The scheduler picks due sources (an index on next_fetch_at) and queues them.
• Adaptive frequency: a source that publishes every 5 minutes is polled every few minutes, while one that posts weekly is polled every few hours. If a fetch finds nothing new, wait longer next time. If it finds a lot, poll sooner.
• Use HTTP ETag/If-Modified-Since so unchanged feeds cost almost nothing.
• Be polite: limit concurrent requests per publisher domain.

4.2 Ingesting an article

1. Normalize the URL (remove tracking parameters like utm_source, lowercase the host) and hash it. If the hash exists, skip it (exact duplicate).
2. Classify the topic, using the source's category or a text classifier.
3. Cluster into a story: compare the article with recent stories in the same topic using text similarity (e.g., embeddings or MinHash on the title and summary). If it's close enough, attach it to that story. Otherwise, create a new story.
4. Update the story's score and add or bump it in feed:topic:{id}.

4.3 Reading a feed

Read story IDs from the Redis sorted set with cursor pagination, fetch story details from the cache, and return them. The "for you" feed merges the topic feeds the user follows and re-ranks them with the user's interests.

5Deep Dive A — Keeping the same story out of the feed twiceDeep dive

Every publisher gives the same wire story its own URL, its own headline and its own timestamp. A naive feed shows the reader the Reuters version, the AP version and four newspapers that reprinted it.

%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
  A["New article"] --> N["Normalise - strip nav, ads, tracking params"]
  N --> S["SimHash of body text"]
  S --> Q{"Within 3 bits of an existing hash?"}
  Q -->|"yes"| D["Same article - keep the earliest"]
  Q -->|"no"| K["New article"]

6Deep Dive B — Freshness, spikes and the frontendDeep dive

• Breaking news spike: feeds are precomputed and served from Redis plus CDN caching for anonymous topic pages (a 30-second TTL), so traffic spikes don't hit the DB.
• Precompute vs on-read: topic feeds are shared by millions of people, so precompute them. The personal feed is cheap to assemble on read from a few topic lists.
• Frontend (React): fetch the first page quickly and load more on scroll (infinite scroll with cursors). Render only visible rows (virtualization) so long feeds stay smooth. Cache the last feed for offline reading. Show hover cards for hashtags and mentions with lazy loading.

7Trade-offs & AlternativesTrade-offs

8Common Follow-up QuestionsFollow-ups

• "How do you rank?" Combine recency, the number of sources covering the story (importance), source quality and user interest. Old stories decay over time.
• "Paywalled or removed articles?" Store a status. A periodic check (or publisher signals) marks removed articles and hides them.
• "Local news?" Tag articles with locations and add a location filter to the feed.

9Wrap-UpWrap-up

Poll publisher feeds with an adaptive scheduler, send new items through Kafka to an ingest pipeline that normalizes URLs, fingerprints content and clusters articles into stories, and store them. Precompute topic feeds as Redis sorted sets, serve them with cursor pagination and caching, and build the personal feed by merging and re-ranking topic lists while filtering out already-seen stories.