Content Moderation System (Harmful Post Detection)

0Problem RestatementProblem

Design a system that finds harmful content on a social platform. For example (asked at Meta): posts or ads selling weapons, as well as hate speech, nudity, spam and scams. Content can be text, images or video. The system checks new posts when they are uploaded. Clear violations are blocked, uncertain ones go to human reviewers, and users can report posts and appeal decisions.

The core tension: if we block too much, innocent people get hurt (false positives); if we block too little, harmful content spreads (false negatives).

1RequirementsRequirements

1.1 Functional

• Score every new post, comment, ad and profile for multiple policies (weapons, hate, nudity, spam, ...).
• Take actions: allow, reduce reach, blur with a warning, remove, ban the account.
• Human review queues with tools and priorities.
• User reports and appeals.
• Re-scan old content when policies or models change.

1.2 Non-Functional

• Low latency at upload for high-risk checks (a few hundred ms), so bad content doesn't go live even briefly.
• Scale: billions of items per day.
• Accuracy tuned per policy (weapons sales: high recall; borderline humor: careful).
• Auditability: why was this removed?

1.3 Scale Estimates

• 2B new items/day ≈ 23K/sec, peaks of 100K/sec.
• Images: ~40% of items. Video is fewer but much heavier (sample frames).
• If 0.5% go to human review, that's 10M reviews/day, too many. So ML must handle most cases, and humans handle the uncertain middle.

1.4 API Design

2High-Level ArchitectureArchitecture

2.1 Overview

• Fast inline checks (at upload): hash matching against known bad content (e.g., PhotoDNA-style perceptual hashes), blocked keywords, and a fast text and image classifier.
• Async deep checks: heavier multimodal models (text + image + video frames + author signals), run within seconds to minutes after posting.
• Decision engine: combines scores with per-policy thresholds and account history, then chooses an action.
• Review system: queues by policy, language and severity, with a reviewer UI.
• Feedback loop: reviewer decisions become training labels.

2.2 Architecture Diagram

%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
    UP["New post / ad"] --> FAST["Fast checks: hash match, keywords, light model"]
    FAST -->|"clear violation"| BLK["Block"]
    FAST -->|"publish"| PUB["Post goes live"]
    FAST --> K[("Kafka - content events")]
    K --> DEEP["Deep multimodal models"]
    DEEP --> DEC["Decision engine - thresholds per policy"]
    RPT["User reports"] --> DEC
    DEC -->|"high score"| ACT["Remove / reduce reach"]
    DEC -->|"uncertain"| RQ[("Human review queues")]
    RQ --> REV["Reviewers"]
    REV --> ACT
    REV --> LBL[("Labels for retraining")]
    LBL --> TRAIN["Model training"]

3Data ModelData model

content_scores:  content_id, model_version, policy, score, created_at
decisions:       content_id, action, policy, source (model|reviewer|appeal), actor, reason, created_at
review_tasks:    task_id, content_id, policy, priority, language, status, assigned_to, sla_due
known_hashes:    hash, policy, source   -- known violating images/videos

4Key FlowsFlows

4.1 A new post with an image of a gun for sale

1. Inline: check the perceptual hash against known bad images. No match. The text classifier sees "selling", "DM for price" and a weapon-related term, giving a weapons-sale score of 0.7, which is not high enough to block instantly.
2. The post is published (or held briefly, for high-risk surfaces like ads) and an event is sent to Kafka.
3. Deep model: an image model detects a firearm, the text is a sales intent, and the account is new with a pattern of similar posts. The combined score is 0.95.
4. The decision engine applies the weapons-sale threshold (block at 0.9) → remove and notify the author with the reason and an appeal link.
5. If the score were 0.6–0.9, it would go to the review queue instead, with priority based on predicted reach (viral posts first).

4.2 User report

Reports add a signal and can push content into review sooner. Many reports from trusted reporters raise priority.

5Deep Dive A — What to do with a score of 0.7Deep dive

The classifier returns a number. Turning that number into an action is where a moderation system is actually designed, and where it most often goes wrong.

%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
  S["Score 0.79 - weapons sale"] --> PUB["Published - real harm"]
  S2["Score 0.81 - satire about weapons"] --> DEL["Deleted - wrongful removal"]
  DEL --> APP["Appeal, press coverage, reinstated a week later"]

%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
  SC["Policy score"] --> H{"Very high"}
  H -->|"yes"| RM["Remove and notify - clear violation"]
  H -->|"no"| M{"Moderate"}
  M -->|"yes"| SOFT["Reduce reach, add a warning interstitial"]
  M -->|"no"| L{"Uncertain, high reach"}
  L -->|"yes"| Q["Human review queue - ranked by views at risk"]
  L -->|"no"| PUB["Publish"]
  SOFT --> Q

6Deep Dive B — Adversaries and scaleScale

• Evasion: people misspell words, put text in images, crop or recolor images. Countermeasures: OCR on images, perceptual hashes that survive small edits, embeddings that capture meaning, and account-level signals (new account, many similar posts, links to known bad groups).
• Video: sample frames (e.g., 1 per second plus scene changes) and audio transcripts, and scan the most-viewed videos more deeply.
• Model updates: when a new model or policy ships, re-scan recent content in the background, starting with the most-viewed.
• Reviewer wellbeing and quality: blur by default, limit exposure time, and double-review a sample to measure reviewer accuracy.

7Trade-offs & AlternativesTrade-offs

8Common Follow-up QuestionsFollow-ups

• "Ads vs posts?" Ads are paid and higher risk, so review them before they go live. Posts go live and are checked right after, unless they're high-risk.
• "Multiple languages?" Multilingual models, plus review queues routed by language.
• "How do appeals work?" A different reviewer re-checks the item. If overturned, restore the content and add the case as a training label.

9Wrap-UpWrap-up

Run fast checks at upload (hash matching, keywords, a light model) to stop obvious violations, then deep multimodal models asynchronously. A decision engine applies per-policy thresholds to allow, limit, remove or send to human review, prioritized by reach. Feed reviewer decisions back into training, fight evasion with OCR, perceptual hashes and account signals, and measure success with sampled prevalence.