Problem RestatementProblem
Advertisers want to show ads to specific audiences: e.g., "users in the US who watched sci-fi in the last 30 days", "people on our customer list" (uploaded by the advertiser, matched privacy-safely), or "users similar to our best customers" (lookalikes). Netflix asked this. At ad-serving time, for each request, the system must quickly find which campaigns target this user, which means knowing which segments the user belongs to, in a few milliseconds.
RequirementsRequirements
- Define segments: rule-based (behavior, demographics, geography), advertiser lists, and lookalikes.
- Compute and refresh membership (daily for most, near real time for some).
- At serving time:
get_segments(user_id)→ the list of segment IDs, and then which campaigns target those segments (include and exclude rules). - Estimate audience size when an advertiser builds a segment ("~2.3M users").
- Respect privacy and consent (opt-outs, data retention, minimum audience sizes).
1.1 Scale Estimates
- 200M users, 50K active segments. The average user is in ~200 segments.
- Ad requests: 200K/sec → 200K membership lookups/sec at under ~3 ms.
ArchitectureArchitecture
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
EV["User events + profiles"] --> LAKE[("Data lake")]
ADV["Advertiser lists (hashed)"] --> MATCH["Privacy-safe matching"]
LAKE --> SEG["Batch segment builder - Spark"]
MATCH --> SEG
K[("Real-time events")] --> RTS["Streaming segment updater"]
SEG --> MEM[("Membership store - user to segments")]
RTS --> MEM
SEG --> BM[("Segment bitmaps - sizes, overlaps")]
AS["Ad server"] -->|"get_segments(user)"| MEM
AS --> IDX["Campaign index: segment to campaigns"]
UI["Audience builder UI"] --> BMDeep Dive — Storing who is in which audienceDeep dive
Two very different questions run against the same data: "which segments is this user in?" at ad-serving latency, and "how big is US AND sci-fi AND NOT existing-customers?" for an advertiser sizing a campaign.
A row per membership
segment_members(segment_id, user_id), one row per pair.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
T[("200M users x ~200 segments")] --> R["40 billion rows"]
SERVE["Ad request - which segments for user 42?"] --> IDX["Index lookup returning ~200 rows"]
IDX --> LAT["Per-request round trips inside a sub-100 ms budget"]
SIZE["US AND sci-fi AND NOT customers"] --> JOIN["Multi-way join over billions of rows"]
JOIN --> MIN["Minutes - advertiser is waiting in a form"]The relational shape makes both questions expensive: serving pays for a multi-row lookup on every ad request, and sizing pays for joins across billions of rows while somebody watches a spinner.
Pick one direction and index it
Store user → [segment ids] as a single compact value. Serving becomes one key-value lookup, which is exactly what the ad path needs.
Serving is solved. Sizing is now worse: answering "how many users are in this combination?" means scanning every user's segment list, because the only index runs the other way. The advertiser-facing half of the product has no efficient path at all.
Keep both views, each shaped for its question
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
BUILD["Segment build jobs"] --> V1[("user -> segments - KV store")]
BUILD --> V2[("segment -> users - Roaring bitmaps")]
AD["Ad request"] --> V1
V1 --> ONE["One lookup, ~200 sorted ints, delta-encoded"]
ADV["Advertiser: US AND sci-fi NOT customers"] --> V2
V2 --> BITS["Bitmap AND / ANDNOT"]
BITS --> MS["Milliseconds over 100M users"]user → segmentsfor serving. A key-value store keyed by user id, holding sorted, delta-encoded segment ids. One lookup per ad request. Roughly 200M users × ~200 segments × 2 bytes is about 80 GB — large, and it fits in a memory-heavy cluster.segment → usersas Roaring bitmaps. Map user ids to integers and store each segment as a compressed bitmap. Boolean targeting is then bitmapANDandANDNOT, which run in milliseconds across a hundred million users, and cardinality — the audience size — comes straight off the result.
Two derived views of the same memberships is a deliberate denormalisation, and the thing to say about it is how they stay consistent: both are built by the same job from the same source events, so neither is updated in place by the serving path. They can be a few minutes apart, which is fine — audiences are defined over windows like "in the last 30 days", so minute-level freshness was never part of the promise.
Building Segments
- Batch (daily): Spark jobs evaluate each rule-based segment over the data lake, produce bitmaps, then invert them into per-user lists and bulk-load the serving store (write a new version, then switch, so nothing is half-updated).
- Streaming (for fast segments like "viewed a car ad in the last hour"): a stream processor updates membership for affected users within seconds, with a TTL.
- Advertiser lists: the advertiser uploads hashed emails or phone numbers. We match them to our users with the same hashing (in a secure environment), and never reveal which users matched. Enforce minimum sizes (e.g., at least 1,000 matched) to prevent targeting individuals.
- Lookalikes: train a model or use embeddings to find users similar to a seed segment, and take the top N by score.
Serving: Matching Campaigns
- The ad request arrives with
user_id→ fetch the user's segment list (one KV read, cached locally for a few minutes for repeat requests). - A campaign index maps
segment_id → campaignsthat include it, plus exclusion rules. Candidate campaigns = the union over the user's segments, minus those whose exclusion segments the user is in. - Combine with other targeting (geo, device, time), frequency caps and pacing, then run the auction or selection.
Privacy and Correctness
- Consent: users who opted out of personalized ads are excluded at build time and checked at serve time.
- Deletion: removing a user removes them from all segments at the next build, with streaming removal for opt-outs.
- Freshness labels: each segment has a "last built" time, so advertisers know how fresh it is.
Trade-offs & AlternativesTrade-offs
| Decision | Choice | Why | Alternative |
|---|---|---|---|
| Serving lookup | User → segment list in a KV store | One fast read per request | Check 50K segment bitmaps per request: slow |
| Analytics | Roaring bitmaps per segment | Instant sizes and overlaps | SQL counts: slow on 200M users |
| Refresh | Daily batch + streaming for fast segments | Cheap and fresh where needed | Everything streaming: costly and complex |
| Advertiser data | Hashed matching + minimum sizes | Privacy-safe | Raw PII sharing: unacceptable |
Wrap-UpWrap-up
Keep two views of membership: a per-user segment list in a fast key-value store for ad serving, and compressed bitmaps per segment for sizing and boolean combinations. Build most segments in daily batch jobs (swapped in atomically), update fast segments with streaming and TTLs, and match advertiser lists privately with minimum audience sizes. At serve time, fetch the user's segments once and use a segment → campaign index (with exclusions) to find eligible campaigns, respecting consent throughout.