Problem RestatementProblem
Netflix asked about the supply side of an ad platform: publishers own many websites, apps, channels and ad placements, and each needs custom configuration rules, e.g.:
- blocked ad categories (no alcohol ads on a kids' channel),
- floor prices (minimum price per impression),
- max ads per break, frequency limits, allowed ad formats.
Rules can be set at the publisher level and overridden lower down (site, then placement). Ad servers all over the world must apply the right rules on every ad request, in microseconds. A related question asked about config rollout: changes must reach servers globally within minutes, safely, because a bad config can break ad serving everywhere.
RequirementsRequirements
- CRUD for rules at publisher, site and placement levels, with inheritance and overrides.
- Compute the effective config for a placement.
- Distribute changes to all ad servers within a few minutes.
- Validation, versioning, audit log, staged rollout and instant rollback.
Data ModelData model
entities: entity_id, type (publisher|site|placement), parent_id
rules: entity_id, key (e.g. "blocked_categories", "floor_price_cents"), value (JSON),
mode (set | append | remove), version, updated_by, updated_atset overrides, while append/remove modify lists (e.g., the site adds one more blocked category).
ArchitectureArchitecture
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
UI["Publisher / admin UI"] --> CS["Config Service - validate, version, audit"]
CS --> DB[("Rules DB")]
CS --> BLD["Snapshot builder - effective configs"]
BLD --> OS[("Versioned snapshots - object storage")]
CS --> RO["Rollout controller - stages, health"]
RO -->|"notify version N"| AS1["Ad servers - canary"]
RO -->|"then"| AS2["Ad servers - region 1"]
RO -->|"then"| AS3["Ad servers - all regions"]
AS1 -->|"fetch snapshot"| OS
AS2 --> OS
AS3 --> OS
AS1 -->|"metrics"| ROKey FlowsFlows
4.1 Changing a rule
- Validate: schema (types, ranges, e.g., a floor price can't be negative), references (the category exists), and a dry-run diff showing which placements' effective config will change.
- Save it as a new version with an audit record (who, what, why).
- The snapshot builder computes effective configs for affected placements, and writes a new versioned snapshot (the full config, or a delta) to object storage.
4.2 Rolling it out
- The rollout controller tells canary servers (e.g., 1%) to load version N.
- It watches metrics: ad fill rate, errors, revenue per request, latency. If they're healthy after a few minutes, it moves to one region, then all.
- If metrics degrade → automatic rollback: tell servers to go back to version N-1 (still cached locally).
4.3 Ad server side
- Keeps the effective config in memory, keyed by placement ID. The lookup takes microseconds.
- Loads new snapshots in the background and swaps atomically, so requests never see half-applied config.
- If it can't fetch updates, it keeps serving with the last good version (fail static), rather than failing.
Deep Dive — Getting config to the ad serversDeep dive
Publishers change floor prices and blocked categories all day. The ad server needs the current rules for every bid request, at ad-serving latency, on hundreds of machines.
Read the config database per request
The ad server looks up the placement's rules when a request arrives.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
REQ["Bid request - under 100 ms budget"] --> AS["Ad server"]
AS --> DB[("Config DB")]
DB --> LAT["A round trip inside the decision path"]
DB --> LOAD["Millions of QPS against a database holding thousands of rows"]
DB --> SPOF["Config DB slow, ad serving stops"]The data is tiny and barely changes, and this design queries it millions of times a second. It also puts a database in the critical path of every ad decision, so its availability becomes the ad server's availability.
Cache it and poll for changes
Each ad server keeps the config in memory and re-fetches every 30 seconds.
The database load collapses and the latency disappears. Two problems: hundreds of servers polling on their own timers produce a steady drumbeat of queries for data that usually has not changed, and the propagation delay is unbounded in practice — a publisher blocks alcohol ads and watches them keep serving for half a minute, which for a kids' channel is a compliance incident rather than a delay.
Push versioned snapshots, evaluate locally
Build the whole config into an immutable, versioned snapshot when anything changes, and push it to every ad server.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
ED["Publisher edits a rule"] --> BUILD["Build snapshot v1482 - full config, immutable"]
BUILD --> OBJ[("Object storage / distribution tier")]
BUILD --> NOTIFY["Notify servers - new version available"]
NOTIFY --> AS1["Ad server - load into memory, swap atomically"]
NOTIFY --> AS2["Ad server"]
AS1 --> DEC["Decide locally - zero network calls"]
DEC --> LOG["Log decision with config_version 1482"]- Push, do not poll. Propagation becomes a second or two, and there is no steady query load for unchanged data.
- Snapshots, not deltas. A server loads one complete, self-consistent version and swaps it in atomically. A server that misses a delta is silently wrong; a server on an old snapshot is merely stale, and it says which version it is on.
- Stamp the version on every decision. When a publisher asks why an alcohol ad served at 14:03, the log says
config_version 1481and the question becomes answerable. This is the single most valuable line in the design for an ad platform, because these questions are contractual. - Keep a local copy on disk. A server restarting while the distribution tier is unavailable boots on its last snapshot rather than serving with no rules at all — which for blocked categories must never happen.
Roll new snapshots to a few servers first and watch for anomalies before the fleet takes them. A bad config reaching every server at once is the failure this architecture makes fast, and fast is not always good.
Trade-offs & AlternativesTrade-offs
| Decision | Choice | Why | Alternative |
|---|---|---|---|
| Config model | Hierarchy with overrides | Less duplication, easy defaults | Flat per placement: huge and repetitive |
| Serving | Precomputed effective config in memory | Microsecond reads | Resolve inheritance per request: slower |
| Distribution | Versioned snapshots + notify + pull | Fast, consistent, cacheable | DB queries from every server: overload |
| Safety | Validate + canary + auto rollback | Bad config caught early | Instant global push: global outages |
Wrap-UpWrap-up
Store rules at publisher, site and placement levels with clear override semantics, validate every change (including a dry-run of affected placements), and version and audit it. Build versioned snapshots of effective configs, and roll them out in stages (canary → region → global) guarded by health metrics with automatic rollback. Ad servers keep the effective config in memory, swap versions atomically, and keep serving the last good version if updates fail.