Problem RestatementProblem
Advertisers create campaigns with a budget (e.g., $70,000), a flight (start and end dates, e.g., 7 days), targeting, and a delivery goal (impressions or spend). Without pacing, a campaign that wins many auctions could spend everything in the first few hours (a waste: bad timing and less reach). Pacing spreads delivery smoothly over the flight, following when viewers actually watch, and adapts to traffic changes. Netflix asked this for its video ads platform.
RequirementsRequirements
- Deliver close to 100% of the budget by the end of the flight, without going over (or within a tiny tolerance).
- Spread delivery following the audience's traffic curve (more in the evening, less at 4 AM).
- Adapt quickly to changes: traffic spikes, targeting changes, budget edits.
- Work across hundreds of ad servers deciding in parallel, each in milliseconds.
The Core Idea
- Plan: compute an ideal cumulative spend curve for the flight, using a traffic forecast. For example, if 8–10 PM usually has 20% of the day's matching traffic, plan 20% of the daily budget there.
- Measure: track actual spend in near real time.
- Control: every minute, compare actual vs planned. If spending is ahead, slow down. If behind, speed up.
How do we slow down or speed up? Two common "knobs":
- Throttling probability (pacing rate): the campaign only enters a fraction
pof eligible auctions.pis adjusted every minute. - Bid shading: lower or raise the bid, which wins fewer or more auctions (used in auction-based platforms).
ArchitectureArchitecture
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
AS["Ad servers - use pacing rate p"] -->|"impressions, spend"| K[("Kafka - delivery events")]
K --> SPD["Spend aggregator - per campaign per minute"]
SPD --> ST[("Spend store")]
FC["Traffic forecaster"] --> PLAN["Plan: target curve per campaign"]
PLAN --> CTRL["Pacing controller - every minute"]
ST --> CTRL
CTRL -->|"new p per campaign"| CFG[("Pacing config")]
CFG -->|"push"| ASThe Controller (simple and explainable)
Every minute, for each campaign:
target_so_far = planned cumulative spend at this time
actual_so_far = measured spend
error = (target_so_far - actual_so_far) / remaining_budget
p_new = clamp(p_old × (1 + k × error), p_min, 1.0)- This is a proportional controller (a PID controller without the I and D parts). Adding some smoothing avoids oscillation.
- Also recompute the plan for the remaining time and budget (e.g., if we underspent yesterday, spread the leftover over the remaining days rather than dumping it in one hour).
Deep Dive — Not overspending a budget across hundreds of ad serversDeep dive
A $70,000 campaign is served by hundreds of machines, and spend data arrives seconds to a minute late. Every one of them is deciding, right now, whether to bid.
Check total spend before each bid
Each ad server reads the campaign's spend from a shared store and bids if it is under budget.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
S1["Server 1 - reads spend $69,900"] --> BID["Bids"]
S2["Server 2 - reads $69,900"] --> BID
S3["Server 300 - reads $69,900"] --> BID
BID --> OVER["All 300 serve against the last $100"]
OVER --> DEBT["Overspend - the platform absorbs the difference"]Every server sees the same stale number and reaches the same conclusion simultaneously. The overspend is proportional to the fleet size and lands exactly at the moment the budget is nearly gone — and an advertiser is only ever billed up to their budget, so the excess is the platform's loss.
One central counter, decremented per impression
Make the spend counter authoritative and decrement it atomically as part of serving.
Correct, and it puts a network round trip inside an auction with a single-digit millisecond budget. It also concentrates every impression of every campaign onto one counter per campaign — the hot-key problem, at ad-serving QPS. Accuracy bought at the price of the latency the auction cannot spend.
Lease budget slices, and keep a separate hard stop
Let the controller hand each ad server a small allowance for the next minute. Servers spend locally, with no coordination, and refill on the next tick.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
CTRL["Pacing controller - every minute"] --> SLICE["Allowance per server for the next minute"]
SLICE --> AS1["Ad server - spends locally, no network calls"]
SLICE --> AS2["Ad server"]
AS1 -->|"slice exhausted"| PAUSE["Stop this campaign locally until refill"]
AS1 --> RPT["Report actual spend"]
RPT --> CTRL
CTRL -->|"budget nearly gone"| TIGHT["Smaller slices, lower serve probability"]
SPEND["Total spend reaches budget"] --> HARD["Hard stop - pushed to every server immediately"]
HARD --> AS1
HARD --> AS2- Leases remove coordination from the hot path. A server checks a local number, so the auction pays nothing. The worst-case overspend is bounded by what is currently leased — which the controller controls directly.
- The margin tightens as the budget empties. Early in the flight, slices are generous. Near the end, slices shrink and the serve probability drops, so the fleet glides into the budget instead of racing at it.
- A hard stop, on a separate path. When total spend reaches the budget, a small high-priority signal is pushed to every server at once. It does not wait for the next minute's tick, and it is deliberately simpler than the pacing logic — this is the check that has to work when the controller is wrong.
Say the trade-off out loud: this design accepts a small, bounded overspend in exchange for removing a network call from every auction. That is the right trade at ad-serving scale, and knowing the bound is what makes it defensible.
Forecasting and Goals
- Traffic forecast: from historical impressions for the campaign's targeting (by hour of day and day of week), updated with recent trends.
- Goals: pacing by spend (most common), by impressions, or with frequency and reach goals (combined with frequency capping).
- End of flight: allow slightly faster delivery near the end if behind (within limits), so the campaign doesn't under-deliver.
Trade-offs & AlternativesTrade-offs
| Decision | Choice | Why | Alternative |
|---|---|---|---|
| Knob | Throttle probability | Simple, works for fixed-price ads | Bid shading: better for auctions, more complex |
| Control loop | Per-minute proportional controller + re-plan | Adapts, explainable | Fixed hourly caps: can't adapt |
| Overspend protection | Budget slices + hard stop | Tight control with delayed data | Rely on delayed spend only: overspend |
| Plan | Traffic-weighted curve | Matches when viewers watch | Even split per hour: wasteful at night |
Wrap-UpWrap-up
Plan a target cumulative spend curve from a traffic forecast, measure spend in near real time from delivery events, and run a per-minute controller that adjusts each campaign's throttle probability (or bid) based on how far ahead or behind plan it is, re-planning the remaining budget. Prevent overspend with per-server budget slices, tighter control near the end, and a fast hard stop pushed to all ad servers.