Problem RestatementProblem
TikTok asked this. Advertisers submit creative plans: a set of ad videos and a set of keywords (or audiences, product categories). The platform needs to know which keywords relate to which videos (e.g., relevance scores), so ads can be matched to searches and contexts. Computing a relationship requires calling an external model API (an ML service owned by another team or vendor) that is slow (seconds per call), rate-limited (N requests/sec), and sometimes fails. Design the processing system that turns plans into stored relationship results.
RequirementsRequirements
- Accept a plan:
{ plan_id, advertiser_id, videos: [...], keywords: [...] }. - For each needed (video, keyword) pair, or each video with a batch of keywords, call the model API and store the result
{ video_id, keyword, score, model_version }. - Show plan progress and status. Results should be usable by ad serving as soon as they're ready.
- Stay under the API's rate limit, retry failures, never lose work, and avoid duplicate paid calls.
1.1 Scale Estimates
- 50K plans/day × 20 videos × 50 keywords = 50M pairs/day. If the API accepts a video + up to 50 keywords per call, that's 1M calls/day ≈ 12 calls/sec, under a quota of, say, 30/sec. Spikes (a big advertiser uploads 10K videos) must be smoothed.
ArchitectureArchitecture
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
ADV["Advertiser upload"] --> API["Plan API"]
API --> DB[("Plans + tasks DB")]
API --> PL["Planner - split into tasks"]
PL --> Q[("Task queue - priority by advertiser tier")]
Q --> W["Workers - concurrency limited"]
W --> RL["Shared rate limiter - API quota"]
RL --> EXT["External model API"]
W --> RES[("Relationship results store")]
W --> DB
W -->|"permanent failures"| DLQ[("Dead-letter queue")]
RES --> SERVE["Ad serving / indexing"]Deep Dive — Not paying twice to score the same pairDeep dive
Relevance comes from a slow, metered vendor API — seconds per call, a hard rate limit, and a bill per request. The design is mostly about calling it as few times as possible.
One call per video-keyword pair
For each pair in the plan, call the API.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
P["Plan: 50 videos x 2,000 keywords"] --> X["100,000 pairs"]
X --> CALLS["100,000 API calls at seconds each"]
CALLS --> RATE["Vendor rate limit hit in the first minute"]
RATE --> R429["429s, retried, hit again"]
P2["Advertiser edits one keyword and resubmits"] --> AGAIN["All 100,000 scored again"]The call count is the product of two lists, and the API is the slowest, most expensive thing in the system. Resubmitting a plan with one change repeats all of it.
Batch keywords per call, and cap concurrency
One task is a video plus as many keywords as a single call accepts. Each worker runs several calls in parallel, capped so timeouts do not pile up.
Call count drops by the batch factor, which is a large win for a change of one constant. Two problems are untouched: workers scale independently, so the fleet can still exceed the vendor's limit even though each worker is polite; and a resubmitted plan is still fully rescored.
Deduplicate on content, meter globally, and retry by error class
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
T["Task: video + keyword batch"] --> DEDUP{"(video_hash, keyword, model_version) already scored?"}
DEDUP -->|"yes"| SKIP["Reuse the stored score - no call, no cost"]
DEDUP -->|"no"| TOK{"Take a token - shared bucket in Redis"}
TOK --> CALL["Vendor API"]
CALL -->|"timeout / 5xx"| BACK["Backoff with jitter, retry"]
CALL -->|"429"| GLOBAL["Back off globally - every worker slows"]
CALL -->|"4xx bad input"| DLQ[("Dead-letter - retrying cannot help")]
CALL -->|"failing heavily"| CB["Circuit breaker - pause, tasks wait"]
CALL --> STORE[("Results keyed by video_hash, keyword, model_version")]- Key the cache on the video's content hash, not its id. Advertisers re-upload the same creative under new ids constantly, and the same file scores the same way. The
model_versionin the key is what makes a model upgrade a controlled rescore rather than a silently mixed dataset. - A shared token bucket in Redis, taken before every call, enforces the vendor limit across the whole fleet. Per-worker limits cannot do this, because the number of workers changes.
- Retry by error class. A timeout or 5xx is worth retrying with backoff; a 429 should slow everyone, not just the caller that hit it; a 4xx is a bad input and retrying it burns quota forever, so it goes to the dead-letter queue.
Add a circuit breaker so a broadly failing vendor pauses calls instead of being retried into the ground — tasks wait on the queue, which is what the queue is for.
FlowFlows
- The plan is saved as
processing. The planner creates tasks and enqueues them. - Workers lease tasks (visibility timeout), get a rate-limit token, call the API, and write results.
- A task done → the plan's completed counter goes up. When all tasks are done or failed, the plan becomes
ready(orpartially_readywith a list of failures). - Ad serving can use results as they arrive (per video), and doesn't wait for the whole plan.
Fairness, Priority and Model Updates
- Priority queues: new plans from live campaigns first, and backfills and re-processing later.
- Fair share per advertiser: a single advertiser uploading 10K videos can't block everyone. Round-robin across advertisers within a priority.
- Model version changes: when the vendor ships a new model, re-process in the background at low priority. Keep both versions until the new one is complete, then switch serving to it.
Trade-offs & AlternativesTrade-offs
| Decision | Choice | Why | Alternative |
|---|---|---|---|
| Calls | Batched per video | Fewer calls, within quota | One call per pair: 50x more calls |
| Rate control | Central token bucket | Respects the vendor limit globally | Per-worker limits: overshoot when scaling |
| Failures | Backoff retries + breaker + DLQ | Robust, nothing lost | Fail the plan on first error |
| Reuse | Cache by content hash + model version | Saves cost | Recompute always: expensive |
Wrap-UpWrap-up
Split each plan into video-level tasks that batch keywords, queue them by priority with fair sharing across advertisers, and let workers call the slow external model API through a shared token-bucket rate limiter with bounded concurrency, backoff retries, a circuit breaker and a dead-letter queue. Skip already-computed pairs using a content-hash + model-version cache, write results idempotently so serving can use them immediately, and re-process in the background when the model version changes.