Problem RestatementProblem
Design a platform (asked at Airbnb) that lets product teams run experiments. A team defines an experiment ("new checkout button"), with variants (control and treatment) and who is eligible. Users are consistently assigned to a variant (the same user always sees the same one). The app shows the assigned variant, the platform logs exposures (the user actually saw it) and outcomes (bookings, clicks), and then computes results with statistical significance.
RequirementsRequirements
- Create, start, ramp (1% → 50%), stop and roll out experiments.
- Targeting: country, platform, new vs existing users.
- Consistent, fast assignment in services and apps.
- Log exposures and outcomes, and compute metrics per variant with confidence intervals.
- Many concurrent experiments without interfering with each other.
- Guardrails: auto-alert or stop if key metrics (errors, revenue) get worse.
ArchitectureArchitecture
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
UI["Experiment UI"] --> CFG["Config Service"]
CFG --> DB[("Experiment configs")]
CFG -->|"push config"| SDK["Assignment SDK in services / apps"]
SDK -->|"exposure events"| K[("Kafka")]
APP["Product events - bookings, clicks"] --> K
K --> LAKE[("Data lake")]
LAKE --> MET["Metrics pipeline - join exposures + outcomes"]
MET --> STAT["Stats engine"]
STAT --> DASH["Results dashboard"]
STAT -->|"guardrail breach"| ALERT["Alerts / auto-stop"]Assignment (the core idea)
bucket = hash(experiment_salt + user_id) % 1000.- The config says: buckets 0–499 = control, 500–999 = treatment (for a 50/50 split at 100% traffic).
- Deterministic: the same user always gets the same bucket, and no database lookup is needed.
- Local evaluation: the SDK has the config in memory and assigns in microseconds, with no network call per request.
- Ramp-up: first expose only buckets 0–9 (1% of users). Increasing to 50% adds buckets, and users already in stay in (no switching).
- Layers: experiments in different layers use different salts, so they're independent. Experiments that might conflict (two tests changing the same button) go in the same layer and get non-overlapping buckets.
Deep Dive — Ramping an experiment from 1% to 50% without reshuffling anyoneDeep dive
An experiment starts at 1% of traffic and grows. The users already in it must stay in it and keep the same variant — otherwise the results are meaningless and users see the interface change under them.
Decide per request
Roll a random number on each request: below the exposure percentage, show treatment.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
R1["Request 1 - random 0.004"] --> T["Treatment"]
R2["Request 2, same user - random 0.7"] --> C["Control"]
T --> FLIP["The checkout button changes between page loads"]
C --> FLIP
FLIP --> BAD["Neither the user experience nor the measurement means anything"]Assignment has to be a property of the user, not of the request. Without that, a single user contributes to both arms and the comparison collapses.
Hash the user against the percentage
hash(experiment_salt + user_id) % 100 < exposure. The same user always hashes to the same number, so assignment is stable, needs no storage and no network call.
Stable per user, and it breaks on the ramp. Going from 1% to 2% keeps everyone already included — but changing the split does not behave: at 1% exposure with a 50/50 split, the implementation usually re-hashes to choose a variant, and any change to exposure, split or salt silently moves users between arms. There is also no way to express "these two experiments must not overlap".
Assign to buckets, and map buckets to variants
Hash the user once into a fixed bucket space — 1,000 buckets — and keep the bucket-to-variant mapping in the experiment's config.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
U["bucket = hash(salt + user_id) % 1000"] --> CFG["Config: bucket ranges to variants"]
CFG --> R1["Ramp 1%: buckets 0-4 control, 5-9 treatment"]
R1 --> R2["Ramp 10%: buckets 0-49 control, 50-99 treatment"]
R2 --> NOTE["Buckets 0-9 keep the variant they already had"]
CFG --> L["Layers: different salt per layer - independent experiments"]
L --> CONF["Conflicting experiments share a layer with non-overlapping buckets"]The user's bucket never changes — only the map does, and ramping extends ranges rather than recomputing membership. Everyone already exposed keeps their variant, which is precisely the property the previous rung could not guarantee.
Two things this unlocks:
- Layers. Each layer hashes with its own salt, so experiments in different layers are statistically independent and can run simultaneously on the same users. Two experiments that would interfere — both changing the checkout button — go in the same layer and are given non-overlapping bucket ranges, which makes the exclusion explicit rather than a matter of hoping.
- Local evaluation. The config is pushed to the SDK, so assignment is a hash and a range lookup in memory — microseconds, no network call, and it still works if the experiment service is down.
Log the exposure when the user actually sees the variant, not when they are assigned. Counting assignments that were never rendered dilutes the effect towards zero and is one of the most common ways experiment results are quietly wrong.
Logging and Metrics
- Log exposure only when the user actually sees the variant (not just when assigned), otherwise results are diluted.
- The event contains
{ user_id, experiment_id, variant, ts }. Deduplicate to the first exposure per user. - The metrics pipeline (daily plus intraday) joins exposures with outcome events after the exposure time, per user. Then it aggregates per variant: conversion rate, revenue per user, etc.
- Metric definitions are shared and reviewed, so every experiment computes "booking rate" the same way.
Statistics (explain simply)
- Compare treatment vs control with a t-test or z-test and report the difference with a 95% confidence interval. If the interval doesn't include 0, the result is significant.
- Power / sample size: decide beforehand how many users are needed to detect the effect you care about (e.g., +1% booking rate).
- Variance reduction (e.g., CUPED): using each user's pre-experiment behavior makes results significant faster.
- Pitfalls to mention:
- Peeking: checking every day and stopping when it looks good inflates false positives. Use fixed durations or sequential testing methods.
- Sample ratio mismatch (SRM): if a 50/50 test has 52/48 users, something is broken (e.g., a bug dropping exposures). Check automatically and block results.
- Novelty effects: run at least 1–2 weeks to cover weekly patterns.
- Multiple metrics: correct for many comparisons, or pick one primary metric up front.
Trade-offs & AlternativesTrade-offs
| Decision | Choice | Why | Alternative |
|---|---|---|---|
| Assignment | Hash-based, local SDK | Consistent, zero latency | Assignment service call: latency, dependency |
| Unit | User ID (or device before login) | Consistent experience | Per request: inconsistent, noisy |
| Isolation | Layers with salts | Many experiments at once | One at a time: too slow for many teams |
| Analysis | Batch pipeline + stats engine | Accurate, auditable | Live counters only: no proper stats |
Wrap-UpWrap-up
Assign users deterministically with hash(salt + user_id) into buckets, evaluated locally by an SDK that receives pushed configs, with layers for independent experiments and bucket-based ramp-up that never reshuffles users. Log real exposures, join them with outcomes in a metrics pipeline using shared metric definitions, and report differences with confidence intervals, while guarding against peeking, sample ratio mismatch and harm to guardrail metrics.