Problem RestatementProblem
Walmart asked: design a service that answers one client request by calling several downstream microservices and combining the answers. For example, a product page needs product details, price, inventory, reviews, delivery estimate and recommendations, each from a different team's service. The aggregator should be fast (low tail latency), resilient (one slow or failing service shouldn't break the page), and not overload struggling services.
RequirementsRequirements
- One API call returns the combined response.
- Latency target, e.g., p99 < 300 ms.
- If an optional part fails, still return the rest (degraded response).
- Protect downstream services from overload and retry storms.
ArchitectureArchitecture
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
C["Client"] --> AGG["Aggregator - fan-out / fan-in"]
AGG -->|"required, 120ms"| P["Product service"]
AGG -->|"required, 120ms"| PR["Price service"]
AGG -->|"required, 150ms"| INV["Inventory service"]
AGG -->|"optional, 200ms"| REV["Reviews service"]
AGG -->|"optional, 200ms"| RECO["Recommendations"]
AGG --> CA[("Cache - per dependency")]Deep Dive — One page, eight downstream servicesDeep dive
The product page needs details, price, stock, reviews, recommendations and a delivery estimate. The page has a latency budget; the services have their own bad days.
Call them one after another
Call each service in turn, passing results along.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
A["Details 40 ms"] --> B["Price 30 ms"]
B --> C["Stock 25 ms"]
C --> D["Reviews 60 ms"]
D --> E["Recommendations 200 ms"]
E --> TOT["Total = the sum: 355 ms and rising"]
E -->|"recommendations down"| FAIL["Whole page fails"]Latency is the sum of everything and availability is the product of everything. Eight dependencies at 99.9% give a page that fails about one visit in 125 — and the least important service can cause it.
Fan out in parallel
Start all independent calls at once and wait for them all. Latency drops from the sum to the maximum.
A big win, and two problems remain. Waiting for all of them means one failure still fails the page. And "the maximum" is set by the slowest service on the day, so a degraded recommendations service still dictates the page's latency.
Deadlines, propagated, with required and optional parts
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
REQ["Request - overall budget 250 ms"] --> FAN["Parallel fan-out"]
FAN --> D1["Details - required, 80 ms"]
FAN --> D2["Price - required, 80 ms"]
FAN --> D3["Reviews - optional, 100 ms"]
FAN --> D4["Recommendations - optional, 120 ms"]
FAN --> HDR["Remaining deadline passed downstream as a header"]
HDR --> STOP["Downstream stops work that can no longer be used"]
D1 --> REND["Render"]
D3 -->|"timed out"| OMIT["Section omitted or shown as 'unavailable'"]
D1 -->|"failed"| ERR["Required part failed - error the page"]- Classify every call as required or optional. The page cannot render without price; it renders perfectly well without recommendations. This single distinction converts most dependency failures from outages into a missing section.
- Give each call a timeout within the overall budget, sized to how important it is — not one global timeout applied uniformly.
- Propagate the remaining deadline downstream. A service that knows only 30 ms remain can decline to start work whose result will be discarded, which saves capacity across the estate. Without it, timed-out work keeps running everywhere.
- Chain only where there is a genuine dependency — the delivery estimate needs the product's warehouse — and keep everything else parallel.
Serve a degraded page rather than an error whenever the required parts are present. The number to watch afterwards is not average latency but the rate of omitted sections, because that is the part users notice and dashboards usually do not show.
Example Timeline
At t=0, send all 5 calls. At 40 ms, product and price are back. Inventory arrives at 90 ms. Reviews hit a timeout at 200 ms → breaker counts a failure → the response omits reviews. Recommendations arrive at 150 ms. The response is sent at ~200 ms with reviews marked unavailable.
Observability
- Distributed tracing across the fan-out shows which dependency drives latency.
- Per-dependency metrics: latency, error rate, timeout rate, breaker state and fallback usage.
- Alert when fallback usage rises, which means the page is degrading quietly.
Wrap-UpWrap-up
Fan out to downstream services in parallel within an overall deadline, giving each call its own timeout and passing the remaining deadline downstream. Separate required from optional parts, with fallbacks and cached data, and protect everything with per-dependency circuit breakers, bulkheads, careful budgeted retries and optional hedging for tail latency. Trace and monitor each dependency so degradation is visible.