Problem RestatementProblem
Instead of designing from scratch, you are given an existing design and asked to review it. For example:
Client → DNS → Load Balancer → Application Service → Database, where the application also calls an external third-party API.
Or you get a written design document that has gaps and unsafe assumptions. Your job is to find the most important risks (scalability, reliability, security, operability), rank them, and propose a better design, plus a plan to prove the improvements work.
This tests judgment. Interviewers want a structured review, not a random list of buzzwords.
How to Structure the Review
Use a simple checklist and go layer by layer:
- Clarify first: what does the system do, how many users, what's the traffic pattern, what is "down" for the business, and what are the latency and availability targets?
- Follow one request end to end, and at each hop ask: what if this is slow? What if it fails? What if traffic is 10x?
- Group findings into availability, scalability, data safety, security, and operability (monitoring, deploys, on-call).
- Rank by impact × likelihood. Fix the things that can take the whole system down or lose data first.
- Propose fixes, then say how you would verify each one (load tests, failure drills, metrics).
Example Review of the Simple Diagram
2.1 Before
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
C["Client"] --> DNS["DNS"]
DNS --> LB["Single Load Balancer"]
LB --> APP["App Service"]
APP --> DB[("Single Database")]
APP --> EXT["Third-party API"]2.2 Findings (ranked)
| # | Risk | Why it matters | Fix |
|---|---|---|---|
| 1 | Single database, no replica or backup mentioned | DB failure = full outage, possible data loss | Primary + standby replica with automatic failover, point-in-time backups, tested restores |
| 2 | Third-party API called synchronously with no timeout | If it gets slow, all app threads hang and the whole site goes down | Short timeouts, retries with backoff and jitter, a circuit breaker, a fallback or async queue |
| 3 | One load balancer, one zone | LB or zone failure = outage | Managed LB across 2–3 availability zones; app instances spread across zones |
| 4 | App tier size unknown, no autoscaling | Traffic spikes overload it | At least 2–3 instances, autoscaling on CPU and latency, stateless app |
| 5 | No caching | DB takes every read and becomes the bottleneck | Cache hot reads (Redis) and static content (CDN) |
| 6 | No monitoring or alerting shown | Problems found by users first | Metrics (latency, errors, saturation), logs, tracing, alerts, dashboards |
| 7 | Security not described | Data leaks, abuse | TLS everywhere, WAF, rate limiting, secrets in a vault, least-privilege DB access |
| 8 | Deploys not described | A bad deploy takes everything down | Rolling or canary deploys with automatic rollback |
The Improved Design
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
C["Client"] --> CDN["CDN + WAF"]
CDN --> LB["Load Balancer - multi-zone"]
LB --> A1["App - zone A"]
LB --> A2["App - zone B"]
A1 --> R[("Redis cache")]
A2 --> R
A1 --> P[("Primary DB")]
A2 --> P
P -->|"replication"| S[("Standby replica - other zone")]
A1 --> CB["Circuit breaker + timeouts"]
A2 --> CB
CB --> EXT["Third-party API"]
A1 --> Q[("Queue for non-urgent third-party calls")]Deep Dive A — The third-party call in the request pathDeep dive
Application Service → external API is one arrow on the diagram and usually the weakest part of the design. Reviewing it well is often the strongest finding you can make.
Call it and wait
The handler calls the provider and uses the response. No timeout, because the HTTP client's default is generous or absent.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
U["User request"] --> APP["App service - thread taken"]
APP --> EXT["Third-party API - degraded, 40 s responses"]
EXT --> HOLD["Threads pile up waiting"]
HOLD --> POOL["Thread pool exhausted"]
POOL --> DOWN["Every endpoint fails, including ones that never call the API"]This is the failure worth naming out loud: the provider does not go down, it goes slow, and slow is worse. Threads accumulate on the blocked call until the pool is gone, and endpoints with no relationship to that provider start failing. One dependency's latency became our total outage.
Timeout and retry with backoff
Set a timeout below your own deadline, and retry twice with exponential backoff and jitter — only for idempotent calls.
Threads are released now, which is the important half. But retries multiply load on a provider that is already struggling, and jitter only spreads the stampede rather than removing it. And a provider that has been failing for ten minutes still gets the full timeout burned on every request before we give up.
Fail fast, isolate, and get off the request path
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
REQ["Request"] --> BH["Bulkhead - dedicated small pool for third-party calls"]
BH --> CB{"Circuit breaker"}
CB -->|"open"| FB["Fail fast - cached value or degraded response"]
CB -->|"closed"| CALL["Call with timeout, capped retries"]
CALL -->|"repeated failures"| OPEN["Open the circuit for a cool-down"]
OPEN --> FB
ASYNC["Answer not needed now - email, webhook"] --> Q[("Queue - retried out of band")]Four things, in order of how much they buy:
- Bulkhead. Give third-party calls their own small connection and thread pool. Whatever happens to the provider, it can consume that pool and nothing else — the failure stops being total.
- Circuit breaker. After repeated failures, stop calling for a cool-down and fail immediately. This protects our latency and gives the provider room to recover instead of being retried into the ground.
- Move it off the request path. If the user does not need the answer now — sending an email, notifying a webhook — put it on a queue. A dependency that is not in the request path cannot take the request down.
- Cache what changes slowly, so a degraded provider is invisible for most calls.
Then ask the question that bounds all of it: what is the provider's SLA? Our availability for any feature cannot exceed the availability of what it synchronously depends on. If they publish 99.5% and we promised 99.9%, the design is wrong on paper before it has failed once.
Deep Dive B — Reviewing a design documentDeep dive
When given a written doc, look for:
- Unstated assumptions: "the queue never loses messages", "clocks are in sync", "this runs once a day, so no concurrency". Ask what happens when each one is wrong.
- Missing numbers: no QPS, data size or growth estimate, so you can't judge the design.
- Data safety: backups, migrations, what happens on partial failure of multi-step writes (a need for idempotency or an outbox).
- Operability: how will on-call know it's broken? How do we roll back?
- Security and privacy: who can access what, and is PII encrypted and logged safely?
How to Verify the Improvements
- Load test to 2–3x expected peak and watch latency and errors.
- Failure drills (chaos testing): kill an app instance, fail over the DB, and make the third-party API slow. Check the system behaves as designed.
- Backup restore test on a schedule, since a backup you never restored is not a backup.
- SLOs and alerts: define "good" (e.g., 99.9% of requests under 300 ms) and alert when you're burning through the error budget.
Trade-offs to MentionTrade-offs
- More replicas and zones cost more money. Match them to the business's actual availability needs.
- Caching adds staleness and invalidation work.
- Circuit breakers and fallbacks mean some users get a degraded experience instead of an error. Agree with product on what "degraded" looks like.
Wrap-UpWrap-up
Review in a fixed order: clarify goals, trace a request, list risks by category, and rank them by impact. The usual top risks are a single database, a synchronous third-party call without timeouts, a single zone, and no monitoring. Fix them with replicas and backups, timeouts, retries, circuit breakers and queues, multi-zone deployment and observability, then prove the fixes with load tests and failure drills.