Problem RestatementProblem
Two interview versions:
- Atlassian: "We scaled our service up, and now it's slower than before. How do you find out why?"
- Meta: "A web server running on a single machine is down or not responding. How do you troubleshoot it, and how do you prevent it next time?"
This isn't about drawing a new system. It tests whether you can debug methodically under pressure: stop the bleeding first, use data instead of guesses, and fix the root cause.
The Process (say this structure out loud)
- Understand the impact: who is affected, since when, how bad (errors? latency? everything or one endpoint?).
- Mitigate first: if something changed recently (deploy, config, traffic), roll back or scale up. Restore service before full diagnosis.
- Look at the data: dashboards, logs, traces. Find where time is spent.
- Form hypotheses and test them one at a time.
- Fix the root cause, verify with metrics, and write a blameless postmortem with action items.
Two Simple Checklists
RED method (for services): Rate (requests/sec), Errors (error rate), Duration (latency percentiles: p50, p99). Check it for each service and dependency to find which one got slower. USE method (for resources such as CPU, memory, disk, network, thread pools and DB connections): Utilization (how busy), Saturation (how much work is waiting in queues), Errors.%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
A["Alert / user report"] --> B["Assess impact - scope, since when"]
B --> C["Mitigate - rollback, scale, shed load"]
C --> D["Find the slow hop - traces, RED per service"]
D --> E["Check resources - USE: CPU, memory, disk, pools"]
E --> F["Hypothesis and test"]
F --> G["Root cause fix + postmortem"]Deep Dive — "We scaled up and it got slower"Scale
Adding servers made latency worse. That is not a capacity problem — it is a signal that something shared is now under more pressure, and the approach you take says as much as the answer.
Add more capacity
Latency is up, so scale out further, or scale the instances up.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
SLOW["Latency up after scaling"] --> MORE["Add more app servers"]
MORE --> CONN["Each opens its own pool of DB connections"]
CONN --> DB[("Database - connection count doubles")]
DB --> WORSE["Context switching and lock waits rise - latency worse again"]
WORSE --> MOREThis is the loop worth naming out loud, because it is genuinely common: if the bottleneck is shared, every new instance adds load to it. Scaling is not neutral here — it is the thing making it worse.
Look at the dashboards
Open CPU, memory and request-rate graphs for the app tier and see what stands out.
Reasonable, and usually inconclusive. The app tier looks fine — CPU is low, memory is flat — because the servers are waiting, not working. Utilisation graphs are blind to queueing, which is exactly what a saturated shared resource produces, so the dashboards say healthy while users say slow.
Find the saturated resource by measuring waiting
Ask, for every resource in the path, three things: how utilised is it, how much saturation (queueing) does it show, and what errors is it reporting. Saturation is the one that finds this class of bug, and it is the one nobody graphs by default.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
T["A slow request's trace"] --> W1["Time in the connection pool queue"]
T --> W2["Time waiting on a lock"]
T --> W3["Time in the downstream call"]
T --> W4["Time actually computing"]
W1 --> FOUND["Whichever grew when the fleet grew is the shared bottleneck"]
W2 --> FOUND
W3 --> FOUNDWork down the shared suspects in order of likelihood:
- Database connections. More app servers means more pools. Hundreds of new connections can cost more in context switching and lock contention than the queries themselves. A connection pooler in front (PgBouncer and friends) usually fixes this outright.
- Pool sizing. Too small and requests queue locally; too large in aggregate and the database drowns. Pool wait time is the metric, not pool size.
- Cold caches. New instances start with empty local caches, so cache hit rate falls exactly when the fleet grows, and the database takes the difference.
- Lock contention. A shared row, a distributed lock, a synchronized block — contention rises with the number of contenders, so this gets worse in direct proportion to the scaling.
- A throttled dependency. More callers hit a rate limit, retries pile on, and a retry storm converts throttling into latency.
- Uneven balancing. Sticky sessions or a poor hash leave some servers hot and new ones idle, so the average looks fine and the p99 does not.
Then verify by removing capacity. If latency improves when you scale back down, the shared bottleneck is confirmed — and that is a cheap experiment that turns a hypothesis into a finding.
Case B — "A single-node web server is down"
Work from the outside in:
- Is it reachable? Ping or DNS, security groups and firewall, load balancer health checks.
- Is the process running?
systemctl status,ps. Crashed? Check logs (journalctl, app logs) for panics or OOM kills (dmesg | grep -i oom). - Resources:
top/htop(CPU),free -m(memory, swap),df -h(disk full is very common, often from logs),iostat(disk), open files (ulimit, too many connections), network (ss -s). - Is it hung? Too many threads blocked (take a thread dump), a deadlock, or a stuck dependency with no timeout.
- What changed? A recent deploy, config change, certificate expiry, or OS update.
Lasting Fixes (after the incident)
- Observability: RED dashboards per service, USE for resources, tracing and alerts on SLOs.
- Resilience: timeouts, retries with backoff and jitter (and retry budgets), circuit breakers, bulkheads.
- Capacity: load tests before scaling events, and connection pool math (instances × pool size ≤ DB limit).
- Safe changes: canary deploys with automatic rollback.
- Postmortem: timeline, root cause, what went well and badly, and owned action items. Blameless, focused on systems, not people.
Common Follow-up QuestionsFollow-ups
- "Latency is high but CPU is low everywhere?" Then requests are waiting: on locks, I/O, pool slots or a slow dependency. Look at saturation (queues) and traces, not CPU.
- "Only p99 got worse?" Look at outliers: GC pauses, a slow shard, a hot key, retries or one bad host. Check latency per host.
- "Errors went up after a deploy but it's not obvious why?" Roll back first, then compare logs and traces between the old and new versions.
Wrap-UpWrap-up
Start with impact and mitigation (roll back, scale, shed load), then use data. RED per service finds the slow hop and USE per resource finds the saturated component. After scaling, the usual culprits are shared ones: database connections and queries, pools, cold or hot caches, locks and rate-limited dependencies. On a single node, check reachability, the process, then CPU, memory, disk and dependencies. Finish with lasting fixes and a blameless postmortem.