Problem RestatementProblem
Design a software load balancer (LB), the component that sits in front of a group of backend servers and spreads incoming traffic across them. It must send traffic only to healthy backends, handle many connections, and never be the single thing that takes the whole site down. Oracle asked to pick layer 4 or layer 7 and justify it. Amazon asked about high availability for HTTP/HTTPS traffic.
RequirementsRequirements
1.1 Functional
- Distribute requests or connections across backend servers.
- Health checks: stop sending traffic to unhealthy backends.
- Add or remove backends without dropping traffic (draining).
- (L7) TLS termination, routing by path or host, sticky sessions.
- A configuration API.
1.2 Non-Functional
- High availability: an LB node failure must not cause an outage.
- Low added latency: well under a millisecond for L4, and a few ms for L7.
- Scale: millions of concurrent connections and hundreds of thousands of requests per second.
L4 vs L7
| Layer 4 (transport) | Layer 7 (application) | |
|---|---|---|
| Looks at | IP addresses and ports (TCP/UDP) | HTTP: path, headers, cookies |
| Speed | Very fast, less CPU | Slower, parses every request |
| Features | Simple spreading of connections | Path/host routing, TLS termination, retries, header rewrites, sticky cookies |
| Examples | AWS NLB, Maglev, IPVS | NGINX, Envoy, HAProxy, AWS ALB |
High-Level ArchitectureArchitecture
3.1 Overview
- Clients resolve DNS to one or a few virtual IPs (VIPs).
- L4 layer: routers announce the same VIP from many machines (anycast plus ECMP, where the router splits traffic over several equal paths), so traffic spreads across L4 nodes. The L4 nodes use consistent hashing on the connection's addresses, so each connection sticks to one L7 proxy.
- L7 proxy fleet (e.g., Envoy): terminates TLS, routes and balances requests across backends.
- Control plane: stores configuration (routes, backend pools, certificates) and pushes it to all proxies. It collects health data.
3.2 Architecture Diagram
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
C["Clients"] --> DNS["DNS - VIP"]
DNS --> R["Edge routers - anycast + ECMP"]
R --> L4A["L4 node A"]
R --> L4B["L4 node B"]
L4A --> P1["L7 proxy 1"]
L4A --> P2["L7 proxy 2"]
L4B --> P1
L4B --> P2
P1 --> BE1["Backend pool - zone A"]
P2 --> BE2["Backend pool - zone B"]
CP["Control plane - config, certs, health"] --> P1
CP --> P2Balancing Algorithms
- Round robin: take turns. Simple, but ignores how busy each server is.
- Weighted round robin: bigger servers get more traffic.
- Least connections / least outstanding requests: send to the server with the fewest active requests. Good when request times vary.
- Power of two random choices: pick 2 servers at random and choose the less loaded one. Nearly as good as least-connections with far less coordination between LB nodes.
- Consistent hashing (by user or session ID): the same user goes to the same server. Useful for caches or sticky state, and only a few users move when servers change.
Key Mechanisms
5.1 Health checks
- Active: every few seconds, call
/healthzon each backend. Mark it down after 3 failures and up after 2 successes. The different thresholds avoid flapping. - Passive: watch real traffic. Too many 5xx errors or timeouts from a backend → eject it temporarily (outlier detection).
- Don't let a failing health endpoint take out the whole pool: if more than ~50% of backends fail at once, the problem is probably the health check itself, so keep serving ("fail open").
5.2 Connection draining
When removing a backend (deploy or scale-in), stop sending new requests to it, but let in-flight requests finish (e.g., up to 30 seconds) before it shuts down.
5.3 TLS termination
The L7 proxy decrypts HTTPS, which lets it route by path, then re-encrypts to backends if needed. Certificates are managed centrally and pushed by the control plane.
Deep Dive — Not being the single point of failureDeep dive
The load balancer exists so that no backend takes the site down. If the balancer itself can take the site down, it has moved the problem rather than solved it.
One load balancer node
DNS points at one address, one machine terminates connections and forwards to backends.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
DNS["DNS - one A record"] --> LB["Load balancer"]
LB --> B1["Backend 1"]
LB --> B2["Backend 2"]
LB --> B3["Backend 3"]
LB --> X["Node dies, or a bad config ships"]
X --> DOWN["Whole site down - healthy backends unreachable"]Nine healthy backends behind one dead balancer is a total outage. DNS will not save it either: records are cached by resolvers for minutes to hours, so repointing is not a failover mechanism.
An active/standby pair with a floating IP
Two nodes share a virtual IP using VRRP or keepalived. The standby watches the active one and takes the IP over within a couple of seconds if it stops responding.
This removes the hardware single point of failure and is the right answer for a small deployment. Two limits show up as you grow: the standby is idle capacity, so the pair can never serve more than one node's worth of traffic; and failover drops every in-flight connection, because the new holder of the IP has none of the old one's connection state.
Active-active, with hashing that survives node loss
Put several L4 nodes behind the same address using anycast or ECMP, so the routers spread packets across all of them. Every node is live, capacity is the sum of the fleet, and a dead node is removed by routing within seconds.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
NET["Routers - ECMP / anycast to one VIP"] --> L4A["L4 node A"]
NET --> L4B["L4 node B"]
NET --> L4C["L4 node C"]
L4A --> CH["Consistent hash of the 5-tuple"]
L4B --> CH
L4C --> CH
CH --> P1["Proxy 1"]
CH --> P2["Proxy 2"]
CH --> P3["Proxy 3"]
P1 --> BE["Backends across zones"]
P2 --> BE
P3 --> BEThe piece that makes it work is consistent hashing on the connection's 5-tuple, the trick behind Google's Maglev. When ECMP reshuffles packets to a different L4 node — which it does whenever the node set changes — that node independently computes the same proxy for the same connection. Existing connections survive a node loss instead of being reset, which is what the active/standby pair could not do.
Two things to say alongside it:
- Spread across zones. Proxies and backends in several availability zones, with cross-zone balancing, so losing a zone costs capacity rather than availability.
- Ship configuration gradually. A bad routing config is a more common cause of load balancer outages than hardware ever is. Roll to a few proxies, watch error rates, then continue — with automatic rollback. Redundancy does not protect against a change applied everywhere at once.
Trade-offs & AlternativesTrade-offs
| Decision | Choice | Why | Alternative |
|---|---|---|---|
| Layer | L4 in front of L7 | Speed plus rich routing | L7 only: simpler, CPU-heavy at the edge |
| Algorithm | Least outstanding / power of two choices | Adapts to slow servers | Round robin: uneven with variable requests |
| HA | Anycast + ECMP active-active | Scales and survives node loss | Active-passive pair: simpler, half the capacity idle |
| Stickiness | Consistent hashing when needed | Few moves on changes | Cookie stickiness: breaks when a server dies |
Common Follow-up QuestionsFollow-ups
- "How do you handle millions of connections?" L4 nodes keep a tiny amount of state per connection (or none, with consistent hashing). L7 proxies use event-driven I/O, and you add more nodes as needed.
- "DDoS?" Absorb at the edge with anycast across many sites, use SYN cookies at L4, and rate limit and filter at L7 (WAF).
- "Global load balancing?" Use GeoDNS or anycast to send users to the nearest healthy region, and fail over to another region when one is down.
Wrap-UpWrap-up
Put a fast L4 layer (anycast + ECMP + consistent hashing) in front of an L7 proxy fleet that terminates TLS and balances requests with least-outstanding or power-of-two-choices. Keep backends healthy with active and passive health checks, drain connections on removal, and run every layer active-active across zones with config pushed gradually, so no single machine or bad config can take everything down.