Problem RestatementProblem
Amazon asked: you run a relational database with one primary (takes all writes) and replicas (copies that follow the primary). How do you make it highly available? Explain replication modes, how to detect that the primary failed, how to promote a replica safely (without two primaries, called "split brain"), how clients find the new primary, and how the old primary rejoins.
Deep Dive — Choosing a replication modeDeep dive
Everything about failover follows from one decision: how long the primary waits for replicas before telling the client a write succeeded.
Asynchronous replication
The primary commits locally and replies immediately; replicas catch up a moment later.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
W["Client write"] --> P["Primary commits"]
P --> ACK["Acknowledged - fastest possible"]
P -.->|"streaming, slightly behind"| R["Replica"]
CRASH["Primary dies"] --> GAP["Transactions committed but not yet replicated"]
GAP --> LOST["Promote the replica - those writes are gone"]
LOST --> WORSE["The client was told they succeeded"]The latency is unbeatable and the failure mode is unacceptable for anything transactional: the database confirmed writes that no longer exist. Payments, orders and ledgers cannot use this, because the client has already acted on the acknowledgement.
Synchronous replication
The primary waits until a replica has the change before replying. Failover loses nothing.
Durability is now correct, and availability has quietly become worse. Every write pays a network round trip, and — the part people miss — if the synchronous replica is down, writes block entirely. A design intended to survive a failure has made a second machine's health a prerequisite for accepting writes.
Wait for one of several replicas
Semi-synchronous, or quorum, replication: the primary waits for any one of N replicas to confirm.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
W["Client write"] --> P["Primary"]
P --> R1["Replica A"]
P --> R2["Replica B"]
P --> R3["Replica C"]
R1 -->|"first to confirm"| ACK["Acknowledge the client"]
R2 --> ACK
R3 --> ACK
DOWN["One replica down or slow"] --> STILL["Writes continue - another confirms"]
FAIL["Primary dies"] --> PROMO["Promote the most advanced replica - no acknowledged write lost"]- No acknowledged write is lost, because every acknowledgement is backed by a durable copy on at least one other machine.
- One replica failing does not stop writes, because the commit waits for whoever answers first rather than for a specific node.
- Latency is the fastest replica's, not the slowest — so a single slow follower no longer taxes every write.
Two details that make it real: promote the most advanced replica, which means tracking replication positions rather than picking arbitrarily; and require a majority to elect a new primary, so a network partition cannot produce two primaries both accepting writes. Getting the mode right and the election wrong still loses data — just more spectacularly.
ArchitectureArchitecture
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
APP["Applications"] --> PX["Proxy / service discovery (VIP, DNS, HAProxy)"]
PX --> P[("Primary - zone A")]
P -->|"sync replication"| R1[("Replica - zone B")]
P -->|"async replication"| R2[("Replica - zone C")]
ORCH["Failover manager (Patroni + etcd)"] --> P
ORCH --> R1
ORCH --> R2
ORCH -->|"update routing"| PXFailover Step by Step
- Detect: the failover manager checks health from multiple observers (to avoid a false alarm from one network glitch). The primary must hold a lease in a consensus store (etcd/ZooKeeper). If it can't renew the lease within, e.g., 10 seconds, it's considered failed.
- Fence the old primary: make sure it can't accept writes anymore. It demotes itself when it loses its lease, and the manager can also cut its network or power (STONITH, "shoot the other node in the head"). This prevents split brain.
- Pick the best replica: the most up to date (highest replication position), ideally the synchronous one (no data loss).
- Promote it to primary. Other replicas re-point to it.
- Redirect clients: update the proxy, the virtual IP or DNS (a low TTL), or clients query service discovery. Connections to the old primary fail and reconnect.
- Old primary rejoins as a replica after it's repaired, rewinding any writes that never reached the others (e.g.,
pg_rewind).
Reads and Lag
- Replicas can serve reads to scale, but they may be slightly behind (replication lag). For "read your own writes", route a user's reads to the primary for a short time after they write, or wait until the replica has caught up to the write's position.
- Monitor lag, and alert when it's high (it also means more data at risk with async replication).
Testing and Operations
- Practice failovers regularly (game days), and measure the real RTO.
- Backups plus point-in-time recovery (base backups + WAL archive), because replication copies mistakes too (a bad
DELETEreplicates instantly). - Cross-region replica for disaster recovery (asynchronous, with a known RPO).
Wrap-UpWrap-up
Replicate from the primary to replicas in other zones, using synchronous or quorum replication for zero data loss (or asynchronous when a small RPO is acceptable). Let a consensus-backed failover manager detect failure via leases and multiple observers, fence the old primary to avoid split brain, promote the most up-to-date replica and redirect clients through a proxy or discovery. Handle replica lag for reads, keep PITR backups, and test failover regularly against RPO and RTO targets.