Problem RestatementProblem
A client may send the same request twice: the user double-clicks "Pay", the app retries after a timeout (the first request actually succeeded), or the network duplicates a packet. For operations like charging a card or creating an order, doing it twice is a serious bug. Design a reliable way (asked at OpenAI) to make the operation take effect exactly once and give the client a consistent response every time it retries.
The Core Idea: Idempotency Keys
- The client generates a unique key (a UUID) per logical operation (not per HTTP attempt) and sends it:
Idempotency-Key: 5f3c.... All retries of the same operation reuse the same key. - The server remembers keys it has processed and their results. If the key was seen before, it returns the saved response instead of doing the work again.
"Idempotent" means doing it once or many times has the same effect.
Deep Dive — Making the second request a no-opDeep dive
The user double-taps Pay, or the client retries after a timeout on a request that actually succeeded. The second request must produce the same outcome, not a second charge.
Check whether it already exists, then create
Look for a matching payment; if none, create one.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
sequenceDiagram
participant C1 as Request 1
participant S as Server
participant C2 as Request 2 - retry
C1->>S: does a payment for this key exist?
S-->>C1: no
C2->>S: does a payment for this key exist?
S-->>C2: no
C1->>S: create payment
C2->>S: create payment
Note over S: two charges - the check and the create are separateCheck-then-act again. The duplicate arrives precisely when the first request is still in flight — that is what a timeout retry is — so the window the design ignores is the window it will always land in.
Record the key after completing the work
Do the work, then store the idempotency key with the result, and return the stored result if the key is seen again.
This handles a retry that arrives after the first finished. It does nothing for the concurrent case, which is the common one: both requests are doing the work before either has written its key. The write also has to be in the same transaction as the work, or a crash between them loses the record and the next retry charges again.
Claim the key first, then do the work
Insert the key as in_progress before touching anything, and let the primary key be the concurrency control:
CREATE TABLE idempotency_keys (
scope TEXT, -- user or merchant, so keys cannot collide across customers
key TEXT,
request_hash TEXT, -- fingerprint of the request body
status TEXT, -- in_progress, completed
response_code INT, response_body JSONB,
locked_until TIMESTAMP, created_at TIMESTAMP,
PRIMARY KEY (scope, key)
);%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
R["Request + Idempotency-Key"] --> INS{"Insert as in_progress"}
INS -->|"inserted"| WORK["Do the work in a transaction"]
WORK --> SAVE["Save response, status completed"]
SAVE --> OUT["Return the response"]
INS -->|"exists, completed"| REPLAY["Return the stored response"]
INS -->|"exists, in_progress"| WAIT["409 or retry-after - the first one is still running"]
INS -->|"exists, different request_hash"| ERR["422 - key reused with a different body"]- The insert is the lock. Exactly one request wins the primary key; the loser learns immediately that another is in flight, so the concurrent case is handled by the database rather than by timing.
- Store the response, not just the fact. A replay must return the same status code and body, or the client sees a success it cannot correlate.
- Hash the request body. The same key with a different body is a client bug — usually a key reused across payments — and answering it with the first payment's response would be worse than an error.
- Scope keys per customer so two tenants cannot collide, and give
in_progressrows alocked_untilso a crashed request does not block its own retries forever.
Other Details
- Expiry: keep keys for 24 hours to 7 days (clients shouldn't retry after that), then clean them up with a TTL.
- Scope: keys are unique per account, so two customers can't collide.
- Where to store: the same database as the business data is best (one transaction). Redis works for low-risk cases (
SET key NX+ a stored response), but loses the atomicity with DB writes. - Natural idempotency: some operations are idempotent by design.
PUT /users/42 {name}sets a value, and "create order with client-provided order_id" uses a unique constraint. Prefer these when possible. - Messaging: for events and queues, the same idea applies with the event ID. Consumers keep processed IDs, or make writes upserts.
Wrap-UpWrap-up
Have clients send one idempotency key per logical operation, reused across retries. On the server, atomically insert the key as in-progress, do the work and store the response in the same transaction, replay the stored response for repeats, reject concurrent or mismatched duplicates, and expire old keys. Pass the key to downstream providers so external side effects are also deduplicated. That's what "exactly once" means in practice: at-least-once delivery plus idempotent processing.