Problem RestatementProblem
Design a coffee ordering system, first for one coffee shop, then for a chain of thousands of stores (asked at Salesforce, time-boxed). Customers browse the menu, customize drinks ("oat milk, extra shot"), pay, and choose pickup or dine-in. Baristas see a queue of orders and move each through states. The customer gets a notification when the order is ready.
RequirementsRequirements
1.1 Functional
- Menu per store (items, sizes, options, prices, availability).
- Place and pay for an order for a specific store.
- Barista queue: see orders in order, and move them
placed → in_progress → ready → picked_up(orcancelled). - Real-time status for the customer, plus a push notification when ready.
1.2 Non-Functional
- Morning peak: many orders in a short window.
- An order must never be lost or made twice.
- Status updates are reliable and fast (seconds).
- Store devices may have flaky internet.
1.3 Scale Estimates
- 5,000 stores × 500 orders/day = 2.5M orders/day, with a peak of ~300 orders/sec around 8 AM.
API Design
GET /v1/stores/{id}/menuPOST /v1/orders(Idempotency-Key){ store_id, items: [{ item_id, size, options }], pickup: true }→{ order_id, status: "placed", eta }GET /v1/stores/{id}/queue?status=placed,in_progress(barista)POST /v1/orders/{id}/status{ status: "ready", expected_version: 3 }- WebSocket or SSE
/v1/orders/{id}/events(customer live status)
ArchitectureArchitecture
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
CU["Customer app"] --> API["Order API"]
API --> PAY["Payments"]
API --> DB[("Orders DB - partitioned by store")]
API --> K[("Order events")]
BA["Barista tablet"] --> API
K --> Q["Store queue updater"]
Q --> BA
K --> NS["Notification Service"]
NS -->|"push / SMS"| CU
K --> RT["Realtime status - WebSocket"]
RT --> CU- Order API: validates against the menu, takes payment (authorize, capture when ready), and saves the order.
- Orders DB: partitioned by
store_id, since queries are almost always per store. - Order events (Kafka or a queue): every state change is published. The barista queue, notifications and live status all consume it.
- Notification Service: push first, SMS fallback. It retries and deduplicates by
(order_id, status).
Data ModelData model
stores: store_id, name, timezone, open_hours
menu_items: store_id, item_id, name, base_price_cents, options (JSON), available
orders: order_id, store_id, customer_id, status, items (JSON), total_cents,
placed_at, ready_at, picked_up_at, version, idempotency_key
order_events: order_id, from_status, to_status, actor, tsKey FlowsFlows
5.1 Placing an order
- Validate items and options against the store's menu (price computed on the server, never trusted from the client).
- Authorize payment. Save the order
placedwith an ETA based on the current queue length. - Publish
order_placed. The barista tablet shows it at the end of the queue.
5.2 Barista updates
- The barista taps "start" →
in_progress, then "ready". - Each update uses optimistic concurrency (
expected_version), so two baristas tapping at once don't create a confusing state. The state machine rejects invalid moves (e.g.,picked_up → in_progress). readytriggers a push notification ("Your oat latte is ready at the counter") and a live update in the app.
Deep Dive — The shop keeps selling when the internet does notDeep dive
One coffee shop, a morning rush, and a flaky connection. A chain of thousands of stores makes this a daily certainty somewhere.
Every action is a call to the central service
The barista tablet reads the order queue from the cloud and writes each status change back.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
T["Barista tablet"] --> NET["Internet"]
NET --> API["Central order service"]
NET -->|"link drops"| DEAD["Queue blank, statuses will not save"]
DEAD --> STOP["Store cannot serve - customers already paid"]
T --> RACE["Two baristas tap the same order - both claim it"]A store that stops working when its connection does is unacceptable for a business that runs on a twenty-minute morning rush. The race is the smaller problem, but it is also unhandled: two baristas claiming the same drink both succeed, and it gets made twice.
Cache the queue locally and retry writes
The tablet keeps a local copy of the order queue and queues status updates for retry when the link returns.
The screen stays populated and updates are not lost, which is most of the practical benefit. What is still undefined is who decides. Two tablets in the same store hold two local copies, and when connectivity returns they replay conflicting claims — the central service accepts whichever arrives first, which may not be the barista who actually made the drink.
The store owns its own queue
Make the store the authority for its own orders. A store-local service (on a tablet or a small in-store box) holds the order queue and is the single writer for it; the cloud is where orders arrive and where completed orders are reported.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
CUST["Customer app"] --> CLOUD["Cloud order service"]
CLOUD -->|"order for store 118"| LOCAL["Store 118 - local queue, single writer"]
T1["Barista tablet A"] --> LOCAL
T2["Barista tablet B"] --> LOCAL
LOCAL --> CLAIM["Claims serialised in the store - no double-made drinks"]
LOCAL -->|"link down"| KEEP["Keeps serving - walk-ins, in-flight orders"]
LOCAL -->|"link returns"| SYNC["Syncs completions and timings upward"]The store keeps working through an outage because the thing it needs — its own queue — is in the building. Claims are serialised by a single local writer, so the double-made drink cannot happen regardless of how many tablets are on the counter.
Two boundaries to be explicit about:
- What the store cannot decide alone. Payment and loyalty balances stay central. An offline store accepts cash and queues card captures; it does not authorise them locally.
- New orders during an outage. Remote orders simply cannot reach the store, and the app should say so — showing "ordered" for a drink nobody will make is worse than showing the store as temporarily unavailable.
Scaling from one shop to a chain then needs nothing new in this path: each store is an independent queue, and the cloud aggregates across them.
From One Shop to a Chain
- One shop: a single service and DB is enough. The tablet polls every few seconds.
- Chain: partition data and queues by store, make services stateless and horizontally scaled, use a CDN-cached menu per store, and pre-scale for the morning peak.
- Store devices offline: the tablet keeps a local copy of the queue and queues status changes, syncing when back online. Online orders for that store may be paused ("store is temporarily not accepting mobile orders").
- Throttling: if a store's queue is too long, show longer ETAs or temporarily limit mobile orders for that store.
Trade-offs & AlternativesTrade-offs
| Decision | Choice | Why | Alternative |
|---|---|---|---|
| Status fan-out | Events → queue view, notifications, live status | Loose coupling, reliable retries | Direct calls from API: fragile |
| Concurrency | Version checks + state machine | Clean states | Last write wins: confusing states |
| Notifications | Push with SMS fallback, deduped | Reliable, no spam | Only in-app: users miss it |
| Partitioning | By store | Matches access pattern | Global table: hot and harder to scale |
Wrap-UpWrap-up
Validate and price orders on the server, take payment, and store orders partitioned by store with a strict state machine and version checks. Publish every state change as an event that drives the barista queue, live customer status (WebSocket/SSE) and a deduplicated push notification when the order is ready. Scale from one shop to a chain by partitioning per store, caching menus, pre-scaling for mornings, and letting store tablets work offline.