Problem RestatementProblem
Walmart asked: design a highly available deployment for a Spring Boot application behind a load balancer, plus resilient background processing for long-running tasks (e.g., generating reports, bulk imports that take minutes). The service must survive instance failures and deployments without errors, and background jobs must not be lost or run twice.
ArchitectureArchitecture
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
C["Clients"] --> LB["Load balancer - health checks"]
LB --> A1["Spring Boot API - instance 1"]
LB --> A2["Spring Boot API - instance 2"]
LB --> A3["Spring Boot API - instance 3"]
A1 --> DB[("Postgres primary + standby")]
A1 --> R[("Redis - sessions / cache")]
A1 -->|"enqueue job"| Q[("Queue - RabbitMQ / Kafka / SQS")]
Q --> W1["Worker instance"]
Q --> W2["Worker instance"]
W1 --> DB
SCH["@Scheduled jobs + ShedLock"] --> DBHigh Availability for the API
- Stateless instances: no user session in local memory. Use tokens (JWT) or store sessions in Redis (Spring Session), so any instance can serve any request, and losing one instance doesn't log users out.
- At least 2–3 instances across availability zones behind a load balancer.
- Health checks with Spring Boot Actuator: a liveness probe (is the process alive?) and a readiness probe (can it serve? DB reachable, warmed up). The LB only sends traffic to ready instances.
- Graceful shutdown (
server.shutdown=graceful): on deploy, stop accepting new requests, finish in-flight ones, then exit. Combined with rolling or blue-green deployments, users see no errors. - Connection pools (HikariCP): size them so instances × pool size fits the database's limit.
- Database HA: a primary with a standby replica and automatic failover, plus backups.
Long-Running Work: Don't Do It in the Request
If a request takes minutes, it ties up a thread and times out at the LB, and if the instance restarts the work is lost. Instead:
- The API validates the request, creates a job record (
status = queued) in the DB, puts a message on a queue, and returns 202 Accepted with a job ID. - Worker instances (a separate Spring Boot app or profile) consume messages, run the task, update progress, and set
succeededorfailed. - The client polls
GET /jobs/{id}(or gets a webhook or notification).
- The queue gives at-least-once delivery: acknowledge only after finishing, so a crashed worker's message is redelivered.
- Make tasks idempotent (check the job status before starting, and write results with the job ID as the key), so redelivery doesn't duplicate work.
- Retries with backoff, and a dead-letter queue after N failures.
- Long jobs send heartbeats or save checkpoints, so a restart can resume.
Deep Dive — A nightly job on three instancesDeep dive
The service runs three replicas for availability. A @Scheduled annotation runs on all three, so the nightly billing job runs three times.
Leave @Scheduled on every instance
Annotate the method and deploy.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
CRON["02:00 - the trigger fires"] --> I1["Instance 1 runs the job"]
CRON --> I2["Instance 2 runs the job"]
CRON --> I3["Instance 3 runs the job"]
I1 --> BILL["Invoices generated"]
I2 --> BILL2["Invoices generated again"]
I3 --> BILL3["And again"]
BILL3 --> HARM["Customers billed three times"]Horizontal scaling silently multiplies every scheduled side effect. The failure scales with the replica count, so it gets worse exactly when the service is scaled up for reliability.
Run the scheduler on one instance only
Use a profile or a flag so only one replica has scheduling enabled.
The duplication stops, and a single point of failure has been introduced into the thing that was made highly available: if that instance is down or mid-deploy at 02:00, the job simply does not run, and nothing reports it. The configuration is also easy to lose in a rolling deploy.
Let the instances compete for a lock
Keep the scheduler enabled everywhere and make the execution exclusive:
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
CRON["02:00 on all three instances"] --> LOCK{"Acquire the lock - DB row or Redis"}
LOCK -->|"won"| RUN["One instance runs the job"]
LOCK -->|"lost"| SKIP["Others skip immediately"]
RUN --> TTL["Lock has a TTL - a crashed holder does not block forever"]
RUN --> DONE["Release on completion"]
ALT["Alternative: enqueue the work"] --> Q[("Queue")]
Q --> W["Workers - retries, visibility, scaling for free"]- Any instance can run it, exactly one does. ShedLock with a lock row, or Quartz in clustered mode, gives this in a few lines and keeps the redundancy the deployment was built for.
- The lock needs a TTL, or an instance that dies mid-job holds it forever and the job never runs again — replacing duplication with silent omission.
- Make the job idempotent anyway. Locks reduce the probability of a double run; they do not eliminate it under partition or clock skew, and a billing job should survive being run twice regardless.
For anything long-running, the better answer is to enqueue rather than execute: the scheduled trigger just publishes a message, and ordinary workers process it with retries, visibility timeouts and independent scaling. That also keeps a long job out of the request-serving instances, which is the same reason heavy work does not belong in a controller.
Observability
- Metrics (Micrometer → Prometheus): request latency, error rate, queue depth, job duration and failures.
- Structured logs with a trace ID (Spring Cloud Sleuth / OpenTelemetry) across the API and workers.
- Alerts on queue backlog growth and on job failure rates.
Wrap-UpWrap-up
Run multiple stateless Spring Boot instances across zones behind a load balancer, with Actuator liveness and readiness checks, externalized sessions, graceful shutdown with rolling or blue-green deploys, right-sized connection pools and an HA database. Move long-running work to a queue: the API returns 202 with a job ID, and idempotent workers process messages at-least-once with retries, a DLQ and checkpoints. Use ShedLock or clustered Quartz so scheduled jobs run once, and monitor everything.