Problem RestatementProblem
Design a service that processes images, for example resizing, applying filters, or running an ML model (classification, captioning). Clients submit jobs (one image or a batch of millions) and get results later. Anthropic asked it as "start with one worker, then evolve safely to many concurrent processors" and "how would you scale batch image pipelines?".
RequirementsRequirements
- Submit a job (one image or a batch manifest), and check status and progress.
- Process each image with one or more steps, and store the outputs.
- No lost images and no duplicate outputs.
- Handle both interactive requests (fast) and huge batch jobs (throughput).
1.1 Scale Estimates
- Interactive: 200 images/sec, target a few seconds each.
- Batch: jobs of 10M images, finished within hours → ~1–3K images/sec for that job.
- Processing: 50–500 ms CPU per image (or GPU for ML).
Stage 1: One Worker
A single process reads jobs from a DB table, processes them, and writes results. It's simple and fine for low volume. But when we add a second worker, both may take the same job, and if a worker crashes mid-job, the job is stuck forever. We need a queue with proper claiming.
Stage 2: Queue + Many Workers
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
C["Client"] --> API["Job API"]
API --> DB[("Jobs + tasks DB")]
API --> SPL["Splitter - batch to tasks"]
SPL --> QI[("Interactive queue")]
SPL --> QB[("Batch queue")]
QI --> W["Worker pool - autoscaled"]
QB --> W
IN[("Input images - object storage")] --> W
W --> OUT[("Outputs - object storage")]
W --> DB
W -->|"failed 5 times"| DLQ[("Dead-letter queue")]- Job API: creates a job. For batches, the client uploads a manifest (a list of image URLs) to object storage.
- Splitter: turns a batch into tasks (one per image, or small groups of ~100 for efficiency) and enqueues them.
- Queues: interactive and batch are separate, so a 10M-image batch doesn't delay a user waiting for one image.
- Workers: pull a task, download the input, process it, upload the output, and mark the task done.
- Progress: count completed tasks per job (e.g., an atomic counter), and show
done / total.
Deep Dive — Choosing how big a task should beDeep dive
A job arrives with two million images. How that job is cut into queue messages decides throughput, failure behaviour and whether anyone can see progress.
One task for the whole job
The job goes on the queue as a single message, and one worker processes two million images.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
J["Job - 2,000,000 images"] --> M["1 queue message"]
M --> W["One worker, one machine"]
W --> SLOW["No parallelism - days"]
W --> CRASH["Crash at image 1,900,000"]
CRASH --> ZERO["Lease expires, task retried from image 0"]The fleet sits idle while one machine works, and because the unit of retry is the whole job, any failure discards all of it. There is also nothing to report: the job is 0% done until it is 100% done.
One task per image
Enqueue two million messages. Every worker takes one, and parallelism is perfect.
The overhead now dominates the work. A thumbnail resize takes 20 ms; the enqueue, receive, ack and delete around it can cost as much. Two million messages also mean two million lease renewals and acks, and the enqueue step itself becomes a long serial job that has to be made resumable. Perfect parallelism at the cost of spending more on the queue than on the images.
Chunks, with per-image results
Group roughly 100 images into one task. The worker processes them as a batch and reports per-image outcomes.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
J["Job - 2,000,000 images"] --> CH["20,000 chunk tasks of ~100"]
CH --> Q[("Queue")]
Q --> W1["Worker - download threads, process pool, upload threads"]
Q --> W2["Worker"]
W1 --> OUT["outputs/job_id/image_id/step.jpg - same key on a retry"]
W1 --> PROG["Per-image results - progress and partial failures visible"]
W1 -->|"one bad image"| DLQ[("Dead-letter after 5 attempts")]Queue overhead is amortised a hundred-fold while the retry unit stays small: a crash costs one chunk, not a job. Progress is reportable because chunks complete continuously.
Three things that come with it:
- Per-image idempotent outputs. Write to
outputs/{job_id}/{image_id}/{step}.jpg. A retried chunk re-processes images it already finished and writes identical bytes — harmless, and much simpler than tracking partial chunk state. - One bad image must not poison the chunk. Record the failure per image and continue; after repeated failures the image goes to the dead-letter queue, not the whole chunk.
- Keep the worker's pipeline full. Download and upload are I/O-bound while processing is CPU- or GPU-bound, so run download threads → a process pool → upload threads. Otherwise the expensive resource idles during every transfer.
Because chunks are leased and outputs are idempotent, batch work is safe on spot instances — the cheapest capacity available, and retry-safe by construction.
Scaling Batch Jobs
- Autoscale workers on queue depth (and on GPU availability for ML steps).
- Batch tasks: groups of ~100 images reduce queue overhead. The worker processes them and reports per-image results.
- Throughput limits: object storage request rates and network bandwidth can be the bottleneck. Spread keys (prefix by hash) and keep workers in the same region as the storage.
- Fairness: several batch jobs share capacity (round-robin across jobs), and interactive work is always served first.
- Cost: use spot/preemptible machines for batch (it's retry-safe anyway, thanks to leases and idempotent outputs).
Trade-offs & AlternativesTrade-offs
| Decision | Choice | Why | Alternative |
|---|---|---|---|
| Delivery | At-least-once + idempotent outputs | Simple and safe | Exactly-once: very hard, not needed |
| Queues | Separate interactive and batch | Batch can't starve users | One queue: long waits for small jobs |
| Task size | ~100 images per batch task | Less overhead | One per image: queue overhead; huge tasks: slow retries |
| Failures | Retries + dead-letter queue | One bad image doesn't block a job | Retry forever: stuck workers |
Wrap-UpWrap-up
Move from a single worker to a queue-based design where workers lease tasks with visibility timeouts, write outputs to deterministic keys so re-runs are harmless, and retry failures before sending them to a dead-letter queue. Split big batches into grouped tasks, keep interactive and batch queues separate, track progress by counting completed tasks, and autoscale workers (spot machines for batch) on queue depth.