Problem RestatementProblem
Design a service where users upload large files (videos, datasets, product images, documents up to many GB), and the system processes them afterwards: scans for viruses, checks the format, makes thumbnails or runs an analysis, and finally shows a result or publishes the file. Examples include sellers uploading product images, users uploading a file for analysis, or an internal upload portal.
The main ideas: don't push big files through our own servers, make uploads resumable, and do the processing asynchronously so users aren't waiting on a slow request.
RequirementsRequirements
1.1 Functional
- Upload large files, and resume after a network drop.
- Show upload and processing progress.
- Validate, scan and process each file (thumbnails, analysis, moderation).
- Make the result or approved file available for download or display.
1.2 Non-Functional
- Reliable: no lost uploads and no stuck jobs.
- Secure: only allowed users can upload or read, and malware never reaches other users.
- Scalable: thousands of uploads at the same time.
- Cheap bandwidth: our API servers should not carry file bytes.
1.3 Scale Estimates
- 1M uploads/day, average 50 MB → 50 TB/day into storage.
- Peak ~50 uploads starting per second, with thousands in progress at once.
- Processing: average 30 seconds of CPU per file → about 350 CPU cores busy on average, more at peak.
1.4 API Design
POST /v1/uploadswith{ file_name, size, content_type }→{ upload_id, part_size, part_urls: [...] }, where each URL is a pre-signed URL.- The client uploads each part with
PUTstraight to object storage. POST /v1/uploads/{id}/completewith the list of uploaded parts.GET /v1/uploads/{id}→{ status: uploading | processing | ready | rejected, progress, result_url }
A pre-signed URL is a link from object storage (like S3) that lets the holder upload or download one specific file for a short time, without any other credentials.
High-Level ArchitectureArchitecture
2.1 Overview
- Upload API: checks the user's permission and quota, creates an upload record, and hands out pre-signed URLs.
- Object storage (S3/GCS): receives the file parts directly from the client.
- Upload events: when the upload completes, storage (or our API) publishes an event.
- Processing queue + workers: scan, validate, process and write results.
- Metadata DB: upload state, ownership and results.
- CDN: serves approved files and results fast.
2.2 Architecture Diagram
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
C["Client"] -->|"1. create upload"| API["Upload API"]
API --> DB[("Upload metadata DB")]
API -->|"pre-signed part URLs"| C
C -->|"2. PUT parts directly"| S3[("Object storage - raw bucket")]
C -->|"3. complete"| API
API --> Q[("Processing queue")]
Q --> W["Workers: scan, validate, process"]
W --> S3P[("Processed bucket")]
W --> DB
W -->|"failed 3 times"| DLQ[("Dead-letter queue")]
S3P --> CDN["CDN"]
CDN --> V["Viewers"]Data ModelData model
uploads:
upload_id, owner_id, file_name, size, content_type,
status (initiated, uploading, uploaded, processing, ready, rejected, failed),
storage_key, parts_done, checksum, result_key, error, created_at, updated_atKey FlowsFlows
4.1 Uploading (multipart and resumable)
- The client asks to start an upload. The API checks the file type and size, and the user's quota.
- The file is split into parts (e.g., 16 MB each), and each part has its own pre-signed URL.
- The client uploads parts in parallel. If the network drops, it asks which parts are done and uploads only the missing ones. That is what makes it resumable.
- The client calls
complete. Storage joins the parts into one file, and we verify the checksum.
4.2 Processing
- The API sets status to
uploadedand puts a job{ upload_id }on the queue. - A worker downloads the file, runs a virus scan, and checks that the content really matches the type (a ".jpg" that is actually an executable is rejected).
- It processes the file (thumbnails, transcoding, analysis) and writes results to the processed bucket.
- It sets status to
ready(orrejectedwith a reason) and notifies the user by webhook, WebSocket or email.
Deep Dive A — Processing that survives a dying workerDeep dive
A worker picks up a 4 GB video, spends six minutes transcoding it, and the host is reclaimed. What the queue does next decides whether the user sees a thumbnail or a spinner forever.
Take the job off the queue, then work
The worker pops the job — removing it — and starts processing.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
sequenceDiagram
participant Q as Queue
participant W as Worker
participant U as User
W->>Q: pop job - upload 8812
Q-->>W: job removed
W-->>W: transcoding... host reclaimed
U->>U: still "processing" after an hour
Note over Q: no job, no error, nothing to retryThe pop is the only record that the job existed. A crash deletes it, and because nothing failed, no alert fires. The upload sits in processing until somebody complains.
Hide the job instead of deleting it
Use a visibility timeout: the job stays on the queue but is invisible to other workers while one holds it. If the worker does not delete it within the timeout, it reappears and someone else picks it up.
Nothing is lost now. The new problem is that the job can run twice — the slow worker and its replacement both transcode, and both write output. If the write path appends, adds a database row, or names files by attempt, the user ends up with two thumbnails, two rows, or a half-written file overwritten by a finishing one.
Assume it runs twice, and make that harmless
Accept at-least-once delivery and remove the consequences:
- Output keys depend only on the input. Results are written to
processed/{upload_id}/thumb.jpg. Two runs write the same bytes to the same key; the second is indistinguishable from the first. No attempt numbers in paths, no appends. - Database effects are upserts keyed by
upload_id, never inserts. - Retry with backoff, then stop. After three failures the job moves to a dead-letter queue and the upload is marked
failed. A job that retries forever is how one malformed file consumes a worker pool.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
J["Job - upload_id"] --> INV["Invisible while a worker holds it"]
INV -->|"worker dies"| BACK["Reappears on the queue"]
BACK --> INV
INV -->|"done"| OUT["Write processed/upload_id/... - same key every time"]
INV -->|"3 failures"| DLQ[("Dead-letter queue - human looks")]
DLQ --> MARK["Upload marked failed"]One cleanup job that is easy to forget: uploads that were started and never finished. After 24 hours, delete the record and abort the multipart upload. Unfinished multipart parts are invisible in the bucket listing and are billed indefinitely — this is one of the classic quiet cloud bills.
Deep Dive B — SecurityDeep dive
- Scoped tokens: pre-signed URLs work only for one file, one method (PUT), and a few minutes. Download URLs for private files are short-lived too.
- Two buckets: raw uploads go to a private "quarantine" bucket. Only files that pass scanning are copied to the public/processed bucket.
- Authorization for large transfers: use a token (e.g., a JWT) with claims such as
user_id,upload_idandmax_size, and check them on the server. Never trust the client's size or type fields alone. - Limits: max file size, a per-user daily quota and rate limits to stop abuse.
Trade-offs & AlternativesTrade-offs
| Decision | Choice | Why | Alternative |
|---|---|---|---|
| Upload path | Direct to object storage with pre-signed URLs | API servers don't carry GBs | Proxy through API: more control, huge bandwidth cost |
| Resumability | Multipart upload | Retry only failed parts | Single PUT: simple, restart from zero on failure |
| Processing | Queue + workers | Scales, retries, isolates slow work | Process in the request: timeouts, bad UX |
| Status updates | Polling + webhook/WebSocket | Works everywhere | Polling only: simple, more requests |
Common Follow-up QuestionsFollow-ups
- "Files of 100 GB?" Use bigger parts (e.g., 100 MB) and let processing work on chunks in parallel (e.g., video segments).
- "How does the client show a progress bar?" Upload progress comes from the client itself (bytes sent). Processing progress comes from the worker updating
progressin the DB. - "Users far away?" Use storage transfer acceleration or regional buckets, so users upload to the nearest region.
Wrap-UpWrap-up
Hand out short-lived pre-signed URLs so clients upload big files in resumable parts straight to object storage. Then queue a processing job that idempotent workers pick up to scan, validate and process the file, with retries and a dead-letter queue. Keep raw files in a private quarantine bucket, publish only approved results through a CDN, and track every upload's state in a metadata DB.