Problem RestatementProblem
Design a platform like LeetCode or Codeforces. Users browse problems, write code in the browser in many languages, and submit it. The system runs the code against hidden test cases and returns a verdict: Accepted, Wrong Answer, Time Limit Exceeded, Runtime Error or Compilation Error. Contests add a timer, a burst of submissions at the start and end, and a live leaderboard.
The hardest part is that we run untrusted code from strangers, so it must be isolated and limited.
RequirementsRequirements
1.1 Functional
- Browse and search problems, and view statements and examples.
- Run code against sample tests, and submit against hidden tests.
- Get a verdict with runtime and memory.
- Contests: registration, timed window, scoring and a live leaderboard.
- Submission history.
1.2 Non-Functional
- Security: user code can never reach the network, other users' data or our servers.
- Fairness: same code gives the same verdict and similar timings.
- Fast feedback: verdict in a few seconds normally.
- Handle spikes: contest start or end can bring 10x traffic.
1.3 Scale Estimates
- 5M daily users, 2M submissions/day ≈ 25/sec average.
- Contest with 50K participants: in the last 10 minutes, maybe 1,000 submissions/sec.
- Each submission runs ~50 tests, taking 2–10 seconds of CPU in total. At 1,000/sec, that is 2,000–10,000 CPU cores busy, so runners must autoscale.
1.4 API Design
/v1/submissions{ problem_id, language: "python3", code, contest_id? } → { submission_id, status: "queued" }/v1/submissions/{id}→ { status, verdict, runtime_ms, memory_kb, failed_test? } (or a WebSocket push)/v1/contests/{id}/leaderboard?page=1High-Level ArchitectureArchitecture
2.1 Overview
- Problem Service: problems, statements and sample tests (cached heavily and served via CDN).
- Submission Service: saves the submission and puts a job on the queue.
- Queue: separate queues for contest submissions (priority) and practice.
- Runner fleet: workers that compile and run code inside a sandbox against test cases.
- Test case store: hidden test files in object storage, cached on runners.
- Result delivery: updates the DB and pushes the verdict to the user.
- Leaderboard Service: updates contest rankings from accepted results.
2.2 Architecture Diagram
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
U["User browser"] --> API["Submission Service"]
API --> DB[("Submissions DB")]
API --> Q[("Job queue - contest priority")]
Q --> R1["Runner - sandbox"]
Q --> R2["Runner - sandbox"]
TC[("Hidden test cases - object storage")] --> R1
TC --> R2
R1 -->|"verdict"| RES["Result handler"]
R2 -->|"verdict"| RES
RES --> DB
RES -->|"push"| U
RES --> LB["Leaderboard Service"]
LB --> RD[("Redis sorted set")]Data ModelData model
problems: problem_id, title, statement, difficulty, time_limit_ms, memory_limit_mb, tests_version
submissions: submission_id, user_id, problem_id, contest_id, language, code_ref,
status (queued, running, done), verdict, runtime_ms, memory_kb, created_at
contest_scores: contest_id, user_id, score, penalty, solved_count, last_accepted_atKey FlowsFlows
4.1 Submit
- Save the submission (code in object storage or DB) with status
queued. - Push a job
{ submission_id }to the queue. Return right away. - A runner takes the job, loads the code and the problem's tests (cached locally by
tests_version), compiles if needed, and runs each test inside a sandbox with limits. - It stops at the first failing test (as most judges do), and returns the verdict with time and memory.
- The result handler saves the verdict and pushes it to the user over WebSocket (or the client polls every second).
4.2 Leaderboard update
When an accepted verdict arrives during a contest, update the user's score (points, plus a penalty for time and wrong tries) in a Redis sorted set, a structure that keeps items ordered by score. Reading the top 100 is then instant.
Deep Dive A — Running code a stranger wroteDeep dive
Every submission is arbitrary code from an anonymous user, and it runs on our machines. Assume it is hostile: it will try to read other submissions, reach the network, fork forever and never exit.
Run it as a process with a timeout
Write the file to /tmp, run the compiler, run the binary with a wall-clock timeout, capture stdout.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
S["Submitted code"] --> P["Process on the runner host"]
P --> F1["Reads /etc and other submissions"]
P --> F2["Opens a socket - exfiltrates or attacks"]
P --> F3["fork bomb - host unusable"]
P --> F4["Writes 40 GB to disk"]A timeout bounds one thing and nothing else. The process shares a file system with every other submission, has the network, and can spawn children that outlive the parent the timeout kills.
A container per submission with resource limits
Run each submission in a fresh container: its own file system and process namespace, a CPU quota and a memory cap through cgroups, a PID limit that stops fork bombs, no network, an unprivileged user with capabilities dropped.
This closes almost everything and is a defensible answer. What is left is that containers share the host kernel. A kernel escape — and they surface regularly — is a full compromise of the runner, and the code running against it was written specifically to try. That risk is acceptable for a build farm you own and not for a public judge.
A kernel boundary, and destroy it afterwards
Put a kernel between the submission and the host: gVisor, which intercepts system calls in userspace, or a Firecracker microVM with its own kernel and a ~100 ms boot.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
SUB["Submission"] --> VM["Firecracker microVM - own kernel"]
VM --> LIM["cgroups - CPU time, 256 MB, PID cap, output cap"]
VM --> RO["Read-only rootfs + small tmpfs"]
VM --> NONET["No network device"]
VM --> SEC["seccomp - dangerous syscalls blocked"]
VM --> DESTROY["Destroyed after the run - nothing survives"]The sandbox is destroyed after every submission rather than reset, so nothing a run leaves behind can reach the next one. Layer the limits anyway — the microVM is the boundary, the cgroup limits and seccomp filters are what stop a run from ruining the host's day without escaping anything.
One thing this does not fix: timing. A container that has to share a core reports different CPU times run to run, which is the next deep dive.
Deep Dive B — Fair timing and contest spikesDeep dive
- Fair timing: measure CPU time, not wall-clock time. Pin each run to a dedicated CPU core, avoid running too many jobs on one machine, and use the same machine type for all runners. For borderline results, re-run a few times and take the minimum.
- Autoscaling: pre-warm extra runners before a contest starts, because they take minutes to boot. Also autoscale on queue length.
- Priority: contest jobs go first. Practice submissions may wait a little longer during contests.
- Test cache: runners keep test files on local disk, keyed by
tests_version, so there's no download per submission.
Trade-offs & AlternativesTrade-offs
| Decision | Choice | Why | Alternative |
|---|---|---|---|
| Execution | Async queue + runners | Absorbs spikes, retries | Run inside the API request: timeouts, unsafe |
| Sandbox | gVisor / Firecracker | Strong isolation | Plain Docker: faster, weaker isolation |
| Result delivery | WebSocket push (polling fallback) | Instant feedback | Polling only: simple, more load |
| Leaderboard | Redis sorted set | O(log n) updates, fast top-N | SQL ORDER BY: slow at contest scale |
Common Follow-up QuestionsFollow-ups
- "How do you support 20 languages?" One runner image per language (or one image with all toolchains), each with language-specific time multipliers (e.g., Python gets extra time).
- "How do you detect plagiarism?" After the contest, compare submissions with token-based similarity tools (like MOSS) and flag suspicious pairs for review.
- "How do you avoid leaking hidden tests?" Don't return full failing inputs for hidden tests in contests. Only show the test number.
Wrap-UpWrap-up
Save each submission and queue it, and let an autoscaled runner fleet execute code in isolated sandboxes (gVisor or Firecracker) with no network and strict CPU and memory limits. Measure CPU time on dedicated cores for fairness, push verdicts back over WebSockets, and keep contest rankings in a Redis sorted set. Pre-warm runners and prioritize contest jobs to survive spikes.