Problem RestatementProblem
Design a CI/CD system. CI (continuous integration) means every code push is built and tested automatically. CD (continuous delivery or deployment) means passing builds are deployed. When a developer pushes code, the system creates a pipeline: a set of jobs like build, unit tests, integration tests and deploy, where some jobs depend on others. Jobs run on a pool of worker machines, and users watch live status and logs.
Interviewers often focus on: scheduling jobs with dependencies, handling stuck jobs (a worker dies silently), and build caching to make builds fast.
RequirementsRequirements
1.1 Functional
- Trigger pipelines from pushes, pull requests or a schedule.
- Run jobs in dependency order (a DAG), in parallel where possible.
- Match jobs to workers with the right capabilities (Linux, macOS, GPU).
- Stream live logs, and show status per job.
- Retry, cancel and re-run jobs.
- Cache dependencies and build outputs.
1.2 Non-Functional
- Reliable: no job stuck forever, and no job silently lost.
- Fast: start jobs within seconds, and use caches to shorten builds.
- Isolation: one team's build can't read another's secrets.
- Scale: thousands of repos and hundreds of thousands of jobs per day.
1.3 Scale Estimates
- 500K jobs/day, peaking during work hours at ~30 job starts/sec.
- Average job: 5 minutes → about 2,000 jobs running at once, so 2,000+ workers at peak.
- Logs: 1 MB per job average → 500 GB/day of logs.
1.4 API Design
/v1/hooks/git{ repo, commit, branch }/v1/pipelines/{id}→ jobs and status/v1/jobs/{id}/logs?follow=true(streaming)/v1/jobs/{id}/retry, POST /v1/pipelines/{id}/cancel/v1/workers/lease→ a job; POST /v1/jobs/{id}/heartbeat; POST /v1/jobs/{id}/completeHigh-Level ArchitectureArchitecture
2.1 Overview
- Trigger Service: receives webhooks, reads the pipeline config file (e.g., YAML in the repo) and creates the pipeline and its jobs.
- Scheduler: tracks the job DAG. When all of a job's parents succeed, the job becomes "ready" and goes to the queue that matches its labels (e.g.,
linux-large). - Workers / runners: ask for jobs they can run (pull model), run each job in a fresh container or VM, heartbeat, and upload logs and artifacts.
- Log Service: receives log chunks and streams them live to browsers, then stores them in object storage.
- Cache Service: stores dependency caches and build outputs by key.
- State DB: pipelines, jobs and their states.
2.2 Architecture Diagram
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
GIT["Git host"] -->|"webhook"| TR["Trigger Service"]
TR --> DB[("Pipelines and Jobs DB")]
SCH["DAG Scheduler"] --> DB
SCH -->|"ready jobs by label"| Q[("Job queues")]
W["Workers - fresh container per job"] -->|"lease job"| Q
W -->|"heartbeat, status"| SCH
W -->|"log chunks"| LS["Log Service"]
LS --> OS[("Object storage - logs, artifacts")]
W <-->|"get/put cache"| CS["Cache Service"]
UI["Web UI"] --> LS
UI --> DBData ModelData model
pipelines: pipeline_id, repo, commit_sha, trigger, status, created_at
jobs: job_id, pipeline_id, name, labels, depends_on[], status
(pending, ready, leased, running, succeeded, failed, cancelled),
attempt, worker_id, lease_until, started_at, finished_at
artifacts: job_id, name, storage_key, sizeKey FlowsFlows
4.1 Running a pipeline
- A push arrives. The trigger reads the config at that commit and creates jobs with their dependencies.
- The scheduler marks jobs with no parents as
readyand enqueues them. - A worker with matching labels leases a job (
lease_until = now + 60s), pulls the code, restores caches, runs the steps and streams logs. - On finish, the worker reports the result. The scheduler marks children ready once all their parents have succeeded. If a parent fails, its children are skipped.
4.2 Live logs
The worker sends log chunks every second. The log service appends them to a short-term buffer (e.g., Redis streams) that browsers read over SSE/WebSocket. When the job finishes, the full log is written to object storage.
Deep Dive A — Jobs that never come backDeep dive
A worker can crash, lose the network, or simply hang on a test that waits forever. From the scheduler's side all three look identical: a job marked running that stops saying anything.
Mark it running and wait for the worker
The scheduler sets status = running and waits for the worker to report success or failure.
A crashed worker never reports either. The job sits in running until a human notices, the pull request never gets a result, and the executor slot stays allocated. One bad host quietly eats capacity all afternoon.
Heartbeats and leases
The worker renews a lease every 15 seconds. A reaper scans for lease_until < now, fails that attempt as "lost worker", and re-queues it if retries remain.
Stuck jobs now resolve themselves, and this is most of the answer. The hole is the worker that was never dead — a long GC pause, a network partition — whose lease expired while it kept running. The retry starts, and then the original worker reports success for the same job. Two attempts write results, and the last writer wins at random.
Lease, fence, and one terminal state
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
sequenceDiagram
participant W1 as Worker A - attempt 1
participant S as Scheduler
participant W2 as Worker B - attempt 2
W1->>S: heartbeat, lease renewed
Note over W1: network partition - heartbeats stop
S->>S: lease expired - attempt 1 marked lost
S->>W2: run attempt 2
W2->>S: success - attempt 2
S->>S: job terminal - passed
W1->>S: success - attempt 1 - arrives late
S-->>W1: rejected - stale attemptThree rules together:
- Fencing by attempt number. Every result carries the attempt it came from, and the scheduler only accepts the current one. The zombie's late success is discarded instead of overwriting a real failure.
- One terminal state. The state machine allows a single transition out of
running. Once a job ispassedorfailedit cannot be moved again, whatever arrives afterwards. - A hard timeout on top of the lease — say 60 minutes. A worker that is alive and heartbeating but wedged on a hanging test renews its lease forever; only a wall-clock cap ends that.
Cancellation falls out of the same machinery: set cancelled, and the worker sees it in the response to its next heartbeat and tears down the container.
Deep Dive B — Build cachingDeep dive
Rebuilding everything on every push is slow. There are two kinds of cache:
- Dependency cache: e.g.,
node_modulesor the Maven repo. The key is a hash of the lockfile (hash(package-lock.json)). If the lockfile didn't change, restore the saved archive. - Build output cache: compiled outputs keyed by a hash of the inputs (source files + compiler version + flags), as Bazel does. The same inputs give the same output, so we can skip the step entirely.
- Container layer cache: reuse Docker image layers that didn't change.
Never let untrusted pull requests (e.g., from forks) write to the shared cache, or they could poison other builds.
Trade-offs & AlternativesTrade-offs
| Decision | Choice | Why | Alternative |
|---|---|---|---|
| Work assignment | Workers pull (lease) jobs | Workers only take what they can run, easy scaling | Scheduler pushes to workers: needs worker tracking |
| Isolation | Fresh container/VM per job | Clean, secure builds | Reused machines: faster, leaks state between jobs |
| Failure detection | Leases + heartbeats + timeouts | Catches crashes and hangs | Rely on worker reports: jobs stuck forever |
| Caching | Content-hash keys | Correct reuse | Time-based caches: stale or wrong builds |
Common Follow-up QuestionsFollow-ups
- "How do you deploy safely?" Deploy jobs roll out gradually (canary → 10% → 100%), watch health metrics, and roll back automatically on errors. For AI services, also run an evaluation suite before promoting a model.
- "How do you handle secrets?" Store them in a vault, inject them only into jobs of that repo and branch, and mask them in logs.
- "Flaky tests?" Track pass/fail history per test, auto-retry known flaky tests once, and report flakiness to owners.
Wrap-UpWrap-up
Turn pushes into a DAG of jobs, enqueue ready jobs by worker label, and let workers lease jobs and heartbeat while running them in fresh containers. Leases, timeouts and fenced attempt numbers catch stuck and zombie jobs. Stream logs through a buffer to the UI and store them in object storage, and speed builds with content-hash keyed caches that untrusted builds can't write to.