Problem RestatementProblem
Snowflake asked: design a system that watches Jira tickets and automatically produces pull requests (PRs). When a ticket is labeled (e.g., auto-fix), an AI coding agent checks out the repository, implements the change, runs tests, and opens a PR linked to the ticket. Many tickets can be processed at once, each in an isolated sandbox, with no duplicate PRs and clear status reporting. The interviewer's focus: async job processing and queue-based architecture.
RequirementsRequirements
- Trigger from Jira (webhook when a label is added, or a comment command).
- Run each job in an isolated sandbox with the repo checked out.
- The agent edits code, runs tests and lint, and iterates up to limits.
- Open a PR (with the description and test results), link it to the ticket, and comment on Jira.
- Retries on transient failures, and escalation to humans on failure.
- Never create duplicate PRs for the same ticket.
ArchitectureArchitecture
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
JIRA["Jira webhook"] --> IN["Intake service - verify, dedupe"]
IN --> DB[("Jobs DB - one active job per ticket")]
IN --> Q[("Job queue")]
Q --> W["Worker pool"]
W --> SB["Sandbox - container/microVM per job"]
SB --> REPO["Git host - clone, branch, push"]
SB --> LLM["LLM agent"]
SB --> CI["Tests / lint"]
W --> DB
W -->|"open PR"| REPO
W -->|"status comments"| JIRAJob Lifecycle
received → queued → running → (tests_passed → pr_opened) | failed | needs_human
- Intake: verify the webhook signature, fetch the ticket, and check eligibility (right project, has a description, repo mapping known). Create a job with a unique constraint on (ticket_id, active), so a second webhook for the same ticket doesn't create a second job. That's idempotency.
- Queue: jobs wait with a priority. Concurrency limits apply per repo (avoid 20 agents fighting over one repo) and overall (cost and GPU/LLM quotas).
- Worker leases a job (with heartbeats, as in any job scheduler) and starts a fresh sandbox: clone the repo, create the branch
auto/JIRA-123, and install dependencies from a cache. - Agent loop: the LLM reads the ticket and code, proposes edits, runs tests and lint in the sandbox, and fixes failures. It's bounded by time, steps and token budget.
- Result: if the tests pass, push the branch and open (or update, if a PR for this branch already exists) the PR with a summary, then comment the link on Jira. If it fails, comment the reason and logs, and set
needs_human.
Deep Dive — One ticket, one pull requestDeep dive
Webhooks are delivered at least once, workers crash mid-run, and tickets get edited while a job is in flight. Each of those can produce a second pull request for the same ticket.
Every webhook starts a job
The auto-fix label fires a webhook; the handler starts an agent run.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
WH["Jira webhook"] --> J1["Job starts"]
WH2["Same webhook redelivered"] --> J2["Second job starts"]
EDIT["Someone edits the ticket description"] --> J3["Third job starts"]
J1 --> PR1["PR #1"]
J2 --> PR2["PR #2 - same change"]
J3 --> PR3["PR #3"]
PR3 --> REV["Reviewers see three competing PRs for one ticket"]Webhook delivery is at-least-once by design, and any ticket update looks like a new trigger. Each duplicate costs a sandbox, an agent run and reviewer attention.
Deduplicate by event id
Record the webhook's event id and ignore repeats.
This removes redelivery duplicates, which is the most common source. It does not help with distinct events for the same ticket — a label added, then the description edited, then a comment — each of which is a genuinely new event id and starts another job against the same ticket.
One active job per ticket, and an idempotent PR step
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
WH["Webhook"] --> DD{"Event id already seen?"}
DD -->|"yes"| IGN["Ignore"]
DD -->|"no"| ONE{"Active job for this ticket?"}
ONE -->|"yes"| POL["Cancel and restart, or finish and note the ticket changed"]
ONE -->|"no"| RUN["Start job - lease + heartbeat"]
RUN -->|"worker crashes"| EXP["Lease expires - retried in a fresh sandbox"]
EXP --> RUN
RUN --> BR["Branch name derived from the ticket id"]
BR --> LOOK{"PR already exists for this branch?"}
LOOK -->|"yes"| UPD["Update it"]
LOOK -->|"no"| CRE["Create it"]- A uniqueness rule at the ticket level, not just the event level: at most one active job per ticket. That is what collapses a burst of distinct events into a single run.
- Derive the branch name from the ticket id, and look for an existing PR on that branch before creating one. A retried job then updates its own pull request instead of opening a sibling — which is what makes crash recovery safe.
- Leases handle crashes: the job is retried in a fresh sandbox, and because the PR step is keyed by branch, the retry converges on the same PR.
- Decide what a mid-flight ticket edit means. Cancel and restart, or finish and annotate that the ticket changed — both are defensible, and leaving it undecided produces both behaviours depending on timing.
Retry transient failures — Git, LLM rate limits — with backoff, and fail permanent ones fast with a message on the ticket. An agent that retries a genuinely impossible task burns sandbox time and tells nobody.
Security
- Sandboxes: no access to production, limited network egress (only Git, package mirrors and the LLM API), ephemeral and destroyed after the job.
- Scoped credentials: a short-lived token that can only push branches to that repo and open PRs. It cannot merge.
- Secrets never go into prompts or logs. Humans review and merge every PR (a quality gate), and CI runs normally on the PR.
Measuring Success
PR acceptance rate, time from ticket to PR, reviewer edits needed, test pass rate, cost per job, and failure reasons (to improve prompts and tooling).
Wrap-UpWrap-up
Receive Jira webhooks through an intake service that verifies them and creates one active job per ticket (idempotently), queue jobs with per-repo and global concurrency limits, and let leased workers run each job in a fresh, locked-down sandbox where an LLM agent edits code and runs tests within budgets. Push a branch and open or update a single PR with scoped, merge-less credentials, report status back to Jira, retry transient failures, escalate the rest to humans, and track acceptance and cost.