Problem RestatementProblem
Microsoft asked: design a ticket platform where every new support ticket is:
- linked to the affected user,
- correlated with all other relevant tickets for that user (and ideally similar tickets from other users),
- mapped to the responsible system (the service or component that's probably broken),
- enriched with that system's operational status at the time (was there an outage, a deploy, alerts?).
It must work under high concurrency: when a big service fails, thousands of tickets arrive within minutes.
RequirementsRequirements
- Ingest tickets from email, portal, chat and API.
- Identify the user and tenant (from login, email, account ID).
- Correlate: same user, same issue, same time window, same system, and similar text.
- Classify the responsible system (a service catalog) and attach status snapshots (incidents, alerts, deploys).
- Group tickets into incidents during outages, so one fix closes many tickets.
- Fast search and a view for support engineers.
ArchitectureArchitecture
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
IN["Email, portal, chat, API"] --> ING["Ingestion API"]
ING --> DB[("Tickets DB")]
ING --> K[("Ticket events")]
K --> ENR["Enrichment workers"]
ENR --> ID["Identity service - user, tenant"]
ENR --> CLS["Classifier - responsible system"]
ENR --> ST["Status history - incidents, alerts, deploys"]
ENR --> COR["Correlator - rules + similarity search"]
COR --> VEC[("Search / vector index of recent tickets")]
COR --> INC[("Incident groups")]
UI["Support engineer UI"] --> DB
UI --> INCDeep Dive — Recognising that 400 tickets are one outageDeep dive
Checkout breaks and tickets arrive in a flood. Each one is handled as an individual report unless the system can tell they share a cause.
Group tickets by keyword
Cluster tickets whose text overlaps — "checkout", "payment failed".
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
T["Incoming tickets"] --> KW["Keyword overlap"]
KW --> G1["'payment failed' - today's outage"]
KW --> G2["'payment failed' - a routine expired-card ticket, merged in"]
KW --> MISS["'can't complete my order' - same outage, no shared keyword"]
MISS --> SEP["Filed separately - outage looks smaller than it is"]Users describe the same failure in completely different words, and unrelated tickets share vocabulary. Grouping on surface text both splits one incident and merges distinct ones, so the count — the thing on-call reacts to — is wrong in both directions.
Classify each ticket to a service
Use the product-area fields, keywords and a text classifier trained on past tickets to map a ticket to a service in the service catalogue.
Now tickets group by what they are actually about, and the catalogue supplies an owner, so routing works. But a service label alone does not say whether anything is wrong: forty checkout tickets in an hour may be an outage or may be a Monday. Without knowing the state of the service at the time, the platform still cannot tell.
Classify, then snapshot the service's state at the ticket's time
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
ING["Ingest - save as 'new', publish an event"] --> FAST["Ingestion stays fast during a spike"]
FAST --> ASYNC["Async enrichment"]
ASYNC --> USER["Identify user and tenant - unknown senders flagged"]
ASYNC --> CLS["Classify to a service - fields, keywords, ML classifier"]
CLS --> CAT[("Service catalogue - owners, dependencies")]
CAT --> SNAP["Query incidents, alerts and deploys for that service and its dependencies, at the ticket's timestamp"]
SNAP --> STORE["Store the snapshot on the ticket"]
STORE --> CORR["Correlate: same user, same service, same time window"]- Snapshot the state, do not look it up later. The ticket carries what was happening when it arrived — open incidents, firing alerts, recent deploys for that service and its dependencies. An agent reading it next week sees what the user experienced, not today's state.
- Include dependencies. Checkout tickets during a payment-provider incident are about checkout to the user and about the provider to the engineer; the catalogue's dependency graph is what connects them.
- Correlate on three axes — same user, same service, same time window — so one user's related tickets group together and an outage groups across users.
- Ingest first, enrich asynchronously. During a spike the write path must stay trivial; classification and correlation catch up behind it.
The payoff is the thing support actually needs: a ticket that opens with "412 similar tickets, checkout service, incident INC-2291 open since 14:02" instead of one that has to be diagnosed from scratch.
Handling the Spike
- Queue-based enrichment scales with workers, and ingestion never blocks.
- Dedup and merge: the same user submitting twice → merge. Many users with the same error → group under one incident.
- Idempotent processing (by ticket_id) and conditional updates when attaching to incidents, to avoid race conditions in grouping.
- Bulk actions: resolving an incident resolves or notifies all attached tickets.
Data ModelData model
tickets: ticket_id, user_id, tenant_id, channel, subject, body, created_at, status,
system_id, system_confidence, incident_id, status_snapshot (JSON)
ticket_links: ticket_id, related_ticket_id, reason (same_user | similar | same_incident), score
incidents: incident_id, system_id, started_at, status, ticket_countWrap-UpWrap-up
Ingest tickets fast and enrich them asynchronously: resolve the user and tenant, classify the responsible system from the service catalog with rules and ML, snapshot that system's status at the ticket's time, and correlate with the user's other tickets and similar recent tickets via a search/vector index. Group bursts of similar tickets into incidents (joining active ones or proposing new ones), make processing idempotent to survive outage spikes, and give support engineers a linked, incident-centric view.