Problem RestatementProblem
Google asked: design a security monitoring framework for cloud infrastructure. Collect security telemetry from everywhere (cloud audit logs, IAM changes, network flow logs, host and container events, application auth logs), detect threats (compromised credentials, privilege escalation, data exfiltration, crypto-mining), alert and prioritize, support investigations, and ideally respond automatically to common cases.
ArchitectureArchitecture
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
SRC["Cloud audit logs, IAM, VPC flows, host agents, app auth logs"] --> COL["Collectors"]
COL --> K[("Kafka - raw security events")]
K --> NORM["Normalize + enrich (asset owner, geo-IP, threat intel)"]
NORM --> DET["Streaming detections - rules + anomaly models"]
NORM --> LAKE[("Security data lake - long retention")]
DET --> ALERT["Alert manager - dedupe, score, route"]
ALERT --> SOC["SOC analysts - cases"]
ALERT --> SOAR["Automated response playbooks"]
SOC --> LAKECollection and Normalization
- Pull or stream from every source: cloud provider audit logs (who did what via the API), IAM and policy changes, network flow logs, DNS logs, host/EDR agents (process launches, file changes), Kubernetes audit logs, and application login logs.
- Normalize into one schema (e.g., OCSF or ECS):
actor, action, resource, source_ip, result, timestamp. - Enrich: asset owner and criticality, user role, geo-IP, threat-intel matches (known bad IPs or domains). This makes detections and triage far better.
Deep Dive — Deciding what counts as an attackDeep dive
Logs arrive from everywhere. Detection is what turns them into a small number of things a human should look at.
Alert on individual suspicious events
Fire an alert whenever a failed login, a permission change or a large download is seen.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
EV["Millions of events/day"] --> R["One alert per suspicious event"]
R --> VOL["Thousands of alerts daily - nearly all benign"]
VOL --> FATIGUE["Analysts triage by skimming, then by ignoring"]
FATIGUE --> MISS["The real intrusion is in the queue, unread"]Individually, almost every security-relevant event is benign — people do fail logins and download files. Alerting on each one produces a queue nobody can work, which is indistinguishable from having no detection at all.
Windowed rules, written as code
Express detections as rules over a time window: "10+ failed logins followed by a success from a new country within 10 minutes". Keep them in Git, version them, test them against sample logs, review them like code.
This is a large step and it is where most of the value is. Detection-as-code means a rule can be tested before it pages anyone, and its history explains why it exists. What rules cannot cover is the attack nobody has written a rule for — and anything genuinely novel is exactly that.
Rules, baselines, and correlation into cases
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
LOGS["Normalised events"] --> RULES["Detection-as-code rules - versioned, tested, reviewed"]
LOGS --> BASE["Per-user / per-service baselines: usual regions, API calls, data volume"]
BASE --> ANOM["Flag large deviations - weak signals"]
RULES --> CORR["Correlation"]
ANOM --> CORR
CORR --> CASE["One case per entity: suspicious login + new IAM grant + bulk download"]
CASE --> RANK["Ranked by severity and asset value"]
RANK --> AN["Analyst - a handful of cases, each with its full timeline"]- Baselines catch what rules cannot. A service that suddenly calls an API it has never called, from a region it has never used, is worth surfacing even though no rule anticipated it — but only as a weak signal, because on its own it is mostly noise.
- Correlation is what makes weak signals usable. A slightly unusual login is nothing; a slightly unusual login plus a new admin grant plus an outbound transfer, on the same principal within an hour, is an incident. Grouping by entity and time is where detection earns its keep.
- Alert on cases, not events. The analyst receives one case with a timeline, not forty rows to reassemble by hand.
Keep specific high-confidence rules as immediate pages — "security logging was disabled" and "a policy granted *:* to a new principal" need no correlation — and route everything else through the case pipeline. The measure of the system is not how much it detects but how little an analyst must read to find it.
Alerting and Response
- Deduplicate and group related alerts into one case, score them by severity × asset criticality × confidence, and route to on-call.
- Automated response (SOAR) for clear cases: disable a leaked access key, isolate a compromised VM, force re-authentication. Risky actions need a human to approve.
- Tuning: track alert precision per rule, and fix or retire noisy rules. Alert fatigue is the biggest practical problem.
Storage and Investigation
- Hot searchable storage for ~30–90 days (fast queries during incidents), and cheaper long-term storage for 1+ years (compliance, long investigations).
- Protect the monitoring pipeline itself: separate accounts, immutable log storage, and alerts if log sources go silent (attackers often disable logging).
Wrap-UpWrap-up
Collect security telemetry from all cloud, network, host and app sources into a streaming pipeline, normalize it to one schema, and enrich it with asset, identity, geo and threat-intel context. Run version-controlled rules, windowed and correlation detections, and behavioral anomaly models in real time (plus scheduled hunts over a security data lake), then dedupe and score alerts into routed cases, automate responses for clear cases, tune rules to fight alert fatigue, and protect the pipeline's own integrity.