Problem RestatementProblem
Design a platform where thousands of services send their logs to one place, and engineers can search them within seconds. For example: "show all ERROR logs from the checkout service in the last 15 minutes that contain 'timeout'". Logs must not be lost when there are traffic spikes, old logs should be kept cheaply, and access to sensitive logs must be controlled.
RequirementsRequirements
1.1 Functional
- Collect logs from apps, containers and hosts.
- Parse logs into fields (time, level, service, message, trace ID).
- Search by time range, fields and keywords, and tail logs live.
- Keep logs for a set period (e.g., 7 days fast, 90 days cheap, 1 year archive).
- Per-team access control.
1.2 Non-Functional
- Durable: no lost logs, even during spikes.
- Fresh: searchable within ~10 seconds.
- Fast search for recent data (seconds).
- Cost-efficient: log volume is huge.
1.3 Scale Estimates
- 10,000 services, 2 million log lines/sec at peak.
- 300 bytes per line → about 50 TB/day raw. With compression (~10x), about 5 TB/day stored.
- 7 days hot = 35 TB on fast disks. 90 days in object storage.
1.4 API Design
- Agents send batches:
POST /v1/logs(or the Kafka protocol directly). - Search:
GET /v1/search?q=service:checkout AND level:ERROR AND "timeout"&from=-15m&limit=500 - Live tail: a WebSocket stream for a query.
High-Level ArchitectureArchitecture
2.1 Overview
- Agents (Fluent Bit, Vector) on each host: read log files, batch them, compress them, and send them. They buffer on local disk if the backend is slow.
- Kafka: a large durable buffer. If indexing falls behind, logs wait here instead of being dropped.
- Processors: parse text into fields, add metadata (host, region, team), and mask secrets such as passwords or card numbers.
- Indexers: write logs into a search engine (Elasticsearch/OpenSearch), using one index per day per tenant.
- Object storage: cheap long-term storage for older logs.
- Query service: searches hot indexes, and older data when asked.
2.2 Architecture Diagram
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
S["Services + Agents"] --> K[("Kafka - buffered")]
K --> P["Parse, enrich, mask secrets"]
P --> IX["Indexers"]
IX --> ES[("Search cluster - hot, 7 days")]
P --> OS[("Object storage - compressed, 90 days+")]
ES -->|"age out"| OS
U["Engineers / UI"] --> Q["Query Service"]
Q --> ES
Q --> OSData ModelData model
Each log becomes a document:
{ "ts": "2026-09-19T10:15:02.331Z", "level": "ERROR", "service": "checkout",
"host": "web-17", "region": "us-east", "trace_id": "ab12...", "message": "payment timeout after 3000ms" }- Indexes are split by time (e.g.,
logs-checkout-2026.09.19). Most searches are about recent time, so we only search a few indexes, and deleting old data just means dropping a whole index. - Fields like
serviceandlevelare exact-match (keyword) fields.messageis full-text indexed.
Key FlowsFlows
4.1 Ingest
- The agent reads new lines, batches them for up to 1 second or 1 MB, compresses the batch, and sends it.
- If Kafka is unreachable, the agent writes to a local disk buffer and retries.
- Processors turn raw text into fields (JSON logs are easy, and plain text uses patterns), then send documents to indexers and a compressed copy to object storage.
4.2 Search
- The query service picks the indexes that match the time range and the user's allowed teams.
- It runs the query on all shards in parallel and merges the newest results first.
- For data older than 7 days, it scans the compressed files in object storage. That is slower but cheap.
Deep Dive A — The spike that arrives exactly when you need the logsDeep dive
During an incident, error volume jumps ten-fold. That is the moment engineers open the search box, and the moment the ingestion path is most likely to fall over.
Agents write straight to the search cluster
Each agent posts batches to the indexers. One hop, no moving parts, logs are searchable a second after they are written.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
A["Agents - 10x volume during the incident"] --> IX["Indexers at capacity"]
IX --> REJ["429s and timeouts"]
REJ --> DROP["Agent buffer fills - oldest logs dropped"]
DROP --> GAP["The incident's logs are the ones missing"]Indexing is the expensive step, so the indexers saturate first. They push back, agent buffers fill, and the logs that get dropped are precisely the ones from the minutes under investigation. The system fails in the only window where it mattered.
Put Kafka in front
Agents produce to Kafka; indexers consume at whatever rate they can sustain. The burst lands in a durable log with hours of retention, and the indexers catch up afterwards.
This converts loss into delay, which is the right trade. What it does not do is protect services from each other. One service stuck in a logging loop can fill the partitions and push every other service's logs behind hours of its noise — still "not lost", still useless for the person waiting on them.
Buffer, then budget per service
Kafka for durability, plus two controls that decide who the delay lands on:
- Per-service quotas. A service over its rate is sampled or capped at ingestion. Its logs still go to object storage in full, so nothing is destroyed — it just stops occupying the shared indexing pipeline.
- Agent-side backpressure. When the agent cannot ship, it slows down and buffers on local disk rather than consuming the host's bandwidth. A logging agent must never be the reason the service it is watching gets slower.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
A["Agents - local disk buffer, backpressure"] --> K[("Kafka - hours of retention")]
K --> Q{"Service over quota?"}
Q -->|"yes"| SAMP["Sample or cap - full copy still archived"]
Q -->|"no"| IX["Indexers"]
K --> OBJ[("Object storage - everything, always")]
IX --> SRCH["Search - seconds behind, minutes when catching up"]State the guarantee in the interview in one line: logs are delayed, never lost, and one team's noise cannot take another team's search away. Both halves are needed — the first without the second is a promise you break during every incident.
Deep Dive B — Cost controlDeep dive
Full-text indexing everything is expensive (the index can be as big as the data).
- Tiering: hot (fast SSD, 7 days) → warm (cheaper disks, 30 days) → cold (object storage, months).
- Index less: index only key fields and store the message compressed. This is the Grafana Loki approach: much cheaper, with slower keyword search.
- Drop noise: filter DEBUG logs in production, and sample repetitive INFO logs.
- Retention per team: teams pay for (or justify) longer retention.
Trade-offs & AlternativesTrade-offs
| Decision | Choice | Why | Alternative |
|---|---|---|---|
| Buffer | Kafka | No loss during spikes, replay | Direct to indexers: simpler, drops logs under load |
| Index | Full-text for hot data | Fast keyword search | Label-only index (Loki): 5–10x cheaper, slower search |
| Index layout | Per day per tenant | Fast time queries, easy deletes | One huge index: slow and hard to clean |
| Old data | Object storage | Very cheap | Keep in search cluster: fast but costly |
Common Follow-up QuestionsFollow-ups
- "How do you link logs to a request?" Put a
trace_idin every log line, so one search shows the whole journey of a request across services. - "How do you protect sensitive data?" Mask secrets at the processor, restrict indexes by team, and audit who searched what.
- "Alerts on logs?" Run saved searches every minute (e.g., more than 100 "payment failed" in 5 minutes), or turn log patterns into metrics.
Wrap-UpWrap-up
Agents batch and buffer logs into Kafka, processors parse, enrich and mask them, and indexers write time-based indexes for fast recent search, with compressed copies in object storage for cheap retention. Kafka and per-service quotas protect the system during spikes, and tiering plus selective indexing keeps costs under control.