Problem RestatementProblem
Amazon asked: design a platform that ingests data from many customers (tenants) and runs processing pipelines on it (validate, transform, enrich, store). Tenants range from tiny (a few MB/day) to huge (TBs/day). The platform must keep each tenant's data isolated, stop one tenant from slowing everyone else ("noisy neighbor"), support per-tenant configuration, and be reliable.
RequirementsRequirements
- Ingest via API, file uploads and streaming connectors.
- Per-tenant schemas and pipeline configuration (which transforms, where to deliver).
- Isolation: tenants never see each other's data, with per-tenant encryption keys.
- Fairness: quotas and fair scheduling. Big tenants can't starve small ones.
- Reliability: retries, dead-letter queues and replay. Per-tenant visibility and cost tracking.
ArchitectureArchitecture
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
T["Tenants - API, files, connectors"] --> GW["Ingestion gateway - auth, quotas"]
GW --> BUF[("Durable buffer - partitioned by tenant")]
BUF --> SCH["Fair scheduler - per-tenant queues"]
SCH --> W["Shared worker pool"]
SCH --> DW["Dedicated workers - premium tenants"]
CFG[("Tenant config + schemas")] --> W
W --> OUT[("Per-tenant storage - separate prefixes/keys")]
W -->|"failures"| DLQ[("Per-tenant dead-letter")]
W --> MET["Metrics + cost per tenant"]Key Design Points
- Tenant identity everywhere: every record carries
tenant_idfrom authentication (never from the payload). Storage paths, topics or partitions, and encryption keys are per tenant. - Quotas at the gateway: requests/sec and bytes/day per tenant (by plan). Over-quota → throttle with 429, or accept into a lower-priority lane.
- Fair scheduling: instead of one shared FIFO queue (where a tenant dumping 1 TB blocks everyone), keep per-tenant queues, and have the scheduler take work round-robin or weighted-fair across tenants. Each tenant also has a max concurrency.
- Tenancy tiers:
- Shared (pooled) workers and storage for most tenants: cheap.
- Dedicated workers, or even separate clusters, for large or regulated tenants: strong isolation, predictable performance, but higher cost.
- Per-tenant config: schemas (validated with a registry), transform steps and destinations, versioned. Config changes apply to new batches, and bad configs are caught by validation plus a dry run.
- Reliability: at-least-once processing with idempotent writes (dedupe keys), retries with backoff, and per-tenant dead-letter queues so one tenant's bad data doesn't block others. Replay from the durable buffer.
Deep Dive — Keeping one tenant's data away from another'sDeep dive
Tenants range from a few megabytes a day to terabytes, sharing one platform. Isolation has to be strong enough for the largest customer's auditors and cheap enough for the smallest.
A tenant id column and a filter in application code
Everything shares tables; every query adds WHERE tenant_id = ?.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
Q1["Query with tenant filter"] --> OK["Correct"]
Q2["New endpoint - filter forgotten"] --> LEAK["Returns every tenant's rows"]
Q3["Analytics job written in a hurry"] --> LEAK
LEAK --> BREACH["Cross-tenant data exposure - found by a customer"]Isolation depends on every query, in every service, forever, being written correctly. It holds until the first one is not, and the failure is a data breach rather than an error.
Row-level security in the database
Enforce the predicate in the database so a session bound to a tenant physically cannot read other rows, regardless of the query.
This is a real boundary — a forgotten WHERE clause now returns nothing instead of everything. Two gaps remain. Large regulated tenants often require their data on separate infrastructure, which no shared table satisfies. And deleting a tenant means finding and removing their rows across every store, which is slow and hard to prove.
Tiered isolation, with per-tenant keys
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
T["Tenant tier"] --> SMALL["Small: shared stores + row-level security"]
T --> LARGE["Large / regulated: separate buckets, databases, sometimes accounts"]
SMALL --> KMS["Per-tenant KMS key"]
LARGE --> KMS
KMS --> ENC["All data encrypted under the tenant's own key"]
OFF["Offboarding"] --> SHRED["Destroy the key - data is unreadable everywhere at once"]
SMALL --> AUD["Per-tenant access logs and audit"]
LARGE --> AUD- Match isolation to the tier. Physical separation per tenant is the strongest answer and it does not scale to thousands of small customers; row-level security does. Offering both, with a documented boundary, is the honest design.
- Per-tenant encryption keys give crypto-shredding. Destroying the key renders that tenant's data unreadable across every store, backup and archive at once — which is a far stronger deletion guarantee than chasing rows, and far easier to evidence.
- Audit per tenant, so a customer can be shown who accessed their data without that report exposing anyone else's.
Whichever tier a tenant is on, quotas are part of isolation too: rate limits and resource budgets per tenant are what stop the terabyte-a-day customer from starving everyone else. Data isolation and performance isolation are separate problems, and a platform that solves only the first still has one tenant able to take the others down.
Observability and Cost
- Per-tenant dashboards: volume, lag, errors, DLQ size and quota usage.
- Cost attribution: CPU-seconds, bytes stored and processed per tenant, used for pricing and for spotting abusive patterns.
Wrap-UpWrap-up
Authenticate tenants at a gateway that enforces quotas, buffer data durably partitioned by tenant, and use a fair scheduler over per-tenant queues (with per-tenant concurrency caps) feeding a shared worker pool, plus dedicated capacity for big or regulated tenants. Carry tenant identity through every step, isolate storage and keys per tenant, apply versioned per-tenant schemas and configs, process at-least-once with idempotent writes and per-tenant dead-letter queues, and track metrics and cost per tenant.