Problem RestatementProblem
Design an authorization system for a product like Confluence, Jira or Snowflake. It must answer one question very fast and very often: "Can user U do action A on resource R?"
It supports two models together:
- RBAC (Role-Based Access Control): users (or groups) get roles such as Admin, Editor or Viewer, and each role grants a set of permissions.
- Resource ACLs (Access Control Lists): a specific page or project can grant or deny access to specific users or groups ("share this page with Priya").
Resources are often nested (a space contains pages, and pages contain sub-pages), and permissions usually inherit from the parent.
RequirementsRequirements
1.1 Functional
- Manage users, groups (including nested groups), roles and permissions.
- Assign roles at different scopes (whole organization, one project, one page).
- Grant or deny access on individual resources.
- Inheritance: a page follows its space's permissions unless overridden.
check(user, action, resource)→ allow/deny, plus "list resources this user can see".- An audit log of permission changes.
1.2 Non-Functional
- Fast: checks in under ~5 ms, since every request does several.
- Correct: a revoked user must lose access quickly (seconds).
- Scalable: millions of users and resources, 100K+ checks/sec.
1.3 Scale Estimates
- 10M users, 1B resources, 100M ACL entries.
- 200K checks/sec at peak. Most can be served from a cache.
1.4 API Design
/v1/check{ user_id, action: "page.edit", resource: "page:123" } → { allowed: true }/v1/roles/assign{ principal: "group:eng", role: "editor", scope: "space:9" }/v1/acl{ resource: "page:123", principal: "user:42", permission: "view", effect: "allow|deny" }/v1/users/{id}/resources?type=page&action=view(list)High-Level ArchitectureArchitecture
2.1 Overview
- Policy Admin API: changes roles, groups and ACLs, and writes to the DB plus an audit log.
- Authorization DB: the relational schema below (source of truth).
- Check Service: evaluates permissions. Runs close to the apps (sidecar or library) with a local cache.
- Change stream: publishes permission changes so caches can be invalidated.
2.2 Architecture Diagram
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
APP["Product services"] -->|"check user, action, resource"| CK["Check Service + cache"]
CK --> DB[("Authorization DB")]
ADM["Admin UI / APIs"] --> PA["Policy Admin API"]
PA --> DB
PA --> AUD[("Audit log")]
PA --> K[("Permission change events")]
K -->|"invalidate"| CKData ModelData model
users(user_id), groups(group_id), group_members(group_id, member_type, member_id) -- nested groups
roles(role_id, name) -- Admin, Editor, Viewer
role_permissions(role_id, permission) -- 'page.view', 'page.edit', ...
role_assignments(principal_type, principal_id, role_id, scope_type, scope_id)
resources(resource_id, type, parent_id) -- page → space → org
acl_entries(resource_id, principal_type, principal_id, permission, effect) -- allow / denyHow a Check Works
For check(user 42, page.edit, page:123):
- Find the principals: user 42 plus all groups they belong to, following nested groups (cache this per user).
- Walk up the resource tree: page:123 → space:9 → org:1.
- At each level, look for:
- ACL entries for any of the user's principals with
page.edit. - Role assignments at that scope whose role includes
page.edit.
- An explicit deny beats an allow at the same level.
- The nearest level wins (a page-level rule beats a space-level rule), unless the org enforces a rule that can't be overridden.
- The default is deny.
Deep Dive A — Answering 200,000 permission checks a secondDeep dive
Every request does several checks, and each one has to come back in a few milliseconds. The naive check is a multi-way join, run at the rate of the whole platform's traffic.
Query the database on every check
Expand the user's groups, walk the resource's ancestors, join role assignments and ACL entries, apply precedence, return allow or deny.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
REQ["200,000 checks/sec"] --> J["Nested groups + ancestor walk + role joins + ACL scan"]
J --> DB[("Authorization DB")]
DB --> LAT["Tens of ms per check, several checks per request"]
DB --> CAP["Authorization DB becomes the platform's bottleneck"]Correct and unusably slow. Worse, the authorization database is now on the critical path of every request in the company, so its availability caps everything else's.
Cache the answer
Cache (user, action, resource) → allowed for 30–60 seconds and invalidate on permission change events.
Hit rates are good and latency collapses. Two gaps remain. The cache key is extremely specific, so a user touching many distinct pages misses constantly — and every miss is the full join again. And "list all the resources I can see" cannot use this cache at all: you would have to run a check per resource, which for a thousand-item page is a thousand cache misses.
Cache the parts that barely change, and push filtering into the query
The expensive inputs are stable even when the answer is not:
- The user's principals — themselves plus every group, following nested groups. Changes rarely; caching it removes the recursive group expansion from every check.
- The resource's ancestor chain — page → space → org. Changes almost never.
With both cached, a check becomes a small in-memory evaluation of precedence rules rather than a database round trip, and the cache actually hits because those two things are shared across thousands of different checks.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
CK["check(user, action, resource)"] --> P[("Cached: user's principals")]
CK --> A[("Cached: resource ancestors")]
P --> EV["Evaluate in memory - deny wins, nearest wins, default deny"]
A --> EV
EV --> RES["Allow / deny in under a millisecond"]
CH["Permission change event"] --> INV["Invalidate the affected principal / resource entries"]
INV --> P
INV --> A
LIST["List what I can see"] --> FILT["Filter inside the search query by principals and scopes"]
FILT --> PAGE["Then check only the page of results"]For list endpoints, do not check per item. Push the user's principals and scopes into the search or database query as a filter, then run real checks only on the page being returned — twenty checks instead of a million.
Bypass the cache entirely for a small set of sensitive actions. A 60-second window is fine for viewing a page and not fine for exporting a customer list.
Deep Dive B — Scale and consistency (Zanzibar model)Scale
Google's Zanzibar (used for Drive and YouTube) stores everything as relationships: page:123#viewer@group:eng#member. Checks become graph lookups ("is user 42 connected to page:123 via viewer?"). Open-source versions include SpiceDB and OpenFGA.
- It scales by sharding relationships and caching sub-results heavily.
- The "new enemy" problem: Alice removes Bob from a doc, then adds secret content. If a stale cache still lets Bob in, he sees the secret. Zanzibar fixes this with a consistency token (a "zookie") returned when content changes. Checks for that content must use data at least as fresh as the token.
- A simpler approach for most systems: invalidate caches on every permission change, keep cache TTLs short, and for sensitive actions bypass the cache.
Trade-offs & AlternativesTrade-offs
| Decision | Choice | Why | Alternative |
|---|---|---|---|
| Model | RBAC + resource ACLs with inheritance | Covers org roles and sharing | RBAC only: can't share single items |
| Storage | Relational schema | Clear, easy to audit | Relationship graph (Zanzibar): scales further, more complex |
| Speed | Check service with cache + invalidation | Few-ms checks | Query DB on every check: slow |
| Conflicts | Deny wins, nearest wins, default deny | Safe and predictable | Allow wins: risky |
Common Follow-up QuestionsFollow-ups
- "Attribute-based rules?" (e.g., "only during work hours" or "only from the company network"). Add an ABAC layer that evaluates conditions (a policy engine like OPA) after RBAC passes.
- "Multi-tenant?" Every row carries a
tenant_id, and a check never crosses tenants. - "Audit?" Log every permission change (who, what, when), and optionally sampled or all denied checks for security review.
Wrap-UpWrap-up
Store users, nested groups, roles, role assignments per scope and resource ACLs in a relational schema. Resolve a check by expanding the user's groups, walking up the resource hierarchy, and applying clear rules (deny wins, nearest wins, default deny). Serve checks from a nearby service with caches invalidated by change events, and move to a Zanzibar-style relationship store with consistency tokens when scale and strictness demand it.