Problem RestatementProblem
Rippling asked: scale a rules engine that evaluates customer-defined rules against a high volume of events in real time. Example rules in an HR/IT platform:
- "When an employee's department changes to Engineering, add them to the GitHub org and the #eng Slack channel."
- "When a new hire's start date is within 7 days, order a laptop."
- "If an expense > $500 and the category is travel, require manager approval."
Rules as Data
{ "rule_id": "r-91", "tenant": "acme", "version": 3, "enabled": true,
"trigger": { "event": "employee.updated", "field_changed": "department" },
"condition": { "all": [ { "field": "employee.department", "op": "==", "value": "Engineering" },
{ "field": "employee.country", "op": "in", "value": ["US", "CA"] } ] },
"actions": [ { "type": "add_to_group", "app": "github", "group": "eng" } ] }- Trigger: which event type (and optionally which field change) can fire this rule. This is the key to indexing.
- Condition: an expression tree, compiled when saved into fast predicates.
- Actions: side effects run by action workers.
ArchitectureArchitecture
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
EV["Domain events"] --> K[("Kafka - by tenant")]
K --> EVAL["Evaluators - stateless, autoscaled"]
RS["Rule service - save, validate, compile"] --> RDB[("Rules DB - versioned")]
RS -->|"rule changed"| RC[("Compiled rule cache per tenant")]
RC --> EVAL
EVAL --> AQ[("Action queue")]
AQ --> AW["Action workers - idempotent, retries"]
AW --> APPS["Integrations - Slack, GitHub, payroll"]Deep Dive — Evaluating rules at event rateDeep dive
Tenants define thousands of rules. Events arrive continuously. Every event must be checked against every rule that could apply to it.
Loop over all rules, parsing as you go
For each event, load the tenant's rules, parse each condition from JSON, and evaluate.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
E["Event"] --> LOAD["Load all 3,000 rules for the tenant"]
LOAD --> PARSE["Parse each condition from JSON - every time"]
PARSE --> EVAL["Evaluate all 3,000"]
EVAL --> WASTE["2,990 are about other event types"]
EVAL --> ENR["Each rule fetches the same enrichment data separately"]Three separate wastes: evaluating rules that could never match, re-parsing conditions that never change, and fetching the same enrichment once per rule rather than once per event.
Index rules by their trigger
Key rules as (tenant, event_type) → rules so an event only pulls the handful that could apply.
This removes the largest waste and is the single highest-value change. What remains is per-event cost that should be per-deploy cost: conditions are still parsed on every evaluation, and the rule set is fetched from storage rather than held in memory.
Compile once, cache by version, enrich once per event
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
RULES[("Rule definitions - JSON")] --> COMP["Compile to closures or bytecode - on change, not per event"]
COMP --> CACHE["Cached in evaluator memory, keyed by tenant + version"]
CHG["'rules changed' event"] --> INV["Invalidate; periodic version check as backup"]
INV --> CACHE
E["Event"] --> IDX["Index: (tenant, event_type), then equality fields like department == Engineering"]
CACHE --> IDX
IDX --> CAND["Candidate rules - usually a handful"]
E --> ENR["Enrich once per event - shared by all candidates"]
ENR --> EVAL["Evaluate"]- Compile on change, not on use. Rules change a few times a day and are evaluated millions of times, so parsing belongs on the write path. Compiled closures also let the engine hoist common sub-expressions.
- Cache by tenant and version, invalidated by a "rules changed" event with a periodic version check as a backstop — because a missed invalidation means a tenant's rules silently stop taking effect, which is worse than a slow evaluation.
- Index one level deeper where it pays: a simple equality in the condition (
department == Engineering) becomes a hash lookup, cutting candidates again. - Enrich once per event. If several rules need the employee record, fetch it once and pass it to all of them — otherwise the enrichment calls, not the evaluation, become the bottleneck.
Keep the rule definitions as data and the compiled form as a derived cache. That is what lets a tenant change a rule without a deploy while still paying compilation cost only once.
Correctness Issues
- Idempotent actions: an action carries a key
(rule_id, rule_version, event_id), so retries or reprocessing don't add someone to a group twice or order two laptops. - Ordering: per-tenant (or per-entity) ordering, so "department changed to Eng, then back to Sales" doesn't apply in the wrong order.
- Rule updates: evaluation uses the rule version current when the event was processed, and records which version fired (for audit and explainability).
- Loops: an action can generate events that trigger rules again. Limit the chain depth, and detect repeating cycles.
- Consistency vs latency: a rule edit may take a few seconds to reach all evaluators (the cache refresh). Acceptable in most cases. For strict needs, include the version check in the hot path.
Wrap-UpWrap-up
Store rules as versioned data with a trigger, a compiled condition and actions, index them per tenant by event type (and simple equality keys) so each event is evaluated only against relevant rules, and cache compiled rule sets in stateless evaluators that consume a tenant-partitioned event stream. Run actions through a queue with idempotency keys and retries, keep per-entity ordering, record which rule version fired, and guard against rule loops.