Problem RestatementProblem
Companies set expense policies, such as:
- "Meals over $75 need a receipt."
- "No alcohol purchases."
- "Hotels in New York max $300/night; elsewhere $200."
- "Flights over $1,000 need manager pre-approval."
When employees submit expenses (often from a corporate card, automatically), each expense must be checked against the applicable rules. Violations are flagged with a clear reason and routed to approvers. At month-end, volume spikes. Rules change often, and each company (tenant) has its own. Rippling asked this, sometimes together with related pieces: a tagged event counter, a driver-pay ledger, and an employee-termination workflow across external systems.
RequirementsRequirements
- Tenants define rules with conditions (amount, category, merchant, location, employee department or level, receipt present, ...) and actions (flag, block, require approval).
- Evaluate each new or edited expense against that tenant's active rules.
- Explain each violation ("Meal $92 > $75 limit for Sales department").
- Route flagged expenses to approvers, and track resolution.
- Handle month-end spikes. Rule changes apply without downtime.
1.1 Scale Estimates
- 50K companies, 50M expenses/month, with a month-end peak of ~2K expenses/sec.
- Rules per company: 10–500.
Rule Representation
Store rules as data, not code:
{
"rule_id": "r-17", "tenant_id": "acme", "version": 4, "active": true,
"scope": { "categories": ["meals"], "departments": ["sales", "support"] },
"condition": { "all": [
{ "field": "amount_usd", "op": ">", "value": 75 },
{ "field": "has_receipt", "op": "==", "value": false } ] },
"action": { "type": "flag", "severity": "medium",
"message": "Meals over $75 need a receipt" }
}- Scope fields (category, department, country) are used for indexing, meaning quickly narrowing which rules could apply.
- Condition is a small expression tree (all, any, not, comparisons). It's validated when saved.
- Rules are versioned, so every decision records which rule version was used.
ArchitectureArchitecture
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
CARD["Card feed / employee app"] --> EXP["Expense Service"]
EXP --> K[("Expense events - by tenant")]
K --> EV["Policy evaluators"]
EV --> RC[("Compiled rules cache per tenant")]
ADM["Admin rule editor"] --> RS["Rule Service"]
RS --> RDB[("Rules DB - versioned")]
RS -->|"rule changed"| RC
EV --> VDB[("Violations + decisions")]
VDB --> APPR["Approval workflow"]
APPR --> N["Notify approvers"]- Expense events go to Kafka, partitioned by tenant, so bursts are absorbed and order is kept per tenant.
- Evaluators (stateless, autoscaled) load a tenant's compiled rule set from a cache.
- Rule Service saves new versions, compiles and validates them, and publishes a change event that refreshes caches.
Deep Dive — Evaluating thousands of rules per expenseDeep dive
A large customer has thousands of policy rules, and a corporate card feed submits expenses continuously. Every expense has to be checked against every rule that could apply to it.
Loop over all the rules
For each expense, iterate the company's full rule list and evaluate each one.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
E["Expense - meal, $82, New York"] --> LOOP["Evaluate all 4,000 rules"]
LOOP --> IRR["3,990 are about flights, hotels, mileage, software"]
IRR --> WASTE["Almost all the work is on rules that could never match"]
LOOP --> SPIKE["Month-end: 100x volume x 4,000 rules"]The cost is the product of expenses and rules, and both grow with the customer. At month-end, when everyone files at once, that product is exactly when the system is least able to absorb it.
Index rules by what they apply to
Store rules under their selector — category, merchant type, country — and fetch only the candidates for this expense. A New York meal pulls the handful of meal rules and the location rules covering New York.
The work drops from thousands to tens, which is most of the win. Two things remain. Rules are still interpreted one at a time from their stored representation, re-parsing the same conditions on every evaluation. And nothing defines what happens when several rules match with different verdicts.
Compile once, evaluate in a defined order, and version everything
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
R[("Rule definitions")] --> COMP["Compile to an evaluation plan - indexed by selector, conditions pre-parsed"]
COMP --> VER["Versioned - policy v41, effective from a date"]
E["Expense"] --> SEL["Select candidates by category, merchant, country, amount band"]
VER --> SEL
SEL --> EVAL["Evaluate in precedence order"]
EVAL --> OUT["Verdict: auto-approve / needs receipt / needs approval / violation"]
OUT --> WHY["Record which rule decided, and the policy version"]- Compile the rules into a prepared plan when they change, not when an expense arrives. Rules change a few times a month; expenses arrive continuously.
- Define precedence explicitly. When "meals over $75 need a receipt" and "no alcohol" both match, the outcome must be determined by a stated order — most specific wins, violations beat warnings — not by list position.
- Version the policy and stamp the verdict. An expense is judged against the policy in force on its date. Without this, editing a rule silently re-decides last quarter's approved expenses, and no one can reconstruct why something was approved.
- Record which rule fired. "Needs a receipt" is unhelpful; "needs a receipt: meals over $75, policy v41" is actionable, and it is what an auditor asks for.
For month-end, the engine is stateless and CPU-bound, so it scales horizontally — put the expense feed on a queue and add workers. The thing to protect is the approval notifications, which fan out to managers: batch those into a digest rather than sending one message per expense.
Key FlowsFlows
- An expense arrives (card swipe or manual). The service stores it and emits an event.
- The evaluator enriches it, finds candidate rules, and evaluates them.
- It saves the decision:
{ expense_id, rule_id, rule_version, result, message }, which makes the result explainable and auditable. - Violations with the "require approval" action create approval tasks. "Block" actions stop reimbursement. "Flag" actions show a warning.
- If the employee edits the expense (e.g., adds a receipt), it's re-evaluated.
Month-End Spikes and Reliability
- Kafka buffers the spike, and evaluators autoscale on consumer lag.
- Processing is idempotent: a decision is keyed by
(expense_id, expense_version, ruleset_version), so retries don't duplicate violations. - One huge tenant can't starve others: use fair scheduling across tenant partitions, or dedicated capacity for very large tenants.
Trade-offs & AlternativesTrade-offs
| Decision | Choice | Why | Alternative |
|---|---|---|---|
| Rules | Data (JSON expressions), versioned | Admins edit without deploys, auditable | Code per customer: unscalable |
| Matching | Scope indexes, then evaluate | Few rules checked per expense | Evaluate all rules: wasteful |
| Processing | Async via Kafka | Absorbs spikes | Synchronous on submit: slow at peaks |
| Explainability | Store rule version + message per decision | Clear to employees and auditors | Just "violation": confusing |
Wrap-UpWrap-up
Represent policies as versioned JSON rules with a scope (for indexing) and a condition tree (compiled when saved). Stream expenses through Kafka partitioned by tenant into autoscaled evaluators that enrich each expense, pick candidate rules via scope indexes, evaluate them, and store explainable, idempotent decisions that drive approval workflows. Refresh compiled rule caches on change, and re-evaluate open expenses when policies change.