Problem RestatementProblem
LinkedIn asked two related questions:
- Malicious IP detection: watch application and network events, detect bad source IPs (scrapers, credential stuffing, DDoS bots), and give every request a low-latency decision: allow, challenge (CAPTCHA), rate-limit or block. It must work across multiple regions.
- Malicious URL checking: when users share links, check them using an existing (slow, rate-limited)
isMalicious(url)API, with caching, rate limiting and fault tolerance.
Both share the same shape: a fast decision path backed by a cache, fed by a slower detection path.
RequirementsRequirements
- Decision API:
check(ip | url)→{ action, reason, ttl }in under ~5 ms at very high QPS. - Detection: find new bad actors from signals within seconds to minutes.
- Propagate new blocks to all regions quickly (under ~30 s).
- Minimize false positives, and support allowlists and appeals.
- For URLs: don't overload the external API, and handle its outages.
1.1 Scale Estimates
- 500K requests/sec across regions need a decision → decisions must be local (in-memory) lookups.
- Known bad IPs: millions. Known URL verdicts: hundreds of millions (cache the popular ones).
ArchitectureArchitecture
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
REQ["Incoming requests"] --> EDGE["Edge / gateway - local decision cache"]
EDGE -->|"miss or suspicious"| DS["Decision Service"]
DS --> RC[("Regional cache - Redis")]
EDGE -->|"request events"| K[("Kafka - security events")]
K --> DET["Detection: streaming features + rules + model"]
TI["Threat intel feeds"] --> DET
DET -->|"new verdicts"| VS[("Verdict store - global")]
VS -->|"replicate + push"| RC
RC -->|"push invalidations"| EDGE
DS -->|"URL cache miss - rate limited"| EXT["isMalicious API"]Fast Path (decisions)
- Each gateway keeps an in-memory set of blocked and challenged IPs (and CIDR ranges), refreshed by push from the regional cache. The lookup is a hash or prefix-tree (for IP ranges) lookup, taking microseconds.
- Entries have a TTL (e.g., block for 1 hour), because IPs get reused and bad actors move on.
- The action depends on confidence: low → rate-limit, medium → challenge, high → block.
- Allowlists (known partners, internal IPs) always win.
Detection Path (finding bad actors)
- Streaming features per IP: requests/minute, failed logins/minute, number of distinct accounts tried, error rates, user-agent diversity, geo mismatch. These are computed in a stream processor (Flink) with sliding windows.
- Rules: "over 50 failed logins across over 20 accounts in 5 minutes → block 1h".
- ML model scores IPs on combined features, and threat-intel feeds add known bad IPs.
- A verdict (
ip,action,expires_at,reason) is written to the global verdict store and pushed to all regions. Publish it on a global topic, so each region updates its Redis and gateways within seconds.
URL Checking with a Slow External API
- Normalize the URL (lowercase host, remove tracking parameters) and hash it.
- Look it up in the cache (malicious: longer TTL; clean: shorter TTL, since sites can get compromised later).
- On a miss: call
isMaliciousthrough a rate-limited client (token bucket for the API's quota), with request coalescing (many users sharing the same new URL → one call). - If the API is slow or down: circuit breaker + fallback, e.g., allow the post but mark the URL "unverified", and re-check asynchronously, removing or warning if it turns out bad. For high-risk surfaces (like DMs to many users), fail closed or hold the message.
- Re-check popular URLs periodically.
Deep Dive — An IP address is not a personDeep dive
The tempting model is "bad IP, block it". The reason it fails is that an IP is a shared, temporary and cheaply replaced resource.
Block the IP that misbehaved
Credential stuffing from 203.0.113.9, so add it to the block list.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
ABUSE["Abuse from one IP"] --> BLK["Block the IP"]
BLK --> NAT["It is a mobile carrier NAT - 40,000 real users behind it"]
NAT --> OUT["All 40,000 locked out of the app"]
BLK --> BOT["The bot rotates to a new IP in seconds"]
BOT --> USELESS["Attack continues from a fresh address"]The asymmetry is brutal: an attacker discards an IP for pennies while a carrier NAT or a corporate gateway carries tens of thousands of legitimate users behind one address. The block hurts only the people it was not aimed at.
Score the IP, with decay
Maintain a reputation score per IP from observed behaviour, let it decay over time, and act above a threshold.
Better — the score captures repeated behaviour instead of a single event, and decay lets a reassigned address recover. But the unit of judgement is still the IP, so a shared address accumulates a bad score from the few abusers behind it, and a rotating botnet never accumulates one at all. Whatever the threshold, one of those two is mishandled.
Judge the actor, and choose the action by what it costs to be wrong
Two changes. Identify with more than the IP: combine it with a device fingerprint, the account, and behavioural patterns — request timing, endpoint sequences, header shapes. A botnet rotating addresses keeps the same fingerprint and the same pattern; a NAT hides thousands of different ones.
Then pick the response from confidence and collateral damage:
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
EV["Signals - IP, device, account, behaviour"] --> CONF["Confidence"]
IPTYPE["What kind of address? - datacentre, carrier NAT, residential"] --> COLL["Collateral damage if wrong"]
CONF --> D{"Decision"}
COLL --> D
D -->|"high confidence, low collateral"| BLOCK["Block - datacentre range, ASN"]
D -->|"high confidence, high collateral"| CHAL["Challenge - CAPTCHA on a carrier NAT"]
D -->|"medium"| RL["Rate-limit"]
D -->|"low"| ALLOW["Allow, keep watching"]A challenge is the option the binary design lacks, and it does most of the work: real users solve it and continue, bots do not. Behind a shared address that is the correct response almost every time, because it costs a legitimate user five seconds instead of their access.
Two operational notes:
- Challenge solve rate is the feedback signal. If a rule's challenges are being solved, it is firing on humans — tune it. This is the measurement that tells you the classifier is wrong before the support queue does.
- Fail open, with the last known lists. If the decision service is unavailable, gateways keep enforcing the lists they already have; new detections simply pause. Failing closed here means the login page stops working for everyone.
Trade-offs & AlternativesTrade-offs
| Decision | Choice | Why | Alternative |
|---|---|---|---|
| Decision location | In-memory lists at gateways | Microsecond checks | Remote call per request: latency, fragility |
| Verdicts | TTL-based, graded actions | IPs change hands, fewer false blocks | Permanent blocks: collateral damage |
| Propagation | Push via global topic | Seconds across regions | Periodic pull: slower |
| External URL API | Cache + coalescing + rate limit + breaker | Protects quota, survives outages | Call per share: quota exhausted |
Wrap-UpWrap-up
Make decisions locally at the gateway from in-memory sets of blocked or challenged IPs (with TTLs and allowlists), fed by a regional cache that receives pushed verdicts from a global store within seconds. Produce verdicts with a streaming detection pipeline (windowed per-IP features, rules, models and threat intel). For URLs, normalize and cache verdicts, call the slow external API through a rate-limited, coalescing client with a circuit breaker, and choose fail-open or fail-closed per surface.