Problem RestatementProblem
Bloomberg asked: design a system that ingests high-volume text streams (news articles, social posts, press releases), detects mentions of entities (companies, tickers, people), counts mentions over time windows, lets users search recent mentions, and alerts subscribers when mention volume for an entity spikes (e.g., "AAPL mentions are 8x normal in the last 10 minutes").
RequirementsRequirements
- Ingest thousands of documents per second from many sources.
- Extract entity mentions with good accuracy ("Apple" the company vs the fruit).
- Real-time counts per entity per minute, plus sentiment optionally.
- Search: "show recent mentions of TSLA with 'recall'".
- Alerts on spikes, with low false alarms. Latency from publish to alert under ~1 minute.
ArchitectureArchitecture
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
SRC["News wires, social feeds"] --> ING["Ingestion + dedupe"]
ING --> K[("Kafka - documents")]
K --> NER["Entity extraction + disambiguation"]
NER --> KM[("Kafka - mentions: entity, doc, time, sentiment")]
KM --> AGG["Windowed counts per entity"]
AGG --> TS[("Time-series store")]
AGG --> AD["Spike detector vs baseline"]
AD --> AL["Alert service - subscriptions"]
AL --> U["Users / terminals"]
KM --> IDX[("Search index - recent mentions")]
U --> API["Search / chart API"]
API --> IDX
API --> TSDeep Dive — Counting mentions of a companyDeep dive
"How many times was Apple mentioned in the last hour?" sounds like counting a word. Almost everything that makes the number wrong happens before the counting.
Count keyword occurrences
Search each document for the entity's name and increment.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
DOC["Incoming text"] --> KW["Find the string 'Apple'"]
KW --> C1["'apple pie recipe' counted"]
KW --> C2["'Apple Records' counted"]
KW --> MISS["'AAPL shares fell' not counted"]
KW --> MISS2["'the iPhone maker said' not counted"]
C1 --> NOISE["Counts measure the word, not the company"]The name is neither sufficient nor necessary: it matches things that are not the company, and misses tickers, aliases and descriptions that clearly are. For a financial product the resulting series is not usable.
Named-entity recognition
Run NER to find spans that are organisations, rather than matching raw strings.
A real improvement — "apple pie" is no longer an organisation. But NER produces surface forms, not identities: "Apple", "Apple Inc.", "AAPL" and "the Cupertino company" are four different strings and the counts split across them. Ambiguity is untouched too, since Apple Records is also an organisation.
Dedupe first, then link entities to identifiers
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
IN["Articles, posts, releases"] --> DD["Dedupe: content hash, then SimHash / MinHash"]
DD --> NER["NER - candidate entity spans"]
NER --> LINK["Entity linking -> a canonical id"]
ALIAS[("Alias and ticker dictionary - $AAPL, Apple Inc.")] --> LINK
CTX["Context words - iPhone, shares, earnings"] --> LINK
LINK --> REC["{entity_id, doc_id, ts, snippet, sentiment}"]
REC --> AGG["Count per entity per minute - event time, watermarks"]- Deduplicate before counting. Wire stories are republished by dozens of outlets, so without this a single press release looks like a mention spike. Exact hashes catch identical copies; SimHash or MinHash catches lightly edited ones.
- Link to an id, not a string. Every surface form resolves to one canonical entity, so counts are per company rather than per spelling — and tickers like
$AAPLbecome first-class evidence. - Use context for ambiguity. Nearby words decide between Apple the technology company and Apple Records; a model handles the genuinely unclear cases, and low-confidence links should be dropped rather than guessed.
- Aggregate on event time with watermarks, so an article published at 14:58 and crawled at 15:05 lands in the right minute.
This matters because the output drives alerts: a spike that is really syndication, or a spike caused by an unrelated company sharing a name, becomes a trading signal someone acts on. The deduplication and linking stages are not preprocessing — they are what the alert's correctness rests on.
Scale and LatencyScale
- Partition mention streams by entity ID for aggregation. Hot entities (big tech on earnings day) may need pre-aggregation.
- NER models are the heaviest step, so scale them horizontally (GPU workers for large models, fast dictionary matching first).
- Target: document → alert in under 60 seconds.
Wrap-UpWrap-up
Ingest and deduplicate text streams into Kafka, extract and link entity mentions with NER plus alias dictionaries and context, and publish mention events. Aggregate mentions per entity per minute in a stream processor (stored for charts), detect spikes against seasonal baselines with both relative and absolute thresholds, and deliver deduplicated alerts to subscribers, while a time-partitioned search index serves recent-mention search.