Problem RestatementProblem
Apple asked: design a Retrieval-Augmented Generation (RAG) assistant that answers employee questions using an internal knowledge base containing text, images, PDFs and tables. Answers must be grounded in the documents (with citations), respect access permissions, and stay up to date as documents change.
RAG in one sentence: instead of trusting the model's memory, search the company's documents for relevant pieces and give them to the model as context to answer from.ArchitectureArchitecture
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
SRC["Docs, wikis, PDFs, images"] --> ING["Ingestion - parse, OCR, caption, chunk"]
ING --> EMB["Embeddings - text + image"]
EMB --> VDB[("Vector index + keyword index + ACLs")]
U["User question"] --> QS["Query service"]
QS --> VDB
QS --> RR["Re-ranker"]
RR --> LLM["LLM - answer with citations"]
LLM --> U
QS --> LOG[("Logs + feedback")]Deep Dive — Getting a PDF into the indexDeep dive
Retrieval quality is decided at ingestion. A perfect retriever over badly parsed content returns confident nonsense, and no amount of prompt work recovers it.
Extract raw text and split by character count
Pull the text out of each file and cut it into fixed-size chunks.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
PDF["PDF"] --> TXT["Raw text extraction"]
TXT --> SCAN["Scanned pages - no text layer, produce nothing"]
TXT --> TBL["Tables flattened - numbers lose their columns"]
TXT --> IMG["Diagrams and screenshots - invisible"]
TXT --> CUT["Split every 1,000 characters"]
CUT --> MID["Chunks start mid-sentence, headings separated from their content"]Several failure modes at once. A scanned policy document contributes nothing at all, a table becomes a row of unattributed numbers, and chunks split across boundaries so the heading that gives a passage its meaning ends up in a different chunk.
Layout-aware parsing and semantic chunking
Use a parser that understands page structure, and split on headings and paragraphs at roughly 300–800 tokens rather than on character counts.
This fixes the text, which is most of the corpus. It still ignores everything that is not text: scanned pages, the numbers in tables, and the architecture diagram that answers the question being asked.
Handle each format for what it is
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
SRC["Document"] --> T["Text / HTML -> clean text with headings"]
SRC --> P["PDF -> layout-aware parse; OCR for scanned pages"]
SRC --> TB["Tables -> structured rows + a short description of what the table holds"]
SRC --> IM["Images -> vision-model caption + OCR of embedded text + image embedding"]
T --> CH["Chunk 300-800 tokens, on semantic boundaries"]
P --> CH
TB --> CH
IM --> CH
CH --> META["Every chunk keeps its heading path, page number and source id"]
META --> IDX[("Index - text and image embeddings")]- OCR the scans. Without it, an entire class of internal documents is silently absent from the index, and nobody notices because the assistant answers from whatever else it found.
- Keep tables structured, and describe them. Rows preserve the relationship between a number and its column; a one-line description of what the table contains is what makes it retrievable by a question phrased in prose.
- Caption and embed images. A captioned, OCR'd, embedded diagram becomes answerable content instead of a gap, and the image embedding lets a text question find a picture.
- Carry heading path and page number on every chunk. That metadata is what produces a citation the employee can open and verify — and citations are what make the assistant trustworthy rather than merely fluent.
If you only have time for one sentence on this in an interview: most RAG quality problems are ingestion problems, and the fix is almost never a better prompt.
Retrieval
- Understand the query: rewrite it (expand acronyms, add conversation context).
- Hybrid search: vector similarity (meaning) + keyword/BM25 (exact terms like product codes), with results merged.
- Permission filter: only chunks whose ACL includes the user (their groups), applied inside the search, not after, to avoid leaks and empty results.
- Re-rank the top ~50 with a cross-encoder model and keep the best ~5–10. Include images or tables when they're relevant (e.g., a diagram caption matched).
Generation
- The prompt contains the question plus the retrieved chunks (with IDs), and the instructions: answer only from the context, cite sources, say "I don't know" if the context lacks the answer.
- For images, pass the image itself to a multimodal model when needed (e.g., "what does the architecture diagram show?"), or its caption.
- Return the answer with citations that link to the document and page.
Quality, Safety and ScaleScale
- Evaluation: a test set of real questions with expected sources. Measure retrieval recall@k (did we find the right chunk?), groundedness (is every claim supported?), and answer correctness. Track thumbs up/down in production.
- Hallucination control: citations required, low-confidence answers flagged, and grounding checks before display.
- Latency: embedding the query (~20 ms) + search (~50 ms) + re-rank (~100 ms) + LLM (~1–3 s, streamed). Cache frequent questions.
- Scale: millions of chunks → an ANN vector index (HNSW) sharded by tenant or department.
Wrap-UpWrap-up
Ingest every format properly: text, PDFs (with OCR and layout), tables as structured text with summaries, and images via captions, OCR and image embeddings. Chunk with metadata and ACLs, and keep the index fresh from change feeds. Answer with hybrid search filtered by the user's permissions, cross-encoder re-ranking, and an LLM prompted to answer only from the cited context (passing images to a multimodal model when needed), and measure retrieval recall and groundedness continuously.