Retrieval-Augmented Generation (RAG)

Why This Matters

RAG is the most deployed LLM architecture in industry, full stop. Every "chat with your docs" product, every enterprise assistant that knows company policy, every support bot with current product info — RAG. It solves the three fatal flaws of a raw LLM at once: knowledge cutoff (AI-007), no access to private data, and hallucination (AI-060). This lesson assembles everything from Level 2 — tokens, prompting, embeddings, vector search — into the architecture you will actually build.

Everyday Analogy

Two exams. Closed-book: the student answers from memory — fluent, but shaky on details, and prone to confident inventions when memory fails. That is a raw LLM. Open-book: the student first looks up the relevant pages, then answers with the book open, citing page numbers. Same student, dramatically more reliable. RAG is the open-book exam, and the citations mean you can check their work.

The Pipeline, End to End

Offline (once per document): chunk your knowledge base → embed each chunk → index in a vector database. (This is exactly AI-015's pipeline.)

Online (per question) — user asks: "What is our parental leave policy for part-time employees?"

1. Embed the question with the same model used for chunks (AI-014).
2. Retrieve the top-k nearest chunks (say k=5) from the vector DB: the HR policy passages most similar in meaning (AI-015).
3. Assemble the prompt (AI-010's skeleton): system instructions + the 5 retrieved passages + the question + guards: "Answer only from the provided context. Cite the source of each claim. If the context doesn't contain the answer, say so."
4. Generate. The LLM (AI-009) writes the answer from the supplied passages, within its context window (AI-061).
5. Show sources. The UI renders the answer with clickable citations to the original documents.

Total cost: one embedding call + one vector lookup + one LLM call: a few cents, a few seconds.

RAG vs Fine-tuning vs Long Context

Three ways to give a model your knowledge — teams confuse them constantly:

The standard guidance: RAG for knowledge, fine-tuning for behavior. They combine well — a fine-tuned tone with RAG-supplied facts.

Where RAG Breaks (and What to Pull)

RAG quality problems come from specific, diagnosable stages:

• Retrieval misses. The right passage exists but wasn't in the top-k. Levers: better chunking (respect paragraph/section boundaries), hybrid search for exact terms (AI-015), a reranker (a second model reordering the top-50 into a sharper top-5), query rewriting ("it" → the actual topic). See AI-077 for a practitioner-depth walkthrough of chunking strategies, hybrid search, and reranking.
• Bad chunks. A passage cut mid-sentence carries broken meaning. Lever: chunk on structure, overlap edges, keep titles attached.
• Ignored context. The model answers from its weights instead of your passages. Levers: stricter prompt guards, citation requirements, "lost in the middle" placement — put the best chunk first or last (AI-061).
• Stale index. The doc changed; vectors didn't. Lever: re-index on document change, not on a quarterly calendar (AI-006's freshness).

Measure each stage separately — retrieval recall and answer groundedness are different metrics (AI-025) — or you will fix the wrong stage.

Real-World Showcase

• Perplexity built a search company on web-scale RAG: retrieve live results, generate with citations (AI-060's showcase, now with its architecture named).
• Every enterprise copilot, from Microsoft 365 Copilot to Glean and Notion AI Q&A, is RAG over your emails, docs, and wikis with permission-aware retrieval.
• Klarna's assistant (AI-009's showcase) answers policy questions via RAG. The LLM never memorized refund rules; it reads them per query, so a policy edit updates the bot in minutes.

Try It Yourself

1. Run manual RAG: paste a policy document into a chatbot with "Answer only from this document, cite the section, say 'not found' if absent" — then ask one answerable and one unanswerable question. You just executed steps 3–5 by hand, and watched the guards work (AI-060's grounding experiment, now recognizable as RAG).
2. Design retrieval failure: ask about a topic the document mentions with a synonym only ("annual leave" when the doc says "vacation days"). If quality drops, you've seen a retrieval-side miss, and you know the lever (embeddings handle it; keyword search wouldn't).
3. Sketch your own RAG: pick a corpus you know (team wiki, course notes). Write down: chunk size, metadata fields, k, and the guard lines of your prompt. That half-page is a genuine v1 design.

Common Mistakes

• Blaming the LLM for retrieval failures. Most bad RAG answers never had the right passage in the prompt; debug retrieval first.
• Skipping evaluation — without a golden set of question→expected-passage pairs, tuning is guesswork (AI-022, AI-025).
• One-size chunks — legal contracts and Slack threads need different chunking; inspect what your chunks actually look like.
• Believing long context killed RAG — million-token windows still lose the middle and bill per token; retrieval remains cheaper, faster, and more accurate (AI-061).
• No citations — grounding without visible sources wastes RAG's biggest trust dividend (AI-060).

Key Takeaways

• RAG = open-book exam: retrieve → assemble prompt → generate grounded, cited answers.
• It fixes cutoff, private data, and hallucination in one architecture, hence its dominance.
• RAG for knowledge, fine-tuning for behavior; they compose.
• Debug by stage: retrieval recall vs generation groundedness are separate dials.
• Everything in Level 2 (tokens, prompts, embeddings, vector DBs, context) just became one system.