Hallucinations

Why This Matters

Hallucinations are the number one reason AI pilots fail to reach production. A model that is right 95% of the time but confidently wrong the other 5% (with no signal telling you which is which) cannot be trusted for legal advice, medical information, or financial figures without safeguards. Every AI practitioner needs to know why hallucinations happen, how to reduce them, and how to design products that fail gracefully when they slip through.

Everyday Analogy

Imagine a brilliant student taking an exam who is never allowed to leave an answer blank. For questions they know, they write correct answers. For questions they do not know, they write something plausible-sounding in the same confident handwriting. Grading the paper, you cannot tell the difference by tone alone. LLMs are that student: trained to always produce the most plausible next token, never to say nothing.

What Exactly Is a Hallucination?

An LLM is a next-token predictor: it generates the most statistically plausible continuation of the text so far. "Plausible" and "true" usually overlap, but not always. When they diverge, the model produces fluent fiction with the exact same confidence as fact. Crucially, the model has no internal flag that distinguishes recalled knowledge from plausible invention.

The Main Types

1. Factual fabrication: inventing facts, dates, statistics, or events. "The Eiffel Tower was completed in 1901" (it was 1889).
2. Fake citations and sources: inventing papers, court cases, URLs, or book titles that look perfectly formatted but do not exist. The most dangerous type for professional work.
3. Faithfulness failure, where you provide a document and the model summarizes things it does not say. The source is right; the model drifts from it.
4. Instruction drift: you say "answer only from the provided context" and the model quietly blends in its own training knowledge anyway.
5. Capability hallucination, where the model claims it did something it cannot do: "I have sent the email" when it has no email tool.

A Famous Real Case

In 2023, a New York lawyer submitted a legal brief written with ChatGPT. It cited six court cases, complete with names, docket numbers, and quotes. All six were fabricated. The judge sanctioned the lawyer, and the case became the canonical warning: the model produced perfectly formatted legal citations because it learned the format of citations, not a database of real cases.

Why Do Models Hallucinate?

• Training rewards fluency, not truth. Pre-training optimizes "predict the next token", and human raters historically rewarded confident complete answers over honest "I don't know".
• No knowledge boundary. The model cannot reliably tell what it knows from what it can plausibly generate.
• Compression is lossy. Terabytes of text compressed into model weights lose detail, especially rare facts seen only a few times.
• Stale knowledge. Anything after the training cutoff simply is not in the weights, but the model will still answer questions about it.

Reducing Hallucinations: The Practitioner's Toolkit

Ordered roughly by effectiveness:

1. Grounding / RAG (AI-016). Retrieve real documents and instruct the model to answer only from them. Cuts fabrication dramatically for knowledge tasks: the single biggest lever.
2. Require citations to provided sources. Forcing every claim to link to a retrieved chunk turns unverifiable prose into checkable claims, and lets the UI render links a human can click.
3. Permission to say "I don't know." Adding "If the context does not contain the answer, say so" to the system prompt measurably reduces forced fabrication.
4. Lower temperature for factual tasks. Sampling randomness amplifies fabrication; factual Q&A should run near temperature 0.
5. Verification pass. A second model call that checks each claim of the first against the sources, catching faithfulness drift cheaply.
6. Structured outputs and tools. For math, dates, or lookups, have the model call a calculator or database instead of generating the answer from weights.
7. Evals with hallucination metrics (AI-025). Measure groundedness on a golden dataset before and after every prompt change: you cannot manage what you do not measure.

Designing for the Hallucinations That Remain

No technique reaches zero. Production AI products assume some hallucination and design for it:

• Show sources. Perplexity and Bing Copilot render citations inline so users can verify in one click.
• Calibrate the stakes. Brainstorming copy? Hallucination is a minor bug. Quoting drug dosages? It is a lawsuit. Match autonomy to risk (AI-053 covers human-in-the-loop patterns).
• Human review gates for high-stakes output. Legal, medical, and financial content gets a human approval step.
• Honest UI copy. "AI-generated — verify important details" sets user expectations and legally matters.

Real-World Showcase

• Perplexity built its entire product identity on grounded answers with inline citations, a direct product response to hallucination distrust.
• GitHub Copilot frames output as suggestions inside an editor where a compiler and tests catch fabricated APIs: the environment absorbs the errors.
• Air Canada's chatbot hallucinated a bereavement refund policy in 2024; a tribunal held the airline liable for its bot's invented policy. Hallucinations are now legal precedent.

Try It Yourself

Run this three-step experiment with any chatbot:

1. Induce: Ask about something obscure or fictional: "Summarize the 1987 novel 'The Glass Cartographer' by Miriam Voss." (It does not exist.) Watch whether the model invents a plot or admits ignorance.
2. Ground: Paste a short article and ask the same model to answer questions only from it, saying "not in the text" otherwise. Ask one question the article does not answer. Compare behavior.
3. Verify: Ask for a fact with three sources, then actually check the sources. Count how many are real.

You have just reproduced the core hallucination eval loop that AI teams run at scale.

Common Mistakes

• Believing a low-hallucination demo means a low-hallucination product: rare failures appear at scale.
• Trusting confidence: fluent, detailed, well-formatted output is not evidence of truth.
• Treating hallucination as a bug the next model version will fix: it is inherent to next-token prediction; manage it with grounding and design.
• Skipping citation checks because the citations "look real": fake ones always do.

Key Takeaways

• Hallucinations are confident fabrications caused by models optimizing plausibility, not truth.
• The model gives no signal separating recall from invention: you must add that via grounding and verification.
• RAG plus required citations plus permission to say "I don't know" removes most of the problem.
• Design products that fail safely: show sources, gate high-stakes outputs, set honest expectations.
• Measure hallucination with evals before and after every change; never rely on vibes.