TL;DR: A hallucination is a model stating something false with complete fluency and confidence - an invented citation, a plausible-but-wrong API method, a fabricated legal case. It happens because models generate likely-sounding text rather than looking up verified facts. Grounding, abstention, and verification reduce it substantially; nothing eliminates it.
How it works
A language model is a machine for producing plausible continuations. When you ask for a paper supporting some claim, the model does not consult a library - it generates text shaped like the citations in its training data: a real-sounding author, a credible journal, a tidy year. Often the pattern-matching lands on truth, because truth was heavily represented in training. When the model lacks the fact, the same machinery produces the same confident prose anyway. Nothing inside the generation loop distinguishes "I know this" from "this sounds right", which is why the classic real-world failure - lawyers sanctioned for filing briefs citing cases that never existed - looked so convincing on the page.
Several forces feed the behavior. Training data itself contains errors. Compression is lossy: billions of documents squeezed into fixed weights means details blur - the model may correctly retain that a function exists while misremembering its arguments. Aggressive sampling settings like high temperature increase the odds of low-probability continuations. And post-training historically rewarded confident, complete-looking answers, teaching models that guessing beats admitting ignorance - newer training explicitly pushes back by rewarding calibrated "I do not know" responses, one reason abstention rates have improved.
Mitigation is an engineering discipline, not a checkbox. Grounding with RAG changes the task from "recall this" to "read this and answer from it", which models do far more reliably - and citations let humans verify. Prompting the model to say when the context does not contain the answer converts silent fabrication into visible abstention. Guardrails can check claims against source documents before output ships, and evals measure your system's factuality rate so regressions get caught instead of discovered by customers.
The practical stance: treat every unverified factual claim from a model as a draft. Design the product so verification is cheap - link sources, constrain outputs, keep a human in the loop where errors are expensive.
Where it sits in the AI stack
Hallucination is a failure mode of generation itself - it emerges between the model and the user, and the layers around it exist to catch it:
Key tools and implementations
-
RAG grounding
Supplying source documents at query time so answers come from evidence, not recall.
-
Citation enforcement
Requiring the model to attribute each claim to a passage a human can check.
-
Factuality benchmarks
Public tests that measure how often models fabricate versus abstain on hard questions.
-
Groundedness checkers
Eval-layer judges that verify an answer is actually supported by its source context.
Related entries
- RAG (retrieval augmented generation) Fetching relevant documents at query time so a language model can answer from your data instead of memory alone.
- Temperature A setting that controls how random a model's word choices are, from deterministic to creative.
- Evals (AI evaluations) Structured tests that score an AI system's outputs so teams can measure quality and catch regressions.
- Guardrails Programmatic checks on an AI system's inputs, outputs, and actions that block or correct behavior outside defined limits.