Hallucination

In plain terms

A language model is built to produce plausible text, and when plausible and true diverge it has no internal mechanism preferring true. Hallucination is the result: fluent, confident, wrong. It cannot be prompted away, and 2025 research argues it cannot be fully trained away either while training and evaluation reward confident guessing over honest uncertainty.

Hallucination is when a model states something false with complete confidence. The name is slightly unfortunate, because it implies malfunction. The model is doing exactly what it was built to do, which is produce plausible text. When plausible and true diverge, it has no mechanism preferring true, and the result is the characteristic failure of language models: fluent, confident, wrong.

Mechanism

A language model is trained to continue text the way its training distribution would. That objective covers an astonishing range of work, and it contains no term for truth. Where the training distribution is dense and factual, plausible continuation and correct continuation coincide; where it is sparse, contested, or wrong, the model produces what would sound right, with the same fluency either way. The confidence is not a bug bolted on: the model's register of assurance is itself learned text style, uncorrelated with the reliability of the claim underneath.

The research literature distinguishes fabrication with respect to the world (wrong facts) from fabrication with respect to a source (a summary inventing content), surveyed as intrinsic and extrinsic hallucination in Ji et al. (2022). A related, measurable strand is imitative falsehood: TruthfulQA (Lin et al., 2021) showed models reproducing common human misconceptions precisely because they are common in the training data, with larger models sometimes more prone, having learned the misconceptions better.

Why it persists

For years the working assumption was that scale and better training would drive hallucination out. The 2025 analysis Why Language Models Hallucinate (Kalai et al., OpenAI) gave the persistence a structural explanation: some error rate is statistically unavoidable for rare facts under the pretraining objective, and, more correctably, the field's own evaluations reward guessing. Binary pass-fail grading gives a model that guesses when uncertain a better score than a model that says it does not know, so training and leaderboards select for confident answering. The paper's proposed remedy is institutional rather than architectural: score abstention honestly so that calibrated uncertainty stops losing to confident fabrication. The diagnosis connects hallucination to Goodhart's law: accuracy metrics became targets, and honest uncertainty is what they stopped measuring.

Retrieval and tool grounding shrink the space where the model must rely on parametric memory, and abstention training shrinks the space where it answers anyway. Neither closes it. A grounded system can still misread its sources, and deployed frontier assistants in 2026 still carry documented nonzero hallucination rates, lower than their predecessors, not zero.

Why it matters for verification

Hallucination is the everyday proof of a distinction this site leans on: confidence is not observation. A system's statement about the world, or about its own work, is a report whose assurance carries no evidential weight, which is the ground-level case for checking claims against something real rather than asking the model to be careful. It cannot be prompted away, only checked against something real. In our vocabulary, that check is grounding and the trace-backed evidence the evidence gap demands; the failure of skipping it is the persuasive subtype of false convergence, where fluent confidence reads as verified correctness. The deeper version of the same structure, a system whose reports track approval rather than its own knowledge, is eliciting latent knowledge.

Limitations of the term

The word is stretched daily to cover every model error, which blunts it. A wrong answer from a bad source is retrieval failure; a wrong action from a misread instruction is a specification problem; a deliberate-looking falsehood under conflicting objectives belongs to alignment faking territory. The useful core is narrower: fabricated content delivered with unearned confidence. It also names a behavior, not a moral state; the model is not lying, because lying requires a represented truth being withheld, which is exactly what the training objective never installed.

FAQ

Can hallucination be fixed with better prompting?

No. Prompts can reduce the rate, mostly by licensing abstention or forcing source citation, but the underlying objective still produces plausible text when knowledge runs out. The durable mitigations work outside the model: retrieval that narrows what must be remembered, abstention that is actually rewarded, and verification of claims against sources or execution. It cannot be prompted away, only checked against something real.

Will scaling eliminate hallucination?

The 2025 OpenAI analysis argues no, not under current incentives: some error on rare facts is statistically expected from the pretraining objective, and standard evaluations actively reward confident guessing over calibrated uncertainty. Rates fall with scale and with training against the behavior, and deployed models improved measurably through 2026, but the trajectory points to managed-low, not zero, until scoring rewards honesty about uncertainty.

Sources

Ji et al., Survey of Hallucination in Natural Language Generation, arXiv, 2022.
Lin et al., TruthfulQA: Measuring How Models Mimic Human Falsehoods, arXiv, 2021.
Kalai et al., Why Language Models Hallucinate, arXiv, 2025.