Hallucination Reduction Techniques

Hallucination - the model generating confident, plausible-sounding but factually incorrect content - is the most trust-damaging failure mode in LLM applications. It cannot be eliminated, but it can be reduced to acceptable rates with the right architectural choices. This page covers the practical techniques that work in production.

Why Hallucination Happens

LLMs are trained to predict the next token, not to "know" facts. They learn statistical patterns, not a verifiable knowledge base. When a question touches rare or ambiguous territory, the model extrapolates - generating a plausible-seeming answer that may not be grounded in actual training data. The model also does not know what it does not know, so it cannot reliably abstain.

Technique 1: Grounding with Retrieved Context

The single most effective hallucination reduction technique is RAG - giving the model the correct information and instructing it to answer only from that source.

System: Answer the user's question using ONLY the provided context.
If the answer is not explicitly stated in the context, say:
"I don't have information about that in the provided documents."
Do not guess, infer, or use knowledge from outside the provided context.

Context:
[Retrieved documents inserted here]

User: [question]

The "refuse if not in context" instruction is critical - without it, models will supplement missing context with hallucinated details.

Technique 2: Citation Requirements

Requiring the model to cite the source of every factual claim changes its behaviour. When the model must point to specific text, it is less likely to fabricate - fabrications have no source to cite. Implement citations as structured output:

Technique 3: Self-Consistency Voting

Run the same factual query multiple times (5–10) with temperature > 0 and check if the answers agree. Consistent answers across samples indicate higher confidence; divergent answers are a signal to flag for human review or return a lower-confidence response. Expensive - use only for high-stakes queries.

Technique 4: Chain of Verification

A multi-step process: (1) generate initial answer, (2) have the model generate verification questions about its own claims, (3) answer each question independently from the source material, (4) revise the original answer if verification reveals errors.

Technique 5: Lower Temperature

Higher temperature increases token diversity - and increases the probability of selecting an incorrect but syntactically plausible token. For factual tasks, set temperature to 0 or near-0. The model still hallucinates at temperature 0 (deterministic wrong answers) but less frequently than at higher temperatures.

Technique 6: Choose Models That Abstain

Models differ substantially in their tendency to abstain vs. confabulate. Claude models tend to say "I'm not certain about that" more readily than some alternatives. For tasks where abstaining is better than hallucinating (medical, legal, financial), test models explicitly on your domain and pick those with higher calibration.

Measuring Hallucination Rate

You cannot reduce what you cannot measure. Build a hallucination evaluation set:

• Select 100+ factual questions from your domain with known correct answers
• Run the system and score each response as: Correct / Incorrect / Abstained
• Track hallucination rate (incorrect / total non-abstained) across releases
• Use an LLM judge or human annotation for scoring - pattern matching is insufficient

When to Escalate to Human Review

• The model cites a source but the quote does not appear verbatim in the source
• Self-consistency voting returns high variance across samples
• The query touches a domain with known low model coverage
• The stakes are high enough that a wrong answer has significant consequences