How do you reduce hallucination in a RAG system?

RAG reduces hallucination by giving the LLM external evidence, but retrieval alone does not guarantee a grounded answer.

The goal is to make it easier for the model to answer from evidence and harder for it to invent unsupported facts.

1. Improve retrieval quality

If the wrong chunks are retrieved, the model starts with bad evidence.

Focus on:

• Better chunking
• Better embeddings
• Hybrid retrieval
• Metadata filtering
• Appropriate Top-K
• Reranking

The first defense against hallucination is getting the right context.

2. Remove irrelevant context

More context is not always better.

If the prompt contains many weakly related chunks, the model may mix them together or choose the wrong fact.

A better pattern is:

Retrieve broadly → rerank → send only the strongest evidence

This improves signal-to-noise ratio.

3. Tell the model to stay grounded

Prompt instructions should make the expected behavior explicit.

For example:

• Answer using the supplied context.
• Do not invent missing facts.
• If the context does not contain the answer, say so.
• Prefer authoritative or newer sources when documents conflict.
• Cite the supporting source.

This does not eliminate hallucination, but it creates a much stronger constraint.

4. Allow the model to say "I don't know"

A common cause of hallucination is forcing the model to always produce an answer.

If the retrieved evidence is weak or missing, the safest behavior may be:

"The available documents do not contain enough information to answer this."

That is often better than a confident guess.

5. Handle conflicting documents

Suppose retrieval returns:

Old policy: "Refund period is 30 days."

New policy: "Refund period is 45 days."

If both are sent without metadata, the model may choose either one.

Store and use metadata such as:

• Effective date
• Document version
• Source authority
• Department
• Status

Then filter or instruct the model which source should take precedence.

6. Make citations traceable

Ask the system to attach citations to important claims.

Then verify that each citation actually supports the claim.

A citation is useful only if it points to real evidence.

This also makes hallucinations easier to detect during evaluation.

7. Use structured outputs where possible

For tasks involving exact fields, dates, IDs, prices, or policies, structured extraction can reduce free-form guessing.

For example:

Refund period: 45 days Source: Enterprise Policy v3 Effective date: 2026-01-01

The LLM can then generate prose from validated fields.

8. Evaluate faithfulness separately

An answer can sound correct while containing unsupported details.

Evaluate whether each factual claim is supported by the retrieved context.

This is called grounding or faithfulness evaluation.

Important point

RAG does not "remove hallucination."

It reduces the probability by giving the model relevant evidence and constraining how that evidence is used.

A strong RAG system therefore combines:

Good retrieval + clean context + grounding instructions + abstention + citations + evaluation

Interview takeaway

To reduce hallucination in RAG, improve retrieval quality, rerank and remove noisy context, instruct the LLM to answer only from evidence, allow it to abstain when evidence is missing, resolve conflicting sources with metadata, require traceable citations, and evaluate answer faithfulness separately from retrieval accuracy.