How should a RAG system respond when relevant information is not found?

When a RAG system cannot find reliable evidence, it should acknowledge the gap rather than invent an answer. This is called *abstention* or an *insufficient-evidence response*.

Recommended behavior

1. Check retrieval quality: Confirm that the search used appropriate keywords, metadata filters, permissions, and—when useful—hybrid retrieval or query reformulation.
2. Assess evidence, not just similarity: A high vector similarity score does not prove that a passage answers the question. Evaluate whether the retrieved text supports the requested claim.
3. Respond transparently: Say that the available sources do not establish the answer. Avoid presenting model memory as verified organizational knowledge.
4. Offer a useful next step: Suggest a more specific question, another authorized knowledge source, or human review when appropriate.
5. Answer supported parts only: For multi-part questions, provide the parts backed by evidence and clearly identify the unanswered parts.
6. Preserve citations: Cite only actual retrieved passages. Do not fabricate source titles, quotations, URLs, or page numbers.

Beginner-friendly example

A user asks: “What is our company's 2027 work-from-home allowance?”

The knowledge base contains the 2026 policy but no approved 2027 allowance.

Poor response: “The 2027 allowance is ₹25,000.” This number is invented.

Better response: “I couldn't find an approved 2027 work-from-home allowance in the available policy documents. The latest document I found is the 2026 policy. Please check with HR for the 2027 amount.”

The system should mention the 2026 policy only if it was actually retrieved and the user has access to it.

How can the system decide to abstain?

• Use relevance or answerability checks calibrated on labeled questions.
• Detect when no retrieved passage supports a required fact.
• Treat contradictory, stale, or inaccessible documents as insufficient evidence until resolved.
• Avoid relying on one universal similarity-score cutoff across retrievers.

Common trade-off

Abstaining too often frustrates users even when the answer is available. Abstaining too rarely increases unsupported claims. Evaluate both:

• Answer coverage: How often does the system give a supported answer?
• Unsupported-answer rate: How often does it answer without adequate evidence?
• Correct abstention rate: How often does it decline when evidence is genuinely missing?

Interview takeaway

A trustworthy RAG system distinguishes “not found in the available evidence” from “false.” It should abstain on unsupported claims, cite what is known, and guide the user toward a reliable next step.