What happens when the correct document is retrieved but the LLM still gives the wrong answer?

If the correct document is retrieved but the LLM still gives the wrong answer, retrieval has probably done its job.

The failure is now in context construction or generation.

1. Check whether the exact answer is actually inside the retrieved chunk

A document can be correct at the document level but the retrieved chunk may still be incomplete.

Example:

The retrieved document is the refund policy, but the chunk only contains:

"Enterprise customers have a different refund policy."

The actual rule:

"Enterprise customers have 45 days."

may be in the next chunk.

So first confirm that the exact evidence needed to answer the question reached the model.

2. Check for too much irrelevant context

The right chunk may be present, but surrounded by many distracting chunks.

Example:

Relevant chunk: "Enterprise customers have 45 days."

Other chunks: "Standard customers have 30 days." "Promotional purchases have 14 days." "International returns follow separate rules."

If the prompt sends all of them without clear structure, the model may choose the wrong policy.

Better retrieval precision and reranking can help.

3. Check for conflicting or outdated documents

Sometimes retrieval returns multiple documents that disagree.

For example:

Old policy: "Refund period is 30 days."

New policy: "Refund period is 45 days."

If both are provided without metadata such as effective date or version, the model may use the wrong one.

This is not necessarily a retrieval miss. It may be a context-selection or document-versioning problem.

4. Check the prompt

The model should be clearly instructed to answer from the retrieved evidence.

Useful instructions include:

• Use only the supplied context when the answer depends on company-specific information.
• Prefer newer or authoritative sources when documents conflict.
• Say that the answer is not available when the context does not support it.
• Cite the evidence used.

A vague prompt gives the model more freedom to rely on prior knowledge or guess.

5. Check whether the model is reasoning correctly

Sometimes the evidence is present and clear, but the model still makes a reasoning mistake.

This can happen with:

• Multi-step questions
• Numerical comparisons
• Conditions and exceptions
• Negation
• Information spread across multiple chunks

For these cases, better context organization, a stronger model, structured extraction, or breaking the task into steps may help.

6. Run a controlled test

Take only the correct evidence and send it to the LLM with the original question.

If the model now answers correctly:

The larger context or prompt was the problem.

If the model still answers incorrectly:

The issue is more likely the model's reasoning, instruction-following, or the way the evidence is expressed.

This is one of the quickest ways to isolate the failure.

Interview takeaway

When the correct document is retrieved but the answer is still wrong, do not immediately blame retrieval. Verify that the exact answer-bearing chunk reached the model, remove distracting or conflicting context, inspect prompt instructions, and test the model with only the correct evidence. If it still fails, the issue is in generation or reasoning rather than retrieval.