What is answer relevance in RAG?

Answer relevance in RAG measures how directly and completely the generated response addresses the user's actual question. An answer can be supported by retrieved sources yet still be irrelevant, overly broad, or miss the user's requested detail.

Beginner-friendly example

The user asks: “How many paid leave days do employees receive each year?”

• Highly relevant: “Employees receive 20 paid leave days per year.”
• Partly relevant: “The company offers paid leave and requires manager approval for some requests.” This discusses the topic but does not answer *how many*.
• Irrelevant: “Employees can submit expense claims through the HR portal.”

Even if the partial answer is factually correct and faithful to a policy, it fails to provide the requested number.

How is answer relevance different from other RAG metrics?

• Answer relevance: Does the final response answer the user’s question?
• Faithfulness: Are the response's factual claims supported by retrieved evidence?
• Context relevance: Are retrieved passages useful for answering the question?
• Factual correctness: Is the answer accurate against trusted ground truth?

For example, a response might faithfully summarize a retrieved travel policy while failing to answer a question about the reimbursement ceiling. It is faithful but not sufficiently relevant.

How can you evaluate it?

1. Gather a representative set of user questions and generated responses.
2. Judge whether each answer addresses the specific intent, entities, scope, and requested detail in the question.
3. Use human ratings, rubric-based LLM judgments, or evaluated question–answer similarity methods. A common rubric scores each answer from 0 (irrelevant) to 3 (fully addresses the question).
4. Check separately for missing subquestions, unnecessary information, or answers that shift to a related topic.
5. Compare scores across query types; validate any automated judge against human annotations.

Illustrative scoring example:

• Question: “What is the return window for unopened items?”
• Answer A: “Unopened items can be returned within 30 days.” → 3/3 if the source confirms this.
• Answer B: “Returns are accepted under the store policy.” → 1/3 because it does not state the timeframe.
• Answer C: “Shipping takes three days.” → 0/3.

These scores are example rubric judgments, not universal thresholds.

How do you improve relevance?

• Retrieve passages that match the user's actual information need.
• Preserve constraints such as product type, region, timeframe, and requested format.
• Prompt the generator to answer the question directly before adding optional explanation.
• Handle multi-part questions by checking that every required part is addressed.
• When evidence is missing, say exactly what cannot be answered rather than filling the gap with unrelated context.

Interview takeaway

Answer relevance evaluates whether a RAG answer satisfies the user's question; faithfulness evaluates whether its claims are supported by retrieved evidence. High-quality RAG needs both.