Retrieval-Augmented Generation

Retrieval-augmented generation connects a generative model to a searchable knowledge source. Instead of relying only on model weights, the system fetches relevant documents and places them into the prompt or context.

How can conflicting retrieved documents be handled?

Conflicting retrieved documents in RAG are sources that give incompatible answers to the same question. A robust RAG system should detect the disagreement, investigate which sources apply, and avoid silently choosing the most convenient passage. Beginner friendly example A user asks: “How many days do customers have to request a refund?” Retrieval returns: Policy A (2024): Refund requests are allowed within 14 days . Policy B (2026): Refund requests are allowed within 30 days . If both refer to the same product and jurisdiction, the newer approved policy may supersede the older one. But a newer timestamp alone does not prove authority: Policy A might apply to a different country, or Policy B might be an unapproved draft. Recommended workflow 1. Detect the conflict: Compare claims about the same entity, policy, timeframe, and scope. Pay special attention to quantities, dates, exceptions, and negations. 2. Check applicability: Determine whether the documents refer to different regions, products, customer types, or effective dates. Apparent conflicts may be legitimate exceptions. 3. Verify provenance: Inspect source owner, approval status, document version, publication date, and effective date. 4. Apply explicit authority rules: Prefer the organization's designated source of truth or approved supersession chain—not merely the highest similarity score or newest upload. 5. Preserve both sources when unresolved: Do not merge contradictory claims into a single fabricated compromise. 6. Answer with qualifications: Explain which rule applies and cite the supporting passages. If authority cannot be established, state the disagreement and ask for clarification or escalation. What should the response look like? Resolved case: “The approved refund policy effective January 2026 allows requests within 30 days [Policy B]. The earlier 2024 policy stated 14 days [Policy A] and has been superseded.” Only say “superseded” if the source metadata or an authoritative rule confirms it. Unresolved case: “I found two applicable documents that disagree: one specifies 14 days and the other 30 days. I cannot determine which is authoritative from the available records. Please confirm with the policy owner.” Implementation tips Store document id, version, effective date, status, owner, and jurisdiction in chunk metadata. Use metadata filtering and claim level contradiction checks before generation. Maintain source to claim mappings so reviewers can inspect disagreements. For high stakes answers, route unresolved conflicts to a human reviewer. Interview takeaway Conflict resolution in RAG is about authority and applicability, not majority voting. Detect contradictory claims, compare source scope and version, apply trusted precedence rules, and disclose unresolved conflicts instead of guessing.

How do you evaluate a RAG system?

Evaluating a RAG system means checking whether it retrieves useful evidence, generates a supported answer, satisfies the user's question, and works reliably in production. A single accuracy score is not enough because retrieval and generation can fail independently. 1. Prepare representative test cases Collect realistic user questions and record: The expected answer or key facts. Relevant source documents or passages, when known. Important conditions: user permissions, language, document version, and question difficulty. Questions that should produce an “insufficient evidence” response. Include simple fact lookups, multi document questions, ambiguous requests, conflicting sources, and unanswerable queries. 2. Evaluate retrieval Ask: Did the retriever find the evidence needed to answer? Recall@K: How much relevant evidence appears in the top K results? Precision@K: What fraction of the top K results is relevant? MRR / nDCG: How well are relevant results ranked? Access correctness: Did retrieval respect tenant and document permissions? For example, if 2 of 3 relevant passages appear in the top 5, Recall@5 = 2/3 ≈ 0.67 . 3. Evaluate the generated answer Ask: Did the model use the evidence correctly and answer the question? Faithfulness / groundedness: Are factual claims supported by retrieved passages? Answer relevance: Does the response address the user's actual request? Correctness: Does it match trusted reference facts when available? Citation quality: Do cited passages genuinely support their attached claims? Abstention quality: Does the system decline to invent answers when evidence is missing? A relevant sounding answer may still be unfaithful; a faithful answer may still fail to address the question. 4. Measure the complete system Track: End to end task success: Can the user obtain a correct, supported answer? Latency: Retrieval, reranking, generation, and total response time. Cost: Tokens, model calls, and retrieval infrastructure. Reliability: Timeouts, empty retrievals, failures, and stale index incidents. Safety: Prompt injection resistance, permission leakage, and handling of poisoned sources. 5. Diagnose failures by stage Example: A question asks for a refund deadline. Correct policy not retrieved → retrieval problem . Correct policy retrieved but answer invents a different deadline → generation/grounding problem . Correct deadline stated without answering the requested region → answer relevance or scope problem . This separation makes fixes targeted instead of blindly changing the prompt or embedding model. 6. Run repeatable experiments Keep a fixed evaluation dataset, compare pipeline versions, review failures manually, and monitor production feedback for changing document content and query patterns. Calibrate automated LLM judges against human labels; judge scores are estimates, not ground truth. Interview takeaway Evaluate RAG at three levels: retrieval quality, answer quality, and end to end operational performance. Use labeled questions, stage specific metrics, and error analysis to identify what actually needs improvement.

What is Hit Rate in RAG retrieval?

Hit Rate@K measures the proportion of evaluation queries for which at least one relevant document appears among the top K retrieved results. It answers: “For how many questions did retrieval find at least one useful passage?” Formula Hit Rate@K = (number of queries with ≥1 relevant result in top K) / (total evaluated queries) A query counts as a hit if at least one retrieved result is relevant; otherwise it is a miss . Each query contributes either 1 or 0, regardless of how many relevant documents it retrieves. Beginner friendly example Suppose a RAG retriever is tested on 5 questions with K = 3 : Q1: Relevant document found in top 3 → Hit Q2: No relevant document in top 3 → Miss Q3: Two relevant documents in top 3 → Hit Q4: Relevant document ranked third → Hit Q5: No relevant document in top 3 → Miss Hit Rate@3 = 3 / 5 = 0.60 = 60%. This means 60% of the test questions retrieved at least one relevant document within their first three results. Hit Rate versus Recall@K They measure different things: Hit Rate@K: Did the query retrieve any relevant passage? Recall@K: What fraction of all known relevant passages for that query were retrieved? Example: A query has 4 relevant documents , but the top 3 results contain only 1 . Query level Hit@3 = 1 because at least one relevant document was found. Recall@3 = 1/4 = 25% because three relevant documents were missed. A system can therefore have high Hit Rate but low Recall if it usually finds one relevant passage while missing other evidence needed for multi document answers. When is Hit Rate useful? Single fact questions: One relevant passage may be sufficient. Retrieval smoke tests: Quickly detect whether a retriever can find anything useful. Comparing K values: Check how often increasing K turns misses into hits. Monitoring retrieval regressions: Compare the same labeled query set across indexing or embedding changes. Limitations It does not measure whether the retrieved passage is ranked first or last within the top K. It does not measure how many relevant passages were missed. It does not prove the generated answer is correct or faithful. Relevance labels and the definition of a “useful” passage must be consistent across evaluations. For questions requiring several pieces of evidence, evaluate Recall@K and downstream answer quality alongside Hit Rate. Interview takeaway Hit Rate@K is a query level success metric: the percentage of questions with at least one relevant retrieval in the top K. It is simple and useful, but cannot measure evidence completeness or answer correctness.

How does nDCG reward highly relevant documents appearing earlier?

Normalized Discounted Cumulative Gain (nDCG) rewards retrieval systems that place more relevant documents near the top of their ranked results. It uses two ideas: highly relevant results contribute more gain, and results appearing later receive a position discount. Why does ranking position matter? In RAG, the generator often receives only the top few retrieved chunks. A highly relevant passage at rank 1 is usually more useful than the same passage buried at rank 20, where it may never enter the context window. Unlike binary metrics that label passages only “relevant” or “not relevant,” nDCG can use graded relevance , such as: 0: Not relevant 1: Somewhat relevant 2: Relevant 3: Highly relevant Formula For the top K results, a common definition is: DCG@K = Σᵢ₌₁ᴷ (2^relᵢ − 1) / log₂(i + 1) Where: relᵢ is the relevance grade of the document at rank i . 2^relᵢ − 1 rewards higher relevance grades disproportionately. log₂(i + 1) discounts documents that appear farther down. nDCG@K = DCG@K / IDCG@K IDCG@K is the highest possible DCG@K obtained by ordering the available relevance grades ideally. nDCG is typically between 0 and 1 , where 1 indicates ideal ranking. Queries with no relevant documents need an explicit evaluation convention. Beginner friendly numerical example Three retrieved documents have relevance grades 3, 2, 0 . Ranking A: [3, 2, 0] Rank 1 gain: (2³ − 1)/log₂(2) = 7 Rank 2 gain: (2² − 1)/log₂(3) ≈ 1.89 Rank 3 gain: 0 DCG@3 ≈ 8.89 ; this is the ideal ranking, so nDCG@3 = 1.00 . Ranking B: [0, 2, 3] Rank 1 gain: 0 Rank 2 gain: 1.89 Rank 3 gain: (2³ − 1)/log₂(4) = 3.50 DCG@3 ≈ 5.39 nDCG@3 ≈ 5.39/8.89 ≈ 0.61 . Both rankings retrieve the same documents, but Ranking A scores higher because the strongest evidence appears first. When is nDCG useful for RAG? Comparing embedding models, rerankers, or hybrid search configurations. Evaluating results where some passages directly answer the question and others provide only background. Measuring improvements in the order of retrieved evidence—not merely whether evidence appeared. Important limitations nDCG depends on consistent relevance labels and a clearly defined K. High nDCG does not guarantee that the generated answer is correct or faithful. It does not directly measure whether all evidence required for multi hop questions was retrieved. Interview takeaway nDCG rewards both relevance strength and ranking position. Placing highly relevant passages earlier produces a higher score even when the set of retrieved documents stays unchanged.

What is context precision?

Context precision in RAG measures how much of the retrieved context is relevant to answering a user's question, with particular attention to whether useful passages appear near the top of the retrieved list. It helps reveal when a retriever sends too much irrelevant text to the generator. Beginner friendly example A user asks: “How long is the warranty for Product A?” The retriever returns four passages: 1. Product A has a two year warranty . → Relevant 2. Product A is available in blue and black. → Irrelevant 3. Product A warranty covers manufacturing defects. → Relevant 4. The company opened a new office. → Irrelevant Simple context precision at K=4 = 2 relevant passages / 4 retrieved passages = 0.50 (50%). This is the basic Precision@K interpretation of context relevance. Some RAG evaluation frameworks use a rank aware context precision metric that gives more credit when relevant passages appear earlier, so the exact formula depends on the framework. Why does it matter? Irrelevant passages waste context window tokens and increase latency or cost. Distracting content can make the model miss the useful evidence. High context precision often makes grounded answers easier, but it does not guarantee correctness. Precision@K versus rank aware context precision Precision@K counts relevant items among the top K, regardless of their order. A common rank aware approach uses average precision over the retrieved list: Context precision = (Σ precision@i × relevanceᵢ) / (number of relevant items in the evaluated list) Here, relevanceᵢ is 1 for a relevant passage and 0 otherwise. Some frameworks apply additional definitions, reference answer checks, or conventions for zero relevant passages; always inspect the metric implementation. Example with relevance labels [1, 0, 1, 0] : Precision@1 = 1/1 = 1.00 Precision@3 = 2/3 ≈ 0.67 Rank aware score = (1.00 + 0.67) / 2 ≈ 0.83 Moving the two relevant passages to ranks 3 and 4 would lower the rank aware score even though ordinary Precision@4 stays at 0.50. How to improve context precision 1. Use a better query representation and appropriate metadata filters. 2. Combine keyword and vector retrieval when the task benefits from both. 3. Rerank candidate passages by their relevance to the specific question. 4. Remove near duplicate, stale, or clearly irrelevant chunks. 5. Evaluate on realistic labeled queries and compare quality against recall and latency. Important distinction Context precision evaluates retrieved evidence, not the final answer. Answer relevance asks whether the response addresses the user's question, while faithfulness asks whether the response's claims are supported by that evidence. Interview takeaway Context precision tells you whether retrieved passages are useful rather than noisy. Report the metric's exact definition—simple Precision@K or a rank aware variant—because implementations are not interchangeable.

What is context recall?

Context recall in RAG measures whether the retrieved passages contain the information needed to support a reference answer. It asks: “Did retrieval bring back enough evidence to answer the question completely?” Beginner friendly example Question: “What is the refund period, and who pays return shipping?” The trusted reference answer has two facts: Refunds are accepted within 30 days . Customers pay return shipping. The retrieved context contains only: “Refunds are accepted within 30 days.” The retriever found evidence for one of two required facts, so an illustrative fact coverage score is: Context recall = 1 / 2 = 0.50 (50%) The passage is relevant, but it is insufficient to produce a complete grounded answer. How is context recall calculated? One common reference based approach: 1. Break the reference answer into distinct factual statements. 2. Check whether each statement is supported by the retrieved context. 3. Divide the number of supported statements by the total number of statements. Context recall = supported reference statements / total reference statements Different evaluation frameworks may use LLM based attribution, document level relevance labels, or other scoring rules. The exact implementation and handling of unanswerable queries should be specified. Context recall versus Recall@K Recall@K: What fraction of all labeled relevant documents or chunks was retrieved in the top K? Context recall: Does the retrieved context contain the evidence required for the reference answer ? For example, five documents may be labeled relevant, but one retrieved passage could contain all the facts needed for the answer. Document Recall@K may be low while reference answer context recall is high. Context recall versus context precision Context recall: Did we retrieve enough supporting evidence? Context precision: How much of what we retrieved is useful , and how well is it ranked? Retrieving many passages can increase recall but also introduce irrelevant noise. Good RAG retrieval balances both. How can you improve context recall? Use query decomposition for multi part questions. Combine keyword and semantic search when their strengths differ. Tune Top K and reranking without assuming that more context is always better. Improve chunk boundaries so key facts are not split or omitted. Verify filters, permissions, indexing freshness, and coverage of knowledge sources. Test against labeled reference answers, especially multi hop and multi document cases. Limitations Scores depend on the completeness and quality of the reference answer. Automated support judgments can be mistaken; audit samples manually. High context recall does not guarantee faithful or correct generation. For truly unanswerable questions, measure abstention behavior separately. Interview takeaway Context recall measures evidence completeness for the expected answer, whereas context precision measures retrieval noise. Evaluate both before blaming the language model for missing information.

How do you evaluate groundedness in a RAG answer?

Evaluating groundedness in RAG means checking whether each factual claim in the generated answer is supported by the retrieved evidence. Unlike measuring answer relevance, this evaluation asks whether the answer stays within what the sources justify . Practical evaluation workflow 1. Capture the evidence: Save the exact retrieved passages and their source identifiers used for generation. 2. Split the answer into atomic claims: Separate compound statements so each factual assertion can be checked independently. 3. Judge support claim by claim: Label each claim as supported , contradicted , or not established by the retrieved passages. A claim may require combining multiple passages. 4. Compute a groundedness score: Use a simple supported claim ratio or a documented rubric; report the chosen definition. 5. Inspect citations: Verify that each citation points to text that actually supports the associated claim. 6. Review disagreements: Sample human evaluations to validate LLM based judges, especially for numbers, dates, exceptions, and negations. Beginner friendly numerical example Retrieved policy: Employees receive 20 annual leave days . Up to 5 unused days may be carried forward with manager approval. Generated answer: Employees receive 20 annual leave days , can carry forward 10 days , and do not need manager approval. Break this answer into three claims: “20 annual leave days” → Supported . “10 days may be carried forward” → Contradicted ; the source says 5. “No manager approval required” → Contradicted ; approval is required. Using a simple claim support ratio: Groundedness = 1 supported claim / 3 total claims ≈ 0.33 (33%). This is an illustrative scoring method, not a universal groundedness formula. Important distinctions Groundedness / faithfulness: Is each claim supported by the retrieved context? Answer relevance: Does the answer address the user's question? Factual correctness: Does the answer agree with verified real world or organizational truth? A response can be grounded in a stale or incorrect document and still be factually wrong. Groundedness checks evidence use; it does not establish source accuracy. Common evaluation approaches Human rubric: Reviewers label claims and cite supporting passages. LLM as judge: A separate model assesses support using the answer and retrieved context. Calibrate its results against human reviewed examples. Natural language inference (NLI): An entailment model predicts whether context supports or contradicts each claim. Citation verification: Check whether cited spans actually substantiate the answer's statements. Failure cases to include in tests Unsupported numbers, dates, names, or policy exceptions. Claims that subtly reverse a negation. Unsupported conclusions formed by combining otherwise correct facts. Citations to irrelevant passages. Questions with no adequate evidence, where the correct behavior is abstention. Interview takeaway Groundedness evaluation is a claim to evidence audit. Split the answer into factual claims, verify support in retrieved passages, quantify unsupported claims, and validate automated judgments with human review.

How do you evaluate answer relevance in RAG?

Evaluating answer relevance in RAG means measuring how well the generated response addresses the user's actual question, independently of whether its claims are supported by retrieved passages. Practical evaluation workflow 1. Collect realistic questions: Include direct lookups, multi part requests, follow up questions, and queries with explicit constraints. 2. Capture generated answers: Keep the exact user question and model response for each test case. 3. Define a relevance rubric: Judge whether the response addresses the requested entity, scope, intent, and all required subquestions. 4. Score each response: Use trained human reviewers or an LLM judge with clear criteria and examples. 5. Review failures: Identify omissions, off topic content, unnecessary detours, and responses that answer a related but different question. 6. Validate the judge: Compare automated ratings with human labels on a representative sample. Beginner friendly numerical example Question: “What is the refund deadline for enterprise customers in India?” Suppose the generated response says: “Enterprise customers can request refunds through the customer portal.” A simple 0–3 relevance rubric might assign: 0: Off topic or does not address the request. 1: Related to the topic but misses the requested answer. 2: Answers part of the question or omits an important constraint. 3: Directly answers the complete question and its constraints. This response scores 1/3 : it discusses refunds but does not state the deadline or address the India specific scope. A response such as “Enterprise customers in India must request refunds within 30 days” would score 3/3 for relevance if it directly addresses the question—but it still needs separate verification for faithfulness and correctness. These scores are illustrative rubric judgments, not universal metric thresholds. Automated evaluation methods Rubric based LLM as judge: Give a judge the question, answer, scoring criteria, and a few labeled examples. Ask for a score and a short rationale. Question reconstruction: Generate a question implied by the answer and compare its meaning with the original question. This can help detect topic drift, but it is only a proxy. Human evaluation: Reviewers label directness, completeness, and adherence to constraints. Use this to calibrate automated metrics. Avoid relying solely on embedding similarity between question and answer: a response can share many keywords while missing the requested fact. How do you interpret results? Track average relevance by question type, along with the fraction of responses that fully answer all requested parts. Inspect low scoring examples and compare changes using the same evaluation dataset. Important distinction: Relevance asks “Did it answer the question?” Faithfulness asks “Is it supported by retrieved evidence?” A highly relevant hallucination can still score poorly on faithfulness. Interview takeaway Evaluate answer relevance with question–response alignment, completeness, and constraint adherence. Use an explicit rubric, concrete examples, and human calibrated automated judging; evaluate factual support separately.

What is an end-to-end RAG evaluation dataset?

An end to end RAG evaluation dataset is a curated set of realistic user questions, expected answers, and supporting evidence used to test the complete retrieval augmented generation pipeline—from finding documents to producing a final answer. Unlike a retrieval only benchmark, it allows teams to assess whether the whole system produces accurate, grounded, relevant, and appropriately cautious responses. What does each test case contain? A useful evaluation record can include: Question: A realistic user query. Reference answer: The expected answer or required factual points. Relevant sources: Document or chunk IDs that support the answer, when known. Expected behavior: Answer normally, ask for clarification, or abstain if evidence is missing. Metadata: Topic, difficulty, language, tenant permissions, effective date, and dataset version. Optional annotations: Citation expectations, acceptable answer variants, and critical mistakes. Not every test case requires one exact reference sentence; many valid answers can express the same supported facts. Beginner friendly example Imagine an HR policy assistant. Test case 1 Question: “How many annual leave days do employees receive?” Reference answer: “20 days per year.” Relevant document: hr leave policy v3 Expected behavior: Answer with a supporting citation. Test case 2 Question: “Can I carry forward unused leave without approval?” Reference answer: “No. Manager approval is required.” Relevant document: hr leave policy v3 Expected behavior: Explain the approval requirement. Test case 3 Question: “What will the 2028 leave policy be?” Reference answer: No approved information available. Relevant document: None. Expected behavior: Acknowledge insufficient evidence; do not invent a policy. Together, these cases test retrieval, answer generation, and abstention. How do you build the dataset? 1. Sample real questions: Use anonymized, permission safe user queries and supplement them with realistic scenarios. 2. Cover different difficulties: Include direct lookup, multi document reasoning, ambiguous queries, conflicting documents, and unanswerable questions. 3. Annotate trusted answers: Have subject matter experts identify required facts and authoritative evidence. 4. Capture access and time constraints: A correct answer for one region, user role, or policy version may be wrong for another. 5. Review label quality: Check agreement between annotators and resolve inconsistent reference answers. 6. Version the dataset: Freeze a regression test split and keep a separate set for development or tuning. What can you measure? Retrieval: Recall@K, Precision@K, ranking quality, and access correctness. Generation: Faithfulness, answer relevance, factual correctness, and citation support. End to end: Task success, abstention quality, latency, cost, and failure rate. Run the same test cases after changing embeddings, chunking, rerankers, prompts, or language models. Compare results by category rather than relying only on one aggregate score. Common mistakes Evaluating only easy questions. Using synthetic questions that copy source wording too closely. Treating a single reference wording as the only acceptable answer. Mixing test examples into prompt tuning or training. Ignoring document updates, tenant permissions, or unanswered questions. Interview takeaway A good end to end RAG evaluation dataset connects questions, trusted evidence, expected behavior, and reference answers so retrieval and generation can be tested together—and diagnosed separately.

How do you create ground-truth query-document pairs for RAG evaluation?

Ground truth query document pairs are labeled examples linking a realistic user question to the document or chunk that contains evidence needed to answer it. They let you test whether a RAG retriever finds the right source , independently of the final LLM answer. How to create them 1. Choose a trusted corpus. Freeze a version of your source documents and assign stable document IDs. Preserve section and chunk IDs when evaluating chunk retrieval. 2. Collect realistic questions. Start with anonymized user queries, support tickets, or questions written by domain experts. Include natural paraphrases, not just wording copied from the source. 3. Find all relevant evidence. For each question, identify passages that directly support the answer. Label multiple passages if each is independently sufficient or if several are jointly required. 4. Assign relevance labels. Use binary labels (relevant/not relevant) or graded labels (for example, 0–3) with a written rubric. 5. Validate the labels. Have another reviewer check ambiguous cases, conflicting sources, and questions with no answer in the corpus. 6. Store versioned records. Keep question ID, query text, source ID, relevance grade, corpus version, and annotation provenance. 7. Build a held out test set. Keep evaluation queries separate from examples used to tune chunking, embeddings, or reranking. Beginner friendly example Suppose an HR knowledge base contains: doc 17, section 2: “Employees receive 20 annual leave days.” doc 17, section 4: “Unused leave can be carried forward with manager approval.” doc 42: “Travel expenses are reimbursed monthly.” For the query “How many annual leave days do employees receive?” : Pair (q 001, doc 17 section2) → Relevant (grade 3) Pair (q 001, doc 17 section4) → Not relevant (grade 0) Pair (q 001, doc 42) → Not relevant (grade 0) The first pair is a positive example; the other two can serve as negatives. Do not assume an unreviewed document is irrelevant merely because it was not selected. How do you handle difficult cases? Multiple valid sources: Label every independently sufficient passage as relevant. Multi hop questions: Record the complete set of evidence passages required to answer; a single positive document may be insufficient. Unanswerable queries: Mark that no supporting document exists and evaluate abstention separately. Permissions and time: Attach access scope and effective date information so the correct source is evaluated for the right user and document version. Hard negatives: Include topically similar but incorrect, outdated, or incomplete passages; verify labels before using them. What metrics use these labels? Recall@K: Fraction of labeled relevant items found in the top K. Precision@K: Fraction of top K items labeled relevant. MRR: How early the first relevant item appears. nDCG: Whether highly relevant items appear near the top. For multi hop tasks, also measure whether the full required evidence set was retrieved. Common mistakes Labeling only one passage when several are valid. Generating questions directly from a chunk and evaluating on near identical wording. Treating missing labels as confirmed negatives. Mixing document level and chunk level IDs without a mapping. Changing the corpus without updating ground truth. Interview takeaway Good ground truth is more than a question and one answer document. It needs stable source IDs, complete relevance judgments, difficult negatives, and a reproducible corpus version.