How do you perform offline evaluation of a RAG system?

Offline RAG evaluation tests a retrieval-augmented generation system against a fixed, labeled dataset before deploying changes to live users. It lets teams compare versions of retrievers, prompts, rerankers, and language models reproducibly.

Step-by-step workflow

1. Create a representative test set. Include real or expert-written questions, trusted answers, relevant source IDs, unanswerable questions, and multi-document cases. Keep sensitive user data out of the benchmark.
2. Freeze the evaluation conditions. Record the document-corpus version, index snapshot, access permissions, model version, prompts, retrieval settings, and evaluation code.
3. Run the complete RAG pipeline. For each question, log retrieved document IDs and ranks, the assembled context, generated answer, citations, latency, and token usage.
4. Score retrieval separately. Compare retrieved items against relevance labels using Recall@K, Precision@K, MRR, or nDCG, depending on the goal.
5. Score generation separately. Evaluate answer correctness, faithfulness to retrieved context, answer relevance, citation support, and appropriate abstention.
6. Compare with a baseline. Use the *same* questions and evaluation conditions for both versions; examine per-question regressions, not just aggregate averages.
7. Review failures and gate release. Manually inspect important failures and decide whether the candidate meets predefined quality, latency, cost, and safety requirements.

Beginner-friendly example

Imagine a support assistant with 100 labeled customer questions. You are deciding whether to replace its retriever.

Current version

• Finds the required evidence in the top 5 results for 78/100 questions.
• Produces fully supported answers for 70/100 questions.

Candidate version

• Finds the required evidence in the top 5 for 87/100 questions.
• Produces fully supported answers for 76/100 questions.

The candidate improves both observed measures on this benchmark. However, inspect the questions that became worse and check whether latency or cost increased before deployment.

These are illustrative rates; “required evidence found” and “fully supported answer” must be defined consistently. A small sample may not establish that the improvement generalizes.

Useful offline metrics

• Retrieval: Recall@K, nDCG@K, evidence-set coverage.
• Generation: Answer correctness, groundedness/faithfulness, relevance, citation validity.
• Behavior: Abstention accuracy on unanswerable questions.
• Operations: p50/p95 latency, token usage, estimated cost, errors.

Common pitfalls

• Data leakage: Tuning on the same test questions can inflate scores.
• Stale labels: Updated documents may invalidate expected answers.
• Weak automated judges: LLM-based ratings can disagree with domain experts; calibrate on human-reviewed samples.
• Changing multiple variables: It becomes hard to attribute improvements to one component.
• Ignoring slices: Aggregate scores can hide failures for specific languages, regions, tenants, or question types.

Offline versus online evaluation

Offline evaluation uses a controlled dataset and does not expose live users to an untested candidate. Online evaluation measures real production traffic and user outcomes, typically after offline checks pass.

Interview takeaway

Offline RAG evaluation is a reproducible regression test for retrieval, generation, and operational behavior. Freeze the test conditions, compare against a baseline, and inspect failure cases before rollout.