Retrieval-Augmented Generation
Retrieval-augmented generation connects a generative model to a searchable knowledge source. Instead of relying only on model weights, the system fetches relevant documents and places them into the prompt or context.
What about tables, charts and scanned documents?
Tables, charts, and scanned documents need special handling in RAG because important information is often stored in layout and visual structure , not just plain text. 1. Tables Do not flatten a table into random text and lose the relationship between headers and values. For example: Product | Price | Warranty Laptop A | 80,000 | 2 years A useful chunk should preserve: Column headers Row values Table title Nearby explanatory text Page and document metadata For large tables, you can split by groups of rows, but each chunk should still carry the relevant headers. Otherwise a retrieved value such as "2 years" may become meaningless. 2. Charts and graphs Charts usually require visual interpretation. A chart may contain: Title X and Y axes Legend Labels Trends Data points If the underlying chart data is available, store that structured data. If not, a vision model can create a searchable description. For example: "Revenue increased from 120 crore in 2024 to 165 crore in 2025, while operating margin remained nearly flat." Store that description together with a reference to the original chart. For questions that depend on exact visual details, a multimodal model may also need the original chart image at answer time. 3. Scanned documents A scanned PDF may contain no usable text layer at all. The pipeline becomes: Scanned page → OCR → layout detection → cleaned text → chunking → embedding OCR should preserve as much structure as possible, including: Headings Paragraphs Page numbers Form fields Table boundaries Poor OCR can create bad embeddings, so OCR quality is part of retrieval quality. 4. Keep the original source linked For all three cases, preserve metadata such as: Document ID Page number Table or figure number Bounding box or location when useful Original image/page reference This allows the system to return citations and, when necessary, send the original visual to a multimodal LLM. Example Suppose a financial report contains a chart showing that revenue increased 38%, but the surrounding text never states that number. A text only RAG pipeline may completely miss it. A better pipeline can index: The surrounding paragraph A vision generated chart description The original chart reference Then a query such as "How much did revenue grow?" has a chance to retrieve the correct evidence. Interview takeaway Tables should preserve row column relationships, charts should be converted into searchable structured data or visual descriptions, and scanned documents need OCR plus layout reconstruction. In every case, keep the extracted representation linked to the original page or visual so retrieval remains accurate and traceable.
How do embeddings work in a RAG pipeline?
Embeddings are the bridge between raw text and semantic retrieval in a RAG pipeline. They convert document chunks and user queries into numerical vectors so the system can compare them by meaning. During ingestion Each document is split into chunks. Example: "Employees can carry forward up to 10 unused leave days." That chunk is passed through an embedding model: Chunk → embedding model → vector Conceptually: [0.12, 0.44, 0.81, ...] The vector, original chunk, and metadata are stored together in a vector index. At query time Suppose the user asks: "How many unused leave days can I keep for next year?" The query is embedded using the same or a compatible embedding model: Query → embedding model → query vector Now the system can compare that query vector with the stored chunk vectors. Why this works The user and the document may use different words: Query: "How many unused leave days can I keep?" Document: "Employees can carry forward up to 10 unused leave days." Keyword overlap is limited, but the meanings are closely related. A good embedding model places semantically similar text near each other in vector space. Retrieval step The system calculates similarity between the query vector and document vectors. Common measures include: Cosine similarity Dot product Euclidean distance The closest chunks become retrieval candidates. Conceptually: Query embedding ↓ Compare with stored chunk embeddings ↓ Rank by similarity ↓ Return Top K chunks What embeddings do not do Embeddings do not generate the answer. Their main job is to help find relevant evidence. The full flow is: Documents → chunks → embeddings → vector index Then: User query → query embedding → retrieve relevant chunks → LLM → answer Important limitation Embeddings are strong at semantic similarity, but they can be weaker for exact identifiers, codes, numbers, or rare terms. That is why production RAG systems often combine dense embedding search with lexical methods such as BM25. Interview takeaway Embeddings convert document chunks and user queries into comparable vectors. During ingestion, document vectors are stored in an index. At query time, the user query is embedded and compared with those stored vectors so the system can retrieve semantically relevant chunks before the LLM generates the answer.
Dense retrieval vs sparse retrieval — what's the difference?
Dense retrieval and sparse retrieval are two different ways to find relevant information in a RAG system. The simplest difference is: Dense retrieval → searches by meaning Sparse retrieval → searches by matching terms Dense retrieval Dense retrieval converts the query and document chunks into embeddings. Example: Query: "How can I get my money back?" Document: "Customers may request a refund within 30 days." The wording is different, but the meanings are similar. A dense retriever can still match them because the query and document vectors are close in embedding space. Dense retrieval is strong for: Paraphrases Synonyms Conceptual similarity Natural language questions Its weakness is that it can sometimes miss exact identifiers, rare names, codes, or numbers. Sparse retrieval Sparse retrieval represents text using terms or token weights instead of dense semantic vectors. BM25 is the classic example. Suppose the query is: "ERR AUTH 401" A sparse retriever is very good at finding documents containing that exact error code. Sparse retrieval is strong for: Exact keywords Product names Error codes IDs Rare technical terms Numbers and exact phrases But it is weaker when the query uses very different wording from the document. Example Query: "How do I terminate my subscription?" Document A: "You can cancel your plan from account settings." Document B: "Subscription termination policy" Dense retrieval may strongly prefer Document A because the meaning matches. Sparse retrieval may prefer Document B because the exact term "subscription" appears. Why are the vectors called dense and sparse? Dense embeddings usually contain many non zero numerical values, for example: [0.12, 0.34, 0.78, 0.05, ...] Sparse representations have a very large vocabulary sized space, but only a small number of terms have non zero weights. Conceptually: refund → 1.8 policy → 0.9 customer → 0.4 Most other vocabulary terms have a weight of zero. Which one should you use? Use dense retrieval when semantic meaning matters most. Use sparse retrieval when exact wording, identifiers, or rare terms matter. In many production RAG systems, the best answer is not choosing one. It is combining both. Dense search can find meaning based matches, while sparse search catches exact lexical matches. That combination is called hybrid retrieval . Interview takeaway Dense retrieval uses embeddings to find semantically similar content even when the wording differs. Sparse retrieval uses lexical signals such as BM25 to find documents that share important words or terms with the query. Dense is strong for meaning; sparse is strong for exact matches. Production RAG systems often combine both.
What is hybrid retrieval?
Hybrid retrieval combines dense semantic search and sparse keyword search so the system can benefit from both meaning based matching and exact term matching. Instead of asking "Should I use vectors or BM25?", hybrid retrieval says: use both, then combine the results. Why combine them? Dense retrieval is good at meaning. Example: Query: "How can I get my money back?" Document: "Customers may request a refund within 30 days." Even though the wording differs, embeddings can match the meaning. Sparse retrieval is good at exact terms. Example: Query: "ERR AUTH 401" A BM25 style retriever can quickly find documents containing that exact error code. Hybrid retrieval helps when your users ask both kinds of questions. Typical flow User query ↓ Run dense retrieval ↓ Run sparse retrieval ↓ Combine the two result lists ↓ Optional reranking ↓ Send best chunks to the LLM For example: Dense results: 1. Chunk A 2. Chunk C 3. Chunk D Sparse results: 1. Chunk B 2. Chunk A 3. Chunk E The system then fuses these rankings into one final list. How are the results combined? Common approaches include: Reciprocal Rank Fusion (RRF) Weighted score fusion Relative score normalization Rank based fusion Reranking after combining candidates The important point is that dense and sparse systems often produce scores on different scales, so blindly adding raw scores is usually not ideal. That is why rank based methods such as RRF are popular. Example Suppose the query is: "How do I fix ERR AUTH 401 when login tokens expire?" Dense retrieval may find: "Authentication failures can happen when access tokens are no longer valid." Sparse retrieval may find: "ERR AUTH 401: token expired." The sparse result catches the exact error code, while the dense result captures the broader meaning. Together they give the system stronger evidence than either retriever alone. Does hybrid retrieval always improve results? Not automatically. It adds: More retrieval work More latency More tuning A fusion step If dense retrieval already performs extremely well for your dataset, the extra complexity may not be worth it. Use hybrid retrieval when evaluation shows that semantic search misses important exact matches or lexical search misses paraphrased questions. Interview takeaway Hybrid retrieval combines dense vector search with sparse lexical search such as BM25. Dense retrieval captures semantic meaning, while sparse retrieval captures exact words, identifiers, and rare terms. Their result lists are fused, often followed by reranking, to improve retrieval robustness.
If you are using BM25 + vector search, how do you combine the results?
When you use BM25 and vector search together, you usually run both retrievers, collect their candidate results, then fuse the two ranked lists into one final ranking . The important point is that BM25 scores and vector similarity scores are usually on different scales, so you should not blindly add the raw numbers. Step 1: Run both retrievers Example query: "How do I fix ERR AUTH 401 after my token expires?" BM25 might return: 1. Chunk B 2. Chunk A 3. Chunk E Vector search might return: 1. Chunk A 2. Chunk C 3. Chunk D BM25 likes Chunk B because it contains the exact error code. Vector search likes Chunk A because it matches the meaning of the query. Step 2: Merge and deduplicate Take the union of candidates from both retrievers. For example: A, B, C, D, E If the same chunk appears in both lists, keep one copy but remember that both retrievers found it. That overlap is often a strong relevance signal. Step 3: Combine the rankings There are several common approaches. Reciprocal Rank Fusion RRF combines rank positions rather than raw scores. A document ranked highly by either retriever gets a useful contribution, and a document ranked highly by both usually moves further up. This is popular because BM25 and vector scores do not need to be directly comparable. Weighted score fusion Another option is to normalize the scores from each retriever and combine them. Conceptually: Final score = 0.4 × BM25 score + 0.6 × vector score The weights depend on your workload. If exact identifiers matter heavily, you may give BM25 more influence. If semantic questions dominate, vector search may get more weight. The key requirement is to normalize the scores first if their scales differ. Step 4: Optionally rerank After fusion, you may have a candidate list such as: Top 20 fused results → reranker → best 5 A cross encoder or LLM based reranker can inspect the query and each candidate more deeply than the initial retrievers. This is often where the final ordering improves significantly. Example Suppose Chunk A is: "Authentication fails when an access token has expired." Chunk B is: "ERR AUTH 401: access token expired." Vector search may rank A first. BM25 may rank B first. Fusion keeps both strong candidates instead of forcing you to choose only one retrieval method. Which fusion method should you start with? A practical starting point is: BM25 + vector search → RRF → optional reranker RRF is attractive because it is simple and avoids the problem of incompatible raw score scales. Weighted score fusion can work very well too, but it requires more calibration. Interview takeaway When combining BM25 and vector search, retrieve candidates from both, merge and deduplicate them, then fuse their rankings using a method such as Reciprocal Rank Fusion or normalized weighted scores. After fusion, a reranker can reorder the strongest candidates before the final context is sent to the LLM.
Reciprocal Rank Fusion? Weighted scores? Reranking?
Reciprocal Rank Fusion, weighted scores, and reranking are three different ways to improve the final ordering of retrieved documents. They are related, but they solve different parts of the problem. 1. Reciprocal Rank Fusion (RRF) RRF combines multiple ranked lists using rank position , not the original retrieval scores. Suppose BM25 returns: 1. Chunk A 2. Chunk B 3. Chunk C Vector search returns: 1. Chunk B 2. Chunk D 3. Chunk A RRF gives more credit to chunks that rank highly, especially when they appear near the top of multiple lists. A common form is: RRF score = 1 / (k + rank) The exact value of k is a tuning choice. The main benefit is that BM25 scores and vector scores do not need to be on the same scale. Use RRF when you want a simple, robust way to combine retrievers. 2. Weighted score fusion Weighted fusion combines retrieval scores directly, usually after normalization. Conceptually: Final score = 0.4 × BM25 score + 0.6 × vector score This gives you explicit control over how much each retriever contributes. For example, if your application depends heavily on exact product codes, you may increase the BM25 weight. If semantic questions dominate, you may give vector search more weight. The important part is normalization. A BM25 score of 12 and a cosine similarity of 0.83 are not directly comparable. Raw scores should therefore not usually be added without calibration. 3. Reranking Reranking is different. Instead of mathematically combining retrieval scores, a reranker takes a smaller candidate set and evaluates each candidate more carefully against the query. Example: BM25 + vector retrieval ↓ Fuse results ↓ Top 30 candidates ↓ Cross encoder reranker ↓ Best 5 chunks A reranker is usually slower but more accurate because it can jointly examine the query and candidate text. How do they work together? A strong production pipeline may use all three ideas: BM25 + vector search ↓ RRF or weighted fusion ↓ Deduplicate ↓ Rerank the top candidates ↓ Send the best chunks to the LLM Fusion improves candidate combination. Reranking improves final ordering. Which one should you use? Use RRF when: Retriever score scales are incompatible You want a simple fusion method You want minimal tuning Use weighted fusion when: You understand the score distributions You want explicit control over retriever importance You can tune weights on evaluation data Use reranking when: Initial retrieval has good recall but imperfect ordering Final context quality matters enough to justify extra latency Interview takeaway RRF combines ranked lists using rank positions, weighted fusion combines normalized retrieval scores, and reranking applies a stronger second stage model to reorder a smaller candidate set. In practice, retrieval systems often use fusion first and reranking afterward.
Why do we need a reranker after retrieval?
A reranker is used because the first retrieval stage is usually optimized for speed and recall , not perfect ordering. The retriever's job is to quickly find a broad set of potentially relevant chunks. The reranker's job is to look at that smaller set more carefully and decide which chunks are actually the best match for the query. Example Suppose the user asks: "Can I get a refund after 30 days if the product is defective?" Initial retrieval may return: 1. General refund policy 2. Shipping policy 3. Defective product exception 4. Warranty terms 5. Return instructions The correct answer may depend on Chunk 3, but a vector retriever might rank the general refund policy higher because it is broadly similar to the query. A reranker can compare the full query with each candidate and move the defective product exception to the top. Why doesn't the first retriever do this already? Dense vector search and BM25 are designed to search large collections efficiently. They use relatively simple relevance signals: Vector similarity Keyword overlap Term statistics These signals are excellent for finding candidates, but they do not always capture the exact relationship between the full query and the full chunk. A reranker can perform a deeper comparison. Typical pipeline Query ↓ Retrieve Top 20–100 candidates ↓ Reranker ↓ Keep best 3–10 ↓ Send to the LLM This lets you retrieve broadly first, then be selective before generation. What kind of model is used? A common choice is a cross encoder. Instead of embedding the query and document separately, it processes them together: Query + candidate chunk → relevance score Because the model sees both texts at the same time, it can often judge relevance more accurately than simple vector similarity. LLMs can also be used for reranking, although they are typically more expensive and slower. Why reranking helps RAG Better ranking can: Put the strongest evidence first Remove weakly related chunks Reduce irrelevant context Improve answer quality Reduce token usage when fewer chunks are passed forward It is especially useful when retrieval has good recall but poor precision . In other words, the right document is already somewhere in the candidate set, but it is not ranked high enough. What is the trade off? Reranking adds extra computation and latency. That is why you normally do not rerank the entire knowledge base. Instead: Fast retriever → small candidate set → expensive reranker Interview takeaway We use a reranker because first stage retrieval is designed to find relevant candidates quickly, while a reranker performs a deeper query document comparison to improve the final ordering. It is most useful when retrieval already finds the correct evidence but does not rank it high enough.
How do you evaluate retrieval quality?
Retrieval quality is about one question: when a user asks something, does the retriever bring back the right evidence, and does it rank that evidence high enough? You should evaluate retrieval separately from the LLM answer. 1. Build a small evaluation set Create realistic user queries and identify the chunks or documents that should be considered relevant. Example: Query: "What is the refund period for enterprise customers?" Relevant chunk: "Enterprise customers may request a refund within 45 days." This gives you a ground truth to test against. 2. Run retrieval and inspect Top K Suppose the retriever returns: 1. Shipping policy 2. Enterprise refund policy 3. Standard refund policy 4. Warranty policy 5. Billing FAQ The correct chunk is present, but only at rank 2. That is already useful information. 3. Measure retrieval metrics Common metrics include: Recall@K → did we retrieve the relevant item within the top K? Precision@K → how many of the top K results are actually relevant? MRR → how early does the first relevant result appear? NDCG → are highly relevant results ranked near the top? These metrics are especially useful when comparing: Different embedding models Different chunk sizes Dense vs hybrid retrieval Different Top K values Rerankers 4. Measure more than one thing A retriever can have high recall but poor precision. For example: Top 20 contains the correct chunk, but 18 results are irrelevant. That may be acceptable before reranking, but not ideal for the final LLM context. So evaluate both: "Did we find the answer?" and: "How much irrelevant content came with it?" 5. Inspect failure cases manually Metrics tell you that something failed. Failure analysis tells you why. Common causes include: Bad chunking Weak embeddings Exact keywords missed by dense retrieval Missing metadata filters Relevant chunk ranked too low OCR or parsing errors Query wording that needs rewriting A few manually reviewed failures can reveal more than a single aggregate score. 6. Evaluate on realistic queries Do not test only simple questions. Include: Paraphrases Exact IDs or error codes Multi part questions Ambiguous queries Long queries Domain specific terminology Queries that should return no answer This makes the evaluation closer to production behavior. 7. Separate retrieval from generation If the correct chunk is not retrieved, the problem is retrieval. If the correct chunk is retrieved but the LLM still answers incorrectly, the problem is likely in generation, context construction, prompting, or model behavior. This distinction is essential when debugging RAG. Interview takeaway Evaluate retrieval by using a labeled set of realistic queries, checking whether relevant chunks appear in the top results, measuring metrics such as Recall@K, Precision@K, MRR, and NDCG, and reviewing failures manually. Retrieval should be measured independently from final answer quality so you know whether the problem is in retrieval or generation.
What are Recall@K, Precision@K, MRR and NDCG?
Recall@K, Precision@K, MRR, and NDCG are common metrics for measuring how well a retriever ranks relevant documents. They answer slightly different questions. 1. Recall@K Recall@K asks: "Did we retrieve the relevant documents within the top K results?" Suppose there are 2 relevant chunks for a query. Top 5 retrieval contains 1 of them. Recall@5 = 1 / 2 = 0.5 High recall means the retriever is good at not missing useful evidence. In RAG, Recall@K is important because if the correct chunk never reaches the LLM, the model cannot use it. 2. Precision@K Precision@K asks: "How many of the top K retrieved results are actually relevant?" Suppose Top 5 contains: 2 relevant chunks 3 irrelevant chunks Precision@5 = 2 / 5 = 0.4 High precision means the retrieved context contains less noise. That matters because too many irrelevant chunks can waste tokens and confuse the LLM. 3. MRR — Mean Reciprocal Rank MRR focuses on how early the first relevant result appears . For one query: If the first relevant result is at rank 1: Reciprocal rank = 1 / 1 = 1.0 If it appears at rank 2: 1 / 2 = 0.5 If it appears at rank 5: 1 / 5 = 0.2 MRR is the average reciprocal rank across many queries. It is useful when finding one good answer quickly matters. 4. NDCG — Normalized Discounted Cumulative Gain NDCG evaluates the quality of the entire ranking , especially when documents can have different levels of relevance. For example: Highly relevant chunk Partially relevant chunk Weakly related chunk Irrelevant chunk NDCG rewards systems that place the most relevant chunks near the top. It also penalizes relevant chunks that appear too far down the list. Unlike MRR, which mainly cares about the first relevant result, NDCG looks at the broader ranking quality. Simple way to remember them Recall@K: "Did we find the relevant evidence?" Precision@K: "How much of what we retrieved is useful?" MRR: "How quickly did the first relevant result appear?" NDCG: "How good was the overall ordering of results?" Example Suppose the correct chunks are B and D. Retriever returns: 1. B 2. X 3. Y 4. D 5. Z Recall@5 = 2/2 = 1.0 Precision@5 = 2/5 = 0.4 The first relevant result is at rank 1, so reciprocal rank = 1.0. NDCG would also consider that B is ranked first while D appears later at rank 4. Which metric matters most for RAG? There is no single best metric. A useful combination is: Recall@K to check whether the retriever finds the right evidence Precision@K to measure noise MRR to measure how quickly relevant evidence appears NDCG to measure overall ranking quality Interview takeaway Recall@K measures how much relevant evidence was retrieved, Precision@K measures how much of the retrieved set is relevant, MRR measures how early the first relevant result appears, and NDCG measures the quality of the overall ranked list while giving more importance to highly relevant results near the top.
How do you know whether the problem is with retrieval or generation?
The fastest way to debug a RAG system is to separate it into two stages: 1. Did retrieval bring back the right evidence? 2. If yes, did the LLM use that evidence correctly? This distinction prevents you from changing the wrong part of the system. Step 1: Inspect the retrieved chunks Start with the exact user query and look at the Top K chunks returned by the retriever. Ask: Is the correct evidence present? Is it ranked high enough? Is important context missing? Are irrelevant chunks dominating the results? If the correct evidence is not present at all, the problem is retrieval. Retrieval side failure example User asks: "How long can enterprise customers request a refund?" Correct document says: "Enterprise customers have a 45 day refund window." But retrieval returns only: Standard 30 day refund policy Shipping policy Warranty policy The LLM never received the enterprise policy. That is a retrieval problem. Possible causes include: Poor chunking Weak embeddings Wrong Top K Missing metadata filters Dense search missing an exact term Bad OCR or parsing Query wording that needs rewriting Step 2: Test the LLM with the correct context Now manually provide the correct chunk to the LLM with the same question. If the model answers correctly, retrieval was the issue. If it still answers incorrectly, the problem is downstream of retrieval. That may involve: Prompt instructions Context formatting Too much distracting context Conflicting documents The LLM ignoring evidence Unsupported reasoning Generation settings Generation side failure example Suppose retrieval returns: "Enterprise customers have a 45 day refund window." But the model answers: "Enterprise customers have 30 days." The necessary evidence was available, so retrieval succeeded. Now investigate the generation stage. A useful debugging matrix Think in four cases: Correct evidence retrieved + correct answer → System is working. Wrong evidence retrieved + wrong answer → Retrieval problem. Correct evidence retrieved + wrong answer → Generation or context construction problem. Wrong evidence retrieved + correct answer → Dangerous case. The model may be answering from prior knowledge or guessing instead of grounding itself in retrieved evidence. That last case can look successful while hiding a weak RAG pipeline. Evaluate the stages separately Retrieval can be measured with metrics such as: Recall@K Precision@K MRR NDCG Generation can be evaluated for: Correctness Faithfulness to retrieved context Citation accuracy Completeness Whether unsupported claims were added Do not rely only on final answer accuracy, because it hides where the failure happened. Interview takeaway To determine whether a RAG failure comes from retrieval or generation, first inspect the retrieved context. If the required evidence is missing, fix retrieval. If the correct evidence is present but the answer is still wrong, investigate context construction, prompting, or model generation. A useful diagnostic is to manually give the correct context to the LLM and see whether the answer improves.