Retrieval-Augmented Generation

Retrieval-augmented generation connects a generative model to a searchable knowledge source. Instead of relying only on model weights, the system fetches relevant documents and places them into the prompt or context.

What happens when the correct document is retrieved but the LLM still gives the wrong answer?

If the correct document is retrieved but the LLM still gives the wrong answer, retrieval has probably done its job. The failure is now in context construction or generation . 1. Check whether the exact answer is actually inside the retrieved chunk A document can be correct at the document level but the retrieved chunk may still be incomplete. Example: The retrieved document is the refund policy, but the chunk only contains: "Enterprise customers have a different refund policy." The actual rule: "Enterprise customers have 45 days." may be in the next chunk. So first confirm that the exact evidence needed to answer the question reached the model. 2. Check for too much irrelevant context The right chunk may be present, but surrounded by many distracting chunks. Example: Relevant chunk: "Enterprise customers have 45 days." Other chunks: "Standard customers have 30 days." "Promotional purchases have 14 days." "International returns follow separate rules." If the prompt sends all of them without clear structure, the model may choose the wrong policy. Better retrieval precision and reranking can help. 3. Check for conflicting or outdated documents Sometimes retrieval returns multiple documents that disagree. For example: Old policy: "Refund period is 30 days." New policy: "Refund period is 45 days." If both are provided without metadata such as effective date or version, the model may use the wrong one. This is not necessarily a retrieval miss. It may be a context selection or document versioning problem. 4. Check the prompt The model should be clearly instructed to answer from the retrieved evidence. Useful instructions include: Use only the supplied context when the answer depends on company specific information. Prefer newer or authoritative sources when documents conflict. Say that the answer is not available when the context does not support it. Cite the evidence used. A vague prompt gives the model more freedom to rely on prior knowledge or guess. 5. Check whether the model is reasoning correctly Sometimes the evidence is present and clear, but the model still makes a reasoning mistake. This can happen with: Multi step questions Numerical comparisons Conditions and exceptions Negation Information spread across multiple chunks For these cases, better context organization, a stronger model, structured extraction, or breaking the task into steps may help. 6. Run a controlled test Take only the correct evidence and send it to the LLM with the original question. If the model now answers correctly: The larger context or prompt was the problem. If the model still answers incorrectly: The issue is more likely the model's reasoning, instruction following, or the way the evidence is expressed. This is one of the quickest ways to isolate the failure. Interview takeaway When the correct document is retrieved but the answer is still wrong, do not immediately blame retrieval. Verify that the exact answer bearing chunk reached the model, remove distracting or conflicting context, inspect prompt instructions, and test the model with only the correct evidence. If it still fails, the issue is in generation or reasoning rather than retrieval.

How do you reduce hallucination in a RAG system?

RAG reduces hallucination by giving the LLM external evidence, but retrieval alone does not guarantee a grounded answer. The goal is to make it easier for the model to answer from evidence and harder for it to invent unsupported facts. 1. Improve retrieval quality If the wrong chunks are retrieved, the model starts with bad evidence. Focus on: Better chunking Better embeddings Hybrid retrieval Metadata filtering Appropriate Top K Reranking The first defense against hallucination is getting the right context. 2. Remove irrelevant context More context is not always better. If the prompt contains many weakly related chunks, the model may mix them together or choose the wrong fact. A better pattern is: Retrieve broadly → rerank → send only the strongest evidence This improves signal to noise ratio. 3. Tell the model to stay grounded Prompt instructions should make the expected behavior explicit. For example: Answer using the supplied context. Do not invent missing facts. If the context does not contain the answer, say so. Prefer authoritative or newer sources when documents conflict. Cite the supporting source. This does not eliminate hallucination, but it creates a much stronger constraint. 4. Allow the model to say "I don't know" A common cause of hallucination is forcing the model to always produce an answer. If the retrieved evidence is weak or missing, the safest behavior may be: "The available documents do not contain enough information to answer this." That is often better than a confident guess. 5. Handle conflicting documents Suppose retrieval returns: Old policy: "Refund period is 30 days." New policy: "Refund period is 45 days." If both are sent without metadata, the model may choose either one. Store and use metadata such as: Effective date Document version Source authority Department Status Then filter or instruct the model which source should take precedence. 6. Make citations traceable Ask the system to attach citations to important claims. Then verify that each citation actually supports the claim. A citation is useful only if it points to real evidence. This also makes hallucinations easier to detect during evaluation. 7. Use structured outputs where possible For tasks involving exact fields, dates, IDs, prices, or policies, structured extraction can reduce free form guessing. For example: Refund period: 45 days Source: Enterprise Policy v3 Effective date: 2026 01 01 The LLM can then generate prose from validated fields. 8. Evaluate faithfulness separately An answer can sound correct while containing unsupported details. Evaluate whether each factual claim is supported by the retrieved context. This is called grounding or faithfulness evaluation. Important point RAG does not "remove hallucination." It reduces the probability by giving the model relevant evidence and constraining how that evidence is used. A strong RAG system therefore combines: Good retrieval + clean context + grounding instructions + abstention + citations + evaluation Interview takeaway To reduce hallucination in RAG, improve retrieval quality, rerank and remove noisy context, instruct the LLM to answer only from evidence, allow it to abstain when evidence is missing, resolve conflicting sources with metadata, require traceable citations, and evaluate answer faithfulness separately from retrieval accuracy.

How do you handle metadata filtering?

Metadata filtering means narrowing retrieval using structured attributes attached to each document or chunk. Instead of searching the entire vector index, you first restrict the search to items that match conditions such as customer, region, document type, version, date, or product. Example Suppose your knowledge base contains HR policies for India, the US, and Germany. The user asks: "What is the maternity leave policy in India?" A pure vector search may retrieve semantically similar maternity policies from several countries. If every chunk carries metadata such as: country = India document type = HR policy effective date = 2026 01 01 you can filter to India before ranking by semantic similarity. That makes retrieval more precise. What metadata should you store? Useful metadata often includes: Document ID Source Department Product Customer or tenant Region or country Language Document type Version Effective date Access level Section or page Status such as active or archived The exact fields depend on the application. Pre filtering vs post filtering With pre filtering, you apply metadata conditions before vector similarity search. Example: country = India AND status = active Then vector search runs only inside that subset. This is efficient and prevents irrelevant documents from becoming candidates. With post filtering, vector search runs first and metadata rules are applied afterward. That can be easier in some systems, but it may hurt recall if the original Top K contains many documents that are later removed. Be careful with aggressive filtering Metadata filtering can also remove the correct result. Suppose the user asks: "How does the global refund policy apply to enterprise customers in India?" If you filter only to: country = India you may accidentally exclude the global enterprise policy. So filters should reflect the intent of the query rather than being applied blindly. Where do filters come from? Some filters are explicit. Example: "Show policies from 2026." Others can come from application context. For example: Current tenant User's organization Selected product User permissions And some may be extracted from the query: "refund policy for Germany" → country = Germany The system can use rules, entity extraction, or an LLM to identify these filters. Metadata filtering is also important for security In multi tenant RAG, metadata such as tenant id can help scope retrieval to the correct tenant. But application level metadata filtering should not be treated as the only security boundary. Access control still needs to be enforced by the underlying data layer. Interview takeaway Metadata filtering narrows the retrieval search space using structured attributes such as tenant, region, document type, version, date, or status. It improves precision and can reduce latency, but filters must be chosen carefully because overly strict filtering can remove relevant evidence. Pre filtering is generally preferable when the metadata constraint is known before retrieval.

How do you retrieve information from very large documents?

Very large documents should usually be searched in stages instead of treating the entire file as one flat list of chunks. The goal is to first identify the right region of the document, then retrieve the exact evidence from inside that region. 1. Preserve the document hierarchy Large documents often already have useful structure: Document → chapter → section → subsection → paragraph Store that hierarchy as metadata. For example, instead of indexing a chunk only as: "Timeout configuration is 30 seconds." also keep: Document: API Guide Chapter: Networking Section: Timeout Configuration This helps the retriever understand where the chunk belongs. 2. Use hierarchical retrieval A common approach is: Query → retrieve relevant section or chapter → search inside that section → retrieve the final chunks Example: A 1,000 page technical manual may contain 50 chapters. Instead of comparing every query equally against thousands of tiny chunks, first identify the likely chapter, then search within it. This reduces noise and can improve both speed and precision. 3. Use parent child retrieval Index small child chunks because they are easier to match precisely. But when a child is retrieved, also bring back its larger parent section. Example: Child chunk: "Retry limit is 5 attempts." Parent section: The full "Retry Policy" section explaining conditions, exceptions, and backoff behavior. This gives the retriever precision while giving the LLM enough context. 4. Index summaries as navigation layers For extremely large documents, you can create summaries for: Chapters Sections Major topics These summaries can act as a routing layer. The system may retrieve the relevant section summary first, then search the detailed chunks belonging to that section. The summary helps locate the right area; the final answer should still use the original evidence. 5. Use metadata and structural filters Metadata such as: Chapter Section Page range Product Version Document type can reduce the search space. For example, a query about "authentication timeout" may first be narrowed to the Authentication section before detailed vector search. 6. Do not solve the problem by only increasing chunk size Making huge chunks may preserve more context, but it also: Reduces retrieval precision Adds irrelevant text Uses more tokens Makes reranking harder Large documents usually benefit more from better hierarchy than from simply creating larger chunks. Practical pattern For a large document: Parse structure → create section and child chunks → store hierarchy metadata → optionally create section summaries → retrieve the best section → retrieve or rerank child chunks → return the child plus enough parent context Interview takeaway For very large documents, use hierarchical or parent child retrieval rather than treating every chunk independently. Preserve chapter and section metadata, optionally index summaries for routing, narrow the search to the relevant region first, then retrieve the exact answer bearing chunks with enough parent context for the LLM.

How do you deal with frequently changing data?

Frequently changing data creates a freshness problem in RAG. If the source changes but the vector index is not updated, the retriever may keep returning stale information. The solution is to update only what changed and make freshness part of retrieval. 1. Detect changes Track whether a document or record has changed using fields such as: updated at timestamp Version number Content hash Change event Source revision ID This lets the ingestion pipeline identify new, modified, and deleted content. 2. Re index incrementally Do not rebuild the entire knowledge base every time one document changes. A better flow is: Source changes → identify affected document → re parse changed content → regenerate affected chunks → create new embeddings → update the vector index This is called incremental ingestion or incremental indexing. 3. Remove stale chunks Updating content is not enough if the old vectors remain searchable. Suppose a policy changes from: "Refund period is 30 days." to: "Refund period is 45 days." If both old and new chunks remain in the index, retrieval may return conflicting answers. So the pipeline must replace or delete stale chunks when a document changes. 4. Store version and freshness metadata Useful fields include: document version updated at effective from effective to status is active These fields help retrieval prefer current content and exclude outdated versions. For policy or regulatory data, the effective date may be more important than the ingestion date. 5. Choose the right update frequency Not all data needs real time indexing. Examples: Employee handbook → nightly or on change Product catalog → every few minutes Breaking operational status → near real time The update strategy should match how quickly stale information becomes harmful. 6. Event driven vs scheduled updates Scheduled ingestion periodically checks for changes. Example: Every hour → find modified documents → re index them Event driven ingestion reacts immediately to a source event. Example: Database row updated → publish event → update corresponding vector record Event driven pipelines provide lower freshness lag but are more complex. 7. Consider direct retrieval for highly dynamic data Some data changes so often that embedding and re indexing it may not be the best design. Examples: Current inventory Live account balance Real time order status Current stock price For these cases, the RAG system may retrieve stable background knowledge from the vector store but call a database, API, or tool for the live value. Interview takeaway For frequently changing data, use incremental ingestion instead of rebuilding the entire index. Detect changes, regenerate only affected chunks and embeddings, delete stale vectors, keep version and effective date metadata, and choose scheduled or event driven updates based on freshness requirements. For truly real time data, querying the source directly may be better than embedding every change.

What are the trade-offs between retrieval accuracy and latency?

Retrieval accuracy and latency usually move in opposite directions. The more work you do to find the best evidence, the better retrieval can become—but the query often takes longer. The goal is not maximum accuracy at any cost. It is the best accuracy you can achieve within your latency budget. 1. Increasing Top K can improve recall If you retrieve more candidates, you reduce the chance of missing the correct chunk. For example: Top 3 → faster, but may miss useful evidence Top 20 → better recall, but more retrieval work and more candidates to process A larger Top K can therefore improve recall, but it also increases downstream cost. 2. Hybrid retrieval adds extra work Dense retrieval alone may be fast enough. Hybrid retrieval runs: Vector search BM25 or sparse search Fusion This can improve robustness, especially for exact terms and semantic matches, but it adds another retrieval path and a fusion step. 3. Reranking improves precision but adds latency A reranker examines a smaller candidate set more deeply. Example: Retrieve 50 candidates → rerank 50 → keep best 5 This often improves final ranking quality, but cross encoders or LLM based rerankers are slower than basic vector similarity. The more candidates you rerank, the greater the latency. 4. Exact search vs approximate search Large vector databases often use approximate nearest neighbor indexes. Approximate search is much faster than comparing the query against every vector. But aggressive speed settings can slightly reduce recall. You typically tune the index to balance: Speed ↔ Recall 5. More context can increase end to end latency Retrieval latency is only part of the story. If better retrieval returns many large chunks, the LLM receives more tokens. That can increase: Prompt processing time Generation latency API cost Noise in the final context So retrieving more is not automatically better. 6. Filtering can improve both speed and precision Good metadata filtering can reduce the number of candidates before vector search. Example: Search only: tenant = A country = India status = active instead of searching the full knowledge base. This can reduce latency while also preventing irrelevant results. 7. Use stages instead of one expensive search A common production pattern is: Fast first stage retrieval → moderate candidate set → rerank only those candidates → send a small final context to the LLM This is usually more efficient than applying an expensive relevance model to the entire corpus. How do you choose the balance? Measure both retrieval quality and latency on realistic queries. For example, compare: Top 5 vs Top 20 Dense vs hybrid No reranker vs reranking Top 20 Then track: Recall@K MRR or NDCG Retrieval latency End to end response time Token usage Choose the simplest configuration that meets the required quality. Interview takeaway Higher retrieval accuracy often requires more candidates, multiple retrievers, reranking, or deeper search, which increases latency and cost. The practical solution is to retrieve broadly enough for good recall, narrow candidates with filtering and reranking, and measure both retrieval metrics and end to end latency rather than optimizing either one in isolation.

What happens when you increase Top-K?

Top K is the number of retrieval results you keep for the next stage of a RAG pipeline. If Top K increases, you usually improve the chance of including the correct evidence—but you also bring in more noise and more work. Example Suppose the correct answer is in the 7th ranked chunk. If: Top K = 3 the correct chunk is missed. If: Top K = 10 the correct chunk is now included. So increasing Top K can improve recall. But higher Top K also adds irrelevant results Imagine the top 10 results contain: 3 highly relevant chunks 2 partially relevant chunks 5 irrelevant chunks The correct evidence is present, but the LLM now sees much more distracting information. That can reduce answer quality even though retrieval recall improved. It increases token usage If all retrieved chunks are passed directly to the LLM, increasing Top K means a larger prompt. For example: Top 5 × 400 token chunks ≈ 2,000 tokens Top 20 × 400 token chunks ≈ 8,000 tokens That increases: Input token cost Prompt processing latency Context noise It can increase reranking latency A good pipeline often retrieves a larger candidate set and reranks it. For example: Retrieve Top 50 → rerank → keep best 5 This can improve quality because the retriever gets high recall while the reranker removes weaker candidates. But reranking 50 chunks costs more than reranking 10. Bigger Top K does not guarantee a better answer This is the key point. Increasing Top K may improve retrieval recall while making generation worse because the model receives too much irrelevant or conflicting context. So: Higher Top K → usually better recall Higher Top K → usually lower precision Higher Top K → more latency and tokens The final answer quality depends on how these trade offs interact. A practical strategy Do not choose Top K by intuition alone. Test values such as: K = 3 K = 5 K = 10 K = 20 Then measure: Recall@K Precision@K Final answer quality Reranking latency End to end latency Token usage A common production pattern is: Retrieve a moderately large Top K → rerank → send only a small final set to the LLM Interview takeaway Increasing Top K usually improves recall because more candidate chunks are retrieved, but it also introduces more irrelevant context, increases reranking and LLM latency, and consumes more tokens. The best Top K is therefore an evaluated trade off—not simply the largest number the system can afford.

Better retrieval but more tokens and more latency?

Better retrieval often means retrieving more candidates, using stronger ranking methods, or adding extra retrieval stages. That can improve answer quality, but it usually increases token usage and latency. The key is to separate candidate retrieval from final context sent to the LLM . Why better retrieval can cost more Suppose you increase Top K from 5 to 30. You may improve recall because the correct chunk has a better chance of appearing. But now you have: More chunks to score More candidates to rerank More text to potentially send to the LLM More input tokens Higher latency and cost So retrieval quality can improve while system efficiency gets worse. Do not send every retrieved chunk to the LLM A strong pattern is: Retrieve many candidates → rerank → remove duplicates → keep only the strongest few → send those to the LLM For example: Retrieve Top 30 → rerank Top 30 → keep Top 5 → generate answer This gives the retrieval stage room to find the correct evidence without forcing the LLM to read all 30 chunks. Why more context can hurt More context is not automatically better. Extra chunks may contain: Repeated information Weakly related content Conflicting versions Irrelevant details That can distract the model and make the answer less accurate. So the goal is not maximum context. The goal is maximum useful evidence per token . Where does latency come from? Latency may increase at several stages: Running multiple retrievers Searching with a larger Top K Reranking more candidates Processing a larger LLM prompt Generating from a longer context Sometimes the retrieval itself is fast, but the larger prompt becomes the main source of latency. How do you balance it? Measure the system end to end. Compare configurations such as: Top 10 → rerank → keep 3 Top 30 → rerank → keep 5 Top 50 → rerank → keep 8 Then evaluate: Recall@K Final answer accuracy Input token count Retrieval latency Reranking latency Total response time Cost Stop increasing retrieval depth when the quality gain becomes small compared with the extra latency and cost. Interview takeaway Better retrieval can require a larger candidate set, hybrid search, or reranking, which increases latency and computation. But the LLM should not receive every retrieved chunk. A common production strategy is to retrieve broadly for recall, rerank aggressively, and pass only a small high quality context to the model so retrieval improves without unnecessarily increasing tokens and generation latency.

When should you use query rewriting?

Query rewriting means transforming the user's original question into a form that is easier for the retriever to match against the indexed content. It is useful when the user's wording is not the same as the wording used in the documents. When query rewriting helps 1. The query is too conversational User asks: "Why does this thing keep failing when I sign in?" But the documentation uses terms such as: "authentication failure" "expired access token" "login error" A rewritten query such as: "authentication failure caused by expired access token" may retrieve better evidence. 2. The query is vague User asks: "What is the limit?" The retriever does not know whether the user means: API rate limit File upload limit Credit limit Token limit If conversation context shows that the user was discussing API requests, the query can be rewritten as: "What is the API request rate limit?" 3. The user uses synonyms or abbreviations Query: "How long does PTO carry over?" Document: "Unused paid leave may be carried forward for 10 days." A rewritten query can include both forms: "PTO / paid leave carry forward limit" This can improve both dense and sparse retrieval. 4. The question contains several sub questions Example: "What is the refund period, and does it change for enterprise customers?" You may rewrite or decompose it into: What is the standard refund period? What is the enterprise refund period? Each sub query can retrieve more focused evidence. 5. Exact terminology matters For technical systems, users may describe an issue informally while the documentation uses exact product names, error codes, or field names. Query rewriting can map the user's wording to the terminology used in the corpus. Common rewriting approaches You can use: Rule based normalization Synonym expansion Acronym expansion LLM based rewriting Query decomposition Multiple query generation One useful strategy is to retrieve with both the original query and the rewritten version, then combine the results. This reduces the risk of losing useful information from the original wording. When query rewriting can hurt Rewriting is not always beneficial. A rewritten query may: Change the user's intent Remove important exact terms Add assumptions that were never stated Make a specific query too broad Increase latency because another model call is required Example: Original: "ERR AUTH 401" If an LLM rewrites it as: "authentication problem" it may remove the exact error code that BM25 would have matched perfectly. Practical rule Use query rewriting when evaluation shows that user wording and document wording frequently differ. Do not rewrite every query automatically. For exact identifiers, names, codes, or already clear questions, preserving the original query may be better. Interview takeaway Use query rewriting when the user query is vague, conversational, contains synonyms, depends on conversation context, or needs decomposition into smaller questions. The goal is to improve alignment between the query and the indexed content. But preserve important exact terms and validate rewriting carefully, because a bad rewrite can change intent and reduce retrieval quality.

When does RAG actually NOT make sense?

RAG is useful when an LLM needs external knowledge at query time. But not every problem needs retrieval. Sometimes RAG adds infrastructure, latency, and failure points without improving the answer. 1. When the model already knows enough If the task is general reasoning, rewriting, brainstorming, translation, or common knowledge, retrieval may not add much value. Example: "Rewrite this email professionally." There is no external knowledge to fetch. RAG would only make the system more complicated. 2. When the answer should come from a live system Some information changes too quickly to embed and re index reliably. Examples: Current stock price Live inventory Account balance Order status Today's available appointment slots For these cases, calling the source database or API directly is usually better. You can still combine that live data with RAG for background documentation, but the live value should come from the authoritative system. 3. When deterministic logic is enough If the problem can be solved reliably using rules, SQL, calculations, or a normal API, RAG may be unnecessary. Example: "What is the total invoice amount after 18% tax?" A calculator or deterministic function is better than retrieving documents and asking an LLM to compute it. 4. When the corpus is tiny Suppose the system has one short policy document. If the whole relevant document comfortably fits inside the model's context window, building a full ingestion pipeline, vector database, chunking strategy, and retriever may be unnecessary. Directly supplying the document may be simpler. That changes as the corpus grows or retrieval becomes selective. 5. When exact structured lookup matters RAG is designed for relevance based retrieval, not guaranteed exact database lookup. If the user asks: "What is customer 48291's current subscription tier?" you should query the customer database directly. Embedding customer records and hoping semantic search retrieves the exact row would be a poor design. 6. When retrieval errors are unacceptable RAG is probabilistic. The correct document may be: Missed Ranked too low Filtered out Misinterpreted For workflows requiring strict deterministic correctness, retrieval may need to be replaced or supplemented with structured queries, rules, validation, or human review. 7. When the knowledge is better learned as behavior Sometimes the goal is not to retrieve facts but to consistently produce a particular style, format, classification, or task behavior. Prompting, structured examples, or model fine tuning may be a better fit. For example: "Always classify support tickets into these five categories." The problem is primarily behavior, not external knowledge retrieval. 8. When the cost of RAG is greater than the benefit A production RAG system may require: Parsing Chunking Embeddings Vector storage Index updates Retrieval Reranking Evaluation Monitoring If a simpler architecture solves the problem well enough, use the simpler architecture. A useful decision question Ask: "Does this answer require external, selective, non parametric knowledge that cannot simply be queried directly?" If yes, RAG may be a strong fit. If no, consider a simpler solution. Interview takeaway RAG does not make sense when the task does not need external knowledge, when the answer should come directly from a live database or API, when deterministic logic is sufficient, when the corpus is small enough to provide directly, or when retrieval uncertainty is unacceptable. Use RAG for selective knowledge retrieval—not as a default architecture for every LLM application.