Retrieval-Augmented Generation
Retrieval-augmented generation connects a generative model to a searchable knowledge source. Instead of relying only on model weights, the system fetches relevant documents and places them into the prompt or context.
What is Dynamic Top-K retrieval?
Dynamic Top K retrieval is a RAG retrieval strategy that adjusts how many document chunks are selected based on the question, retrieval evidence, or available context budget, instead of always returning a fixed number. How does it work? 1. Retrieve an initial candidate pool: Search the index for relevant chunks using vector, keyword, or hybrid retrieval. 2. Assess the candidates: Examine retrieval or reranker scores, score gaps, document diversity, and the query's complexity. 3. Choose K adaptively: Keep fewer chunks if a short, high confidence set appears sufficient; keep more when the query needs evidence from several sources. 4. Apply constraints: Enforce permissions, deduplicate results, and respect token, latency, and cost limits. 5. Build the context: Send the selected chunks to the LLM and evaluate whether they support the answer. Question → candidate retrieval → relevance assessment → adaptive K → LLM context Practical example An internal knowledge assistant handles two questions: “What is the company's leave allowance?” One current policy section may answer directly, so a small K might be enough. “Compare leave rules for employees across three countries.” The assistant may need several different country specific policy sections, so it can retain more chunks. The chosen number should depend on evidence quality and task requirements, not on question length alone. Ways to choose K dynamically Relevance threshold: Keep candidates above a calibrated relevance cutoff, with a minimum and maximum number of results. Raw scores from different retrieval methods are not automatically comparable. Score gap or ranking signals: Stop when evidence becomes substantially weaker, after validation on real queries. Token budget: Select as many useful, nonredundant chunks as fit the context budget. Query complexity or coverage: Allocate more evidence for comparison and multi part questions; ensure all requested subtopics have support. Reranker based selection: Retrieve a larger initial pool, then retain only candidates that pass a reranker or evidence quality criterion. Benefits and limitations Potential benefits: Less irrelevant context for simple questions, improved evidence coverage for complex questions, and more efficient token usage. Limitations: Poor thresholds can discard essential evidence; noisy similarity scores may select too many chunks; more sophisticated selection adds processing. Production checks: Evaluate Recall@K or evidence coverage, grounded answer quality, token cost, and latency on representative query types. Dynamic Top K is not a single standardized algorithm. Different systems implement different selection rules. A stopping rule based only on similarity should not be treated as proof that retrieved evidence is sufficient. Interview takeaway Fixed Top K always retrieves the same number of chunks. Dynamic Top K adapts the number to relevance, question complexity, and context constraints. Start with a candidate pool, apply a calibrated selection rule, and validate that the resulting answers stay grounded.
When should Dynamic Top-K be used?
Use Dynamic Top K in RAG when different questions need different amounts of supporting evidence. Instead of always selecting the same number of chunks, the system adjusts how many it keeps based on query needs, relevance signals, and context limits. When is Dynamic Top K useful? Mixed question complexity: A simple factual question may need one policy chunk, while a comparison across departments may require several independent passages. Variable evidence quality: Some searches return a few highly relevant passages; others need a wider candidate set to find enough useful evidence. Strict token or latency budgets: The system can avoid sending many low value chunks for straightforward requests. Multi document questions: Answers that compare rules, dates, or products often need evidence from multiple sources. Collections with duplicate or noisy content: Adaptive selection plus deduplication can prevent repeated passages from filling the context. Practical example An HR assistant receives two questions: 1. “How many paid holidays do employees get?” One current policy section may contain the complete answer, so the system can select a small number of chunks. 2. “Compare parental leave policies in India, Germany, and Canada.” The answer may require separate country specific sections, so a larger selected set may be needed. Dynamic Top K can allocate context according to each query's evidence requirements rather than using, for example, five chunks for both. How do you decide whether to expand or stop? A practical implementation can: 1. Retrieve an initial pool of candidates, such as Top 20. 2. Deduplicate and rerank candidates if needed. 3. Check calibrated relevance scores, source diversity, and whether the requested subquestions have supporting evidence. 4. Select enough chunks to cover the question without exceeding token and latency limits. 5. If evidence is inadequate, expand retrieval or run targeted subqueries instead of confidently answering from incomplete context. The numbers are illustrative and must be tuned to the workload. High similarity alone does not prove that a passage answers the question. When might fixed Top K be preferable? A narrow, predictable question type with consistent document structure. A small knowledge base where a measured fixed K already meets retrieval quality goals. Very tight latency requirements where adaptive selection brings little measurable benefit. Dynamic methods introduce tuning, evaluation, and sometimes reranking overhead; they are not automatically better. How should you evaluate the choice? Compare fixed and dynamic strategies on representative queries. Track answer bearing evidence coverage , grounded answer quality, p95 latency, and context token cost. Also inspect failures in which adaptive selection omitted a necessary source. Interview takeaway Dynamic Top K is most useful when RAG questions vary in complexity and the amount of evidence required. Choose chunk counts adaptively using validated relevance and coverage signals, but keep fixed Top K when it is simpler and performs just as well.
What is pre-filtering in RAG retrieval?
Pre filtering in RAG retrieval means restricting which indexed chunks are eligible before or during the similarity search process, using structured conditions such as tenant, country, language, document status, or publication date. How does it work? 1. Store searchable metadata alongside each chunk, such as tenant id, region, document type, and status. 2. Determine safe filter conditions from trusted application context and the user's query. 3. Submit the filter with the retrieval request. The search engine constrains the eligible set while identifying nearest neighbors or lexical matches. 4. Return up to Top K relevant chunks from the permitted set. 5. Rerank or pass those chunks to the LLM with their source references. Query + trusted filter → eligible chunks → ranked matches → RAG context Example An employee asks: “What is the leave policy in India?” Suppose the index holds policies for India, Germany, and Canada. A filter such as: country = "India" AND status = "active" helps retrieval concentrate on current Indian policy chunks instead of other countries' similarly worded policies. For a multi tenant application, an additional mandatory tenant id restriction must come from authenticated application context—not from an untrusted user supplied value. Pre filtering versus post filtering Pre filtering: Constrains eligible results before or during retrieval. It can avoid filling Top K with ineligible chunks. Post filtering: Retrieves candidates first, then removes those that fail the conditions. A small initial candidate pool may leave too few valid results and reduce recall. The exact implementation varies: some vector databases integrate filters into approximate nearest neighbor (ANN) search rather than literally creating a filtered subset first. Benefits Relevant context: Excludes obviously unrelated regions, products, versions, or document types. Better use of Top K: More returned candidates can belong to the desired subset. Potential efficiency: Selective filters can reduce work, depending on indexing and query planning. Important limitations Over filtering hurts recall: Restricting to India may hide a globally applicable policy. Include shared documents where business rules require them. Selective ANN queries can be challenging: Some indexes may need special strategies to maintain recall and latency for restrictive filters; benchmark the chosen engine. Metadata may be incorrect or stale: Keep access levels, versions, and effective dates synchronized with source documents. Filtering is not sufficient authorization: Enforce access rights server side and, where appropriate, at the database/security layer; do not depend on model generated filters alone. Interview takeaway Pre filtering makes RAG retrieval search only eligible chunks using structured metadata constraints before or during ranking. It can improve relevance and effective Top K coverage, but filter correctness, ANN behavior, and secure tenant scoping must be validated.
What is post-filtering in RAG retrieval?
Post filtering in RAG retrieval means applying metadata or business rule conditions after an initial retrieval step has produced candidate chunks. The system first finds potentially relevant passages, then removes candidates that do not satisfy the required filters. How does it work? 1. Retrieve an initial Top K candidate set using vector, lexical, or hybrid search. 2. Inspect each candidate's structured metadata, such as country, language, document status, or source category. 3. Discard chunks that fail the selected conditions. 4. Optionally retrieve more candidates if too few eligible chunks remain. 5. Rerank and pass the remaining permitted evidence to the LLM. Query → initial candidates → filter candidates → eligible passages → LLM Practical example An HR assistant searches for “annual leave rules in India.” Suppose an unfiltered search returns five similar policy chunks: Two documents for India Two for Germany One for Canada Applying country = "India" after retrieving those five candidates leaves only two. Even if more relevant India documents exist elsewhere in the index, the initial Top 5 search may never have retrieved them. This is the central candidate starvation risk of post filtering. Post filtering versus pre filtering Pre filtering: Constrains eligible documents before or during retrieval, allowing Top K to be chosen among matching records. Post filtering: Searches a broader corpus first, then removes mismatches from its candidate set. It can return fewer than K results and reduce recall when filters are selective. Actual execution differs by search engine; some systems integrate filtering into approximate nearest neighbor search rather than using a separate pass. When can post filtering be useful? A small candidate pool with inexpensive filter logic. Additional business rules that the search index cannot express directly. Rechecking retrieved data against current source metadata before display. For restrictive filters, consider supported pre filtering or retrieving a larger candidate pool and repeating the search if needed. Increasing the initial K is a workaround, not a guarantee of complete recall. Security considerations Do not rely on post filtering alone as the security boundary for confidential or multi tenant data. Enforce tenant and document authorization in a trusted retrieval/data access layer. Check authorization before any retrieved text is sent to an LLM, logs, caches, or a client; prefer hard restrictions within retrieval when supported. Interview takeaway Post filtering removes ineligible chunks after candidate retrieval. It is straightforward to implement but can reduce recall and produce fewer than K results if the initial candidate pool contains many nonmatching chunks. Pre filtering is often preferable for selective metadata constraints, especially alongside strict access controls.
How do user permissions affect retrieval in enterprise RAG?
User permissions determine which documents a RAG retriever is allowed to search and return for a particular user. In enterprise systems, a passage can be highly relevant but still must not be retrieved into the LLM's context if the user is not authorized to see it. How permissions affect retrieval 1. Authenticate the user: Establish a trusted identity through the application's identity provider. 2. Resolve permissions: Identify the user's organization, roles, groups, and any document specific access rights from authoritative systems. 3. Attach access metadata: During ingestion, associate chunks with source document IDs, tenant IDs, and access control information (or a secure reference to it). 4. Restrict candidates: Apply server enforced tenant and permission constraints before sensitive text reaches the LLM, logs, or user facing results. 5. Recheck sensitive access: Where permissions can change frequently, validate current authorization with the source of truth before using retrieved content. 6. Preserve citations safely: Return only source references the user can actually open; do not reveal forbidden document titles or snippets. Practical example An enterprise assistant indexes HR policies, public engineering guides, and confidential salary reviews. An engineer can retrieve general engineering guides and company wide HR policies. A compensation manager may also retrieve restricted salary review documents. If an engineer asks about executive compensation, the assistant must not use salary review passages even when those vectors score highest. The right behavior is to retrieve only permitted evidence and, if necessary, say that the available authorized documents do not answer the question. Common authorization patterns RBAC: Grant access by role, such as HR manager or support engineer. ABAC: Evaluate attributes such as region, classification, business unit, and employment status. Document ACLs: Honor per document allowed users or groups inherited from systems such as SharePoint or Google Drive. Tenant isolation: Restrict each query to its authenticated organization or tenant. These approaches are often combined. The LLM should never be the authority deciding whether a user may read a document. Security pitfalls Prompt only rules are insufficient: Telling the model “do not reveal confidential files” cannot replace retrieval layer authorization. Post filtering alone can leak content: Unauthorized passages must not enter model prompts, caches, debug traces, or response snippets before they are removed. Stale ACLs: Group membership and document permissions can change; synchronize or revalidate them, and invalidate affected caches. User provided filters are untrusted: Derive mandatory tenant and access conditions from verified server side identity, not the question text. Embedding and metadata leakage: Protect vector indexes, stored chunks, logs, and citation endpoints consistently. Interview takeaway Enterprise RAG retrieval must be authorization aware. Authenticate the requester, resolve trusted permissions, enforce them at retrieval and data access boundaries, keep ACLs fresh, and never allow unauthorized chunks into the LLM's context—even if they are highly relevant.
What is query expansion in RAG?
Query expansion in RAG improves retrieval by adding related search terms or generating additional queries from the user's original question. The goal is to find relevant source passages that use different wording, while preserving the original intent. How it works 1. Read the original question: Identify the key concept, identifiers, constraints, and what information the user needs. 2. Generate complementary terms or queries: Add useful synonyms, related phrases, acronym expansions, or alternative formulations. 3. Search with the expansions: Run the original query and one or more expanded queries against lexical, dense, or hybrid indexes. 4. Merge results: Deduplicate chunks, combine ranked candidate lists (for example, with Reciprocal Rank Fusion), and optionally rerank them against the original question. 5. Generate an answer: Give the LLM only relevant, authorized evidence from the merged results. Practical example A user asks: “How do I get reimbursed for WFH equipment?” Relevant policy documents might use “home office expenses” , “remote work allowance” , or “equipment reimbursement” instead of “WFH.” Query expansion can search those variants and retrieve the correct policy even when the exact wording differs. The system should preserve any explicit limits in the original query—such as country, effective date, employee group, or product ID—rather than broadening the question indiscriminately. Common expansion techniques Synonym or acronym expansion: Add WFH, work from home, and remote work where appropriate. Domain dictionaries: Include known company, medical, legal, or product terminology. LLM generated multi query retrieval: Produce several intent preserving variations and fuse their retrieved results. Pseudo relevance feedback: Use terms from initially retrieved, likely relevant passages to refine or expand a second search. This must be evaluated carefully because early results can be wrong. Query expansion versus query rewriting Expansion: Enriches an existing search with related terms or additional query variants, often retaining the original. Rewriting: Reformulates a question into a clearer or more search friendly version, potentially replacing the original query. They can overlap, and both may be used together. Benefits and pitfalls Better recall: Finds evidence missed because of vocabulary mismatch. More robust acronym handling: Especially helpful in domain specific documents. Query drift: Overbroad or incorrect synonyms may retrieve unrelated passages. Latency and cost: Running more queries and an LLM expander requires extra work. Safety: Do not let generated expansions relax authorization, tenant boundaries, or the user's explicit constraints. Compare approaches on an evaluation set using Recall@K, evidence precision, answer groundedness, and latency. Expansion is useful when it improves real queries—not automatically for every question. Interview takeaway Query expansion broadens retrieval using synonyms, domain terms, or multiple meaning preserving query variants. Search the original and expanded queries, merge results, and rerank against the user's actual intent to improve recall without introducing irrelevant evidence.
What is query decomposition in RAG?
Query decomposition in RAG breaks a complex question into smaller, focused subquestions that can be searched and answered separately. It helps when a single retrieval query is unlikely to find all the evidence needed for a complete answer. How does it work? 1. Understand the user's goal: Identify the separate facts, comparisons, or reasoning steps required. 2. Split into subquestions: Generate focused, answerable queries without changing the user's constraints. 3. Retrieve evidence for each: Search the appropriate knowledge sources for every subquestion, enforcing the same user permissions. 4. Collect and verify results: Deduplicate sources, check whether each subquestion has supporting evidence, and preserve citations. 5. Synthesize the answer: Combine the supported findings into a response addressing the original question. If evidence is missing, state the gap instead of inventing it. Complex question → focused subquestions → retrieval per subquestion → evidence synthesis → final answer Practical example A user asks: “How did our refund policy change between 2024 and 2026, and does the new policy affect enterprise customers?” The system could retrieve separately for: What did the 2024 refund policy say? What does the 2026 refund policy say? What special terms apply to enterprise customers in the new policy? It then compares the verified policy clauses and cites the correct versions. A single broad query might retrieve only the latest policy or miss the enterprise exception. Decomposition versus query expansion Query expansion: Searches alternative phrasings or related terms for substantially the same information need. Query decomposition: Separates a question into distinct information needs that must be combined to answer it. For example, expanding “remote work allowance” to “WFH reimbursement” is expansion; separately retrieving eligibility, limits, and country specific exceptions is decomposition. When is it helpful? Multi part or comparison questions Questions spanning several documents or time periods Multi hop questions where one fact is needed to find another Requests requiring evidence from different systems Limitations and best practices More cost and latency: Multiple searches can be expensive; independent subqueries can run in parallel. Missing dependencies: Some questions must be answered in sequence—for example, finding a product ID before retrieving its warranty. Error propagation: Incorrect subquestions or misleading early evidence can derail the final answer. Access and traceability: Preserve trusted permissions, source versions, and citations for every retrieved passage. Evaluation: Test subquestion coverage, answer correctness, grounding, and end to end latency against a simpler one query baseline. Interview takeaway Query decomposition transforms a complex RAG question into focused subquestions, retrieves evidence for each, and synthesizes a supported answer. It improves coverage for multi part and multi hop tasks, but adds orchestration, latency, and opportunities for errors.
What is query routing in RAG?
Query routing in RAG means choosing the most appropriate knowledge source, index, retriever, or retrieval strategy for a user's question before gathering evidence. Rather than searching every source in the same way, a router directs each question to the place most likely to contain the answer. How does it work? 1. Inspect the query: Identify the topic, intent, explicit constraints, and relevant conversation context. 2. Choose a destination: Select one or more sources, such as an HR document index, technical documentation, a structured SQL database, or an approved search service. 3. Retrieve evidence: Run the selected source's retrieval method—for example, BM25, vector search, hybrid search, or an authorized database query. 4. Validate the results: Check source relevance, freshness, access permissions, and whether the evidence answers the question. 5. Generate a grounded response: Provide the retrieved evidence and source references to the LLM; if retrieval fails, use a controlled fallback or acknowledge missing information. Question → router → selected source/retriever → evidence → LLM answer Practical example A company assistant has three knowledge sources: HR policies: Leave eligibility and employee benefits. Engineering docs: API errors and deployment runbooks. Product catalog: Pricing and product specifications. For “What is the annual leave policy?” , routing to the HR index is more targeted than searching all three. For “Why does the payments API return 429?” , the engineering index is more relevant. A mixed question may legitimately need multiple destinations. Routing does not have to choose exactly one. Common routing approaches Rule based routing: Use known product, document type, or intent signals. It is simple and auditable but less flexible. Classifier based routing: Predict the most relevant source from labeled examples or embedding similarity. LLM based routing: Ask a model to select tools or sources from descriptions and an allowed set. Validate the output and restrict it to approved destinations. Multi route retrieval: Search several sources when the question spans domains or routing confidence is low, then merge or rerank authorized results. Routing versus query decomposition Routing decides where or how to search . Decomposition decides which smaller questions must be answered . A complex request may be decomposed into subquestions and each subquestion routed to a different source. Production considerations Enforce tenant isolation and document permissions regardless of the route . Treat routing as a prediction: a wrong destination can hide the correct evidence. Track route accuracy, evidence Recall@K, latency, fallback frequency, and grounded answer quality. Keep routing rules and source descriptions updated when indexes or tools change. Use controlled fallbacks rather than exposing unauthorized sources or guessing an answer. Interview takeaway Query routing directs a RAG question to the most relevant retrieval source or strategy. It reduces unnecessary searches and can improve evidence relevance, but must support multi source questions, safe fallbacks, and strict authorization.
What is a cross-encoder reranker?
A cross encoder reranker is a model that scores how relevant a retrieved document chunk is to a query by processing the query and chunk together . In RAG, it improves the order of a small candidate set produced by a faster first stage retriever. How does it work? 1. Retrieve candidates: Use vector search, BM25, or hybrid retrieval to find, for example, 20 potentially relevant chunks. 2. Pair query and chunk: For each candidate, input the original question and the candidate text together into the cross encoder. 3. Predict relevance: The model assigns a score to each query–chunk pair. It can attend to interactions between specific words and phrases in both texts. 4. Sort and select: Rerank the candidates by those scores, then send the best authorized, nonredundant chunks to the LLM. Question → fast retrieval → query–chunk pair scoring → reranked passages → answer Practical example A user asks: “Can I get a refund after 30 days if the item is defective?” Initial retrieval might rank a general 30 day return policy above a defective item exception . A cross encoder can read the entire question together with each passage and give the exception higher relevance because it addresses the important condition “if the item is defective.” That does not guarantee correctness; it improves evidence ordering when the model is effective for the domain. Cross encoder versus bi encoder Bi encoder: Encodes queries and documents separately. Document vectors can be precomputed, making large scale nearest neighbor retrieval efficient. Cross encoder: Evaluates each query–document pair jointly. It often provides more precise relevance judgments but requires a separate model pass for every candidate. Because document representations cannot generally be independently precomputed for cross encoder scoring, cross encoders are normally used after first stage retrieval, not to scan millions of chunks directly. Trade offs and tips Accuracy versus latency: Scoring more candidates may improve ordering but adds computation. Candidate recall: A reranker cannot recover a relevant chunk that was never retrieved. Long passages: Input length limits may truncate important evidence, so chunk size and truncation strategy matter. Score calibration: Scores are often useful for ranking, not universal probabilities that an answer is correct. Security: Enforce tenant and document access restrictions before passing chunk text to a reranking service. Evaluate the reranker on labeled query–passage pairs using metrics such as nDCG@K, MRR, downstream grounded answer quality, and p95 latency. Interview takeaway A cross encoder reranker jointly reads a query and each retrieved candidate to predict relevance. It is typically slower than vector similarity but can rank a small candidate list more accurately, improving which evidence reaches the RAG generator.
What is LLM-based reranking?
LLM based reranking uses a large language model to reassess which retrieved document chunks best answer a user's question. It usually runs after a fast retriever (BM25, vector, or hybrid search) has produced a manageable candidate list. How does it work? 1. Retrieve candidates: Search broadly, for example returning 20 passages from authorized sources. 2. Present the question and candidates: Give the reranking LLM clear instructions to judge relevance to the original question , not simply topical similarity. 3. Rank candidates: The model scores passages or returns an ordered list of candidate IDs. 4. Validate the ranking: Keep only valid IDs from the supplied candidate set, deduplicate them, and retain source metadata. 5. Send selected evidence: Pass the strongest relevant passages to the answer generation model. Question → first stage retrieval → LLM reranking → best evidence → grounded answer Practical example A customer asks: “Can I return an opened device after 30 days if it is defective?” The initial retriever finds a general 30 day return policy, a warranty page, and a defective product exception. An LLM reranker can inspect the full conditional question and prioritize the defective product exception over a more general, superficially similar passage. Common LLM ranking approaches Pointwise: Evaluate each query–passage pair separately and assign a relevance judgment or score. Pairwise: Compare two candidate passages and choose which better answers the question; multiple comparisons produce a ranking. Listwise: Consider several passages together and return their IDs in relevance order. Large sets may need windowing or multiple passes. These techniques have different cost, stability, and ranking quality trade offs. LLM reranker versus cross encoder Cross encoder reranker: A purpose built relevance model that jointly processes each query–chunk pair, often with efficient numerical scoring. LLM reranker: Uses generative or instruction following models to judge relevance, sometimes comparing several passages in one prompt. It can follow complex relevance instructions but may consume more tokens and be slower. Neither approach is always superior; benchmark both on the actual query distribution. Production considerations Cost and latency: Prefer reranking a shortlist rather than the full corpus. Position bias: List ordering can influence LLM judgments; test prompt and ordering stability. Output reliability: Validate returned passage IDs and handle malformed or incomplete rankings. Candidate recall: Reranking cannot recover evidence absent from initial retrieval. Security: Check access permissions before sharing passages with the reranking model or service. Evaluation: Measure nDCG@K, MRR, grounded answer quality, token cost, and p95 latency. Interview takeaway LLM based reranking asks a language model to judge the relevance of retrieved passages—pointwise, pairwise, or listwise—before selecting evidence for RAG generation. It can better account for nuanced questions, but brings extra latency, cost, and output validation requirements.