Retrieval-Augmented Generation

Retrieval-augmented generation connects a generative model to a searchable knowledge source. Instead of relying only on model weights, the system fetches relevant documents and places them into the prompt or context.

What is rule-based reranking?

Rule based reranking reorders retrieved RAG passages using explicit ranking rules instead of relying solely on similarity scores or a learned model. How it works 1. Retrieve a shortlist using BM25, dense search, or hybrid retrieval. 2. Inspect trusted metadata such as publication date, document status, source authority, exact identifiers, or product version. 3. Apply defined boosts, penalties, or ordering priorities. 4. Sort the permitted candidates and pass the most relevant evidence with citations to the LLM. Question → retrieval → business rule scoring → reordered evidence → answer Example A user asks: "What is our current travel reimbursement limit?" Initial retrieval finds: Chunk A: A highly similar but archived 2023 policy. Chunk B: The active 2026 policy. Chunk C: A general travel FAQ. A freshness and source authority rule can prioritize the active policy above the outdated one. Crucially, documents that are unauthorized or explicitly revoked should be excluded with hard filters , not merely demoted. Common reranking rules Freshness: Prefer currently effective documents. Authority: Boost approved policy sources over informal notes. Exact terms: Favor a matching product code, API endpoint, or error identifier. Version: Prefer the version requested by the user. Domain priorities: Boost appropriate source categories, subject to evaluation. A conceptual formula is: final score = relevance score + freshness boost + authority boost This is illustrative. Real systems must calibrate boosts against the retriever's score scale, or use explicit ordering priorities instead. Rule based versus model based reranking Rule based: Explainable, usually inexpensive, and auditable, but limited to encoded signals. Cross encoder or LLM based: Better able to judge nuanced question–passage relevance, but often more costly. Combined: Filter for authorization first, apply relevance reranking, then carefully tuned business priorities. Risks Aggressive or stale rules can push irrelevant chunks above answer bearing passages. Test with representative queries, relevance judgments (for example, nDCG@K), grounded answer quality, and latency. Interview takeaway Rule based reranking adjusts retrieved document order using transparent signals such as freshness, source authority, exact matches, and version relevance. Use it to express ranking preferences, never as a substitute for access control or hard exclusion filters.

What is hybrid reranking?

Hybrid reranking combines multiple ranking signals to reorder the document chunks retrieved for a RAG query. For example, it can combine cross encoder relevance scores with exact term matches, source authority, and document freshness. Hybrid retrieval merges candidates found by methods such as BM25 and vector search. Hybrid reranking takes retrieved candidates and judges their order using more than one signal. How it works 1. Retrieve a candidate shortlist using a suitable search method. 2. Apply mandatory document permissions and eligibility filters before any model processes the passages. 3. Evaluate candidates with a cross encoder or another relevance model. 4. Add validated signals such as an exact product code match, active document version, or trusted source ranking. 5. Fuse calibrated scores or combine ranked lists, then keep the best nonredundant evidence for the LLM. Practical example A question asks: “How do I resolve PAY 409 in the current payments API?” An archived troubleshooting guide matches PAY 409 exactly. The current API guide describes the applicable duplicate transaction behavior. A third chunk discusses an unrelated payment error. A hybrid reranker can consider both semantic relevance and the requested current version rather than treating an exact string match as the only important signal. Ways to combine rankings Weighted scoring: Normalize or calibrate model relevance scores and metadata based boosts before combining them. Staged ranking: Rerank using a cross encoder, then apply controlled version or source authority preferences. Rank fusion: Merge separate ranked lists using a method such as Reciprocal Rank Fusion (RRF). There is no single standardized hybrid reranking algorithm; implementations should specify their ranking signals and combination method. Benefits and limitations Benefit: Better balance of semantic relevance, exact identifiers, freshness, and source reliability. Cost: Additional compute, latency, tuning, and opportunities for conflicting signals. Pitfall: Raw scores from different methods are often not comparable. Uncalibrated boosts can push irrelevant documents too high. Security: Access restrictions are hard filters, never mere ranking penalties. Evaluation: Compare against a simple reranker using nDCG@K, MRR, answer groundedness, and p95 latency. Interview takeaway Hybrid reranking reorders RAG candidates using complementary signals—for example, cross encoder relevance and trusted metadata based priorities. It differs from hybrid retrieval and should be adopted only when measured quality gains justify its complexity.

How many candidates should be retrieved before reranking?

Retrieve enough candidates to give the reranker a good chance of seeing the correct evidence, but not so many that reranking becomes unnecessarily slow or expensive. There is no universal candidate count. Practical starting point Try first stage retrieval depths such as Top 10, Top 20, and Top 50 on a representative evaluation set. These are experimental settings, not fixed best practices. The right value depends on retriever recall, query complexity, reranker cost, and latency constraints. How to choose the candidate count 1. Measure first stage recall: With labeled questions, calculate whether an answer bearing chunk appears in the retrieved pool (for example, Recall@20 versus Recall@50). 2. Test the reranker: Feed each candidate pool into the same reranker and measure how often relevant passages reach the final context (such as the best five chunks). 3. Check latency and cost: More candidates require additional query–passage scoring. Inspect p95 latency and infrastructure or API costs. 4. Consider query types: Exact factual questions may need fewer candidates than ambiguous comparisons or multi document requests. 5. Deduplicate and filter: Remove duplicates and enforce access restrictions before spending reranker capacity on candidates. 6. Pick the smallest effective pool: Increase the depth only while it meaningfully improves evidence coverage or grounded answer quality. Example An HR assistant tests an initial retrieval pool of 10, 20, and 50 policy chunks before sending the top five reranked chunks to its LLM. Top 10: Occasionally misses country specific exceptions entirely. Top 20: Includes those exceptions in most evaluation questions. Top 50: Adds more reranking work with little additional gain. In this illustrative scenario, Top 20 is a reasonable choice. The result must be verified against real data. Candidate count versus final context These are different settings: Retrieve Top 20 → rerank 20 → send Top 5 to the LLM The first number controls potential evidence coverage . The final number controls the context selection and prompt token budget. A reranker cannot recover a correct chunk that initial retrieval omitted . If recall is poor, improve retrieval, chunking, hybrid search, or query transformation rather than simply increasing reranking complexity. Operational considerations Cross encoder inference usually grows with the number and length of candidate passages, although batching may improve throughput. LLM listwise reranking can become expensive or exceed input limits as the pool expands. Very large candidate sets may contain near duplicates that crowd out diverse evidence. Different question categories may justify different candidate depths, provided the routing logic is evaluated. Interview takeaway Tune reranking candidate depth empirically: retrieve enough chunks to achieve strong first stage evidence recall, then select a smaller high quality context after reranking. Optimize Recall@N, final ranking quality, p95 latency, and cost together.

What are the latency trade-offs of reranking?

Reranking improves the ordering of retrieved evidence, but adds processing between retrieval and answer generation. In RAG, the trade off is whether better selected context justifies the extra response time and compute cost. Where does reranking add latency? A typical pipeline is: Query → initial retrieval → reranking → context selection → LLM answer The first stage retriever efficiently returns a candidate shortlist. A cross encoder or LLM reranker then evaluates those candidates more carefully, which takes additional time. Main latency factors Candidate count: A cross encoder usually scores each query–chunk pair. Increasing the shortlist from 20 to 100 candidates increases work, although batching can improve throughput. Passage length: Longer chunks require more input processing and may trigger truncation or larger model context costs. Reranker type: Lightweight cross encoders can be faster than large generative LLM rerankers; actual performance depends on model, hardware, and deployment. Parallelism and batching: GPU batching or concurrent scoring can reduce per request latency, but heavy concurrency may increase queue times. Network overhead: A remote reranking service adds request, transfer, and queuing latency. Multiple passes: Listwise LLM reranking or multi stage reranking may require several calls. Practical example Suppose a support chatbot retrieves 50 troubleshooting chunks. The final response only needs five. Without reranking: The top five from vector search are sent directly to the LLM. It is faster but might miss the precise fix. With reranking: A model reviews the 50 candidates and chooses the best five. Answers may become more relevant, but the additional scoring can delay the response. With a smaller reranking pool: Reranking only 20 candidates may reduce latency—provided those 20 still contain the answer bearing evidence. These are illustrative configurations, not guaranteed speed or accuracy gains. How can you reduce latency? 1. Improve first stage retrieval: Use hybrid search, filters, or better chunking so fewer candidate passages are needed. 2. Tune retrieval depth: Measure evidence Recall@N and choose the smallest candidate pool that preserves required coverage. 3. Use efficient rerankers: Benchmark smaller cross encoders against larger LLM rerankers on your domain. 4. Batch and cache carefully: Batch query–chunk inference; cache only where data freshness and user access boundaries are respected. 5. Rerank selectively: Bypass expensive reranking for validated easy query classes if doing so does not harm answer quality. 6. Watch the full pipeline: Monitor retrieval, reranking, model generation, and p95/p99 end to end response time separately. Quality versus speed Measure nDCG@K or MRR , evidence coverage, grounded answer quality, cost per request, and p95 latency. A reranker is worthwhile when improvements in relevance justify its additional latency for the application's service level target. Interview takeaway Reranking exchanges extra inference, network, and queue time for potentially better evidence ordering. Control latency through candidate pool tuning, smaller models, batching, selective reranking, and end to end benchmarking—without sacrificing first stage recall.

How do you combine the results of vector and graph search?

Combining vector and graph search in RAG brings together two kinds of evidence: vector search finds passages with similar meaning, while graph search follows explicit relationships between entities. This is useful for questions that need both relevant text and connected facts. How can the results be combined? 1. Find semantic matches: Embed the user's question and retrieve relevant chunks or entity nodes from a vector index. 2. Find connected evidence: Use a knowledge graph to follow relevant relationships, such as service → depends on → database. 3. Link the outputs: Use stable document IDs, entity IDs, or chunk to entity references to connect the vector hits to graph results. 4. Merge and deduplicate: Combine evidence from both sources. If both produce ranked passage lists, use rank fusion, such as Reciprocal Rank Fusion (RRF), or calibrated scores. Do not add raw graph scores to vector scores without defining comparable scales. 5. Rerank and validate: Favor evidence that actually answers the question, check relationship provenance, and retain citations. 6. Generate: Provide the relevant text and verified graph relationships to the LLM. A common alternative is vector search first, graph expansion second : find matching entities by embedding similarity, then traverse their neighbors to enrich the context. Practical example A user asks: “Which services could be affected if the payments database fails?” Vector search finds incident reports and runbooks discussing payment failures. Graph search follows explicit dependencies from the payments database to services such as billing and checkout. Combined evidence identifies potentially affected services and supplies supporting runbook sections. The LLM can explain the dependencies and cite their sources. What makes integration reliable? Keep entity and document identifiers consistent across the vector index and graph. Restrict graph traversal by relevant relationship types and a reasonable hop limit. Apply tenant and document permissions to both retrieval branches before exposing content. Track the provenance and freshness of graph edges as well as source passages. Avoid treating a graph connection as proof of causality; relationships need appropriate semantics. Measure retrieval coverage, grounded answer quality, query latency, and cost against vector only retrieval. How does this differ from BM25 + vector hybrid search? BM25 + vector search combines lexical matching with semantic similarity. Vector + graph retrieval combines semantic matching with structured entity relationships and graph traversal. Both may coexist in a broader retrieval system. Interview takeaway Combine vector and graph search by linking semantic matches to graph entities, expanding relevant relationships, merging or reranking evidence, and preserving source citations. It helps RAG answer relationship heavy and multi hop questions that isolated text similarity may miss.

What is relative score fusion?

Relative score fusion combines results from multiple RAG retrievers by normalizing each retriever's scores relative to its own candidate list , then combining the normalized values. It is useful when BM25 and vector search produce scores on different scales. How does it work? 1. Run two retrieval methods, such as BM25 and dense vector search. 2. Keep each result's original retrieval score and stable chunk ID. 3. Normalize scores separately within each retriever . A common approach is min–max normalization: normalized = (score minimum) / (maximum minimum). 4. Merge candidates by chunk ID. When a chunk is absent from one list, follow a defined missing result policy, often assigning that branch zero contribution. 5. Combine the normalized scores with chosen weights, then sort the merged results. 6. Optionally rerank the best candidates before sending them to the LLM. Simple numerical example Suppose the query asks about payment error PAY 409. Two chunks have these example scores: Chunk A: BM25 = 12, vector = 0.70 Chunk B: BM25 = 6, vector = 0.90 Assume the BM25 score range of the retrieved set is 2–12 and the vector score range is 0.50–0.90. After min–max normalization: Chunk A: BM25 = 1.00, vector = 0.50 Chunk B: BM25 = 0.40, vector = 1.00 With equal 50% weights: A: 0.5 × 1.00 + 0.5 × 0.50 = 0.75 B: 0.5 × 0.40 + 0.5 × 1.00 = 0.70 So A ranks above B under this illustrative fusion rule. In production, the normalization range should come from each retriever's complete candidate score list, not merely these two displayed chunks. How is this different from RRF? Relative score fusion uses normalized score magnitudes . The distance between scores affects the final ranking. Reciprocal Rank Fusion (RRF) uses rank positions rather than raw similarity values. It avoids the need to normalize different score scales. Relative score fusion can preserve useful score differences, but it is sensitive to normalization choices and outlier scores. Important cautions If all scores in one list are identical, min–max normalization divides by zero. Define a safe fallback. A single extreme score can compress the remaining normalized values. Weights must be validated on representative queries rather than chosen arbitrarily. Candidate list sizes, missing records, and retriever score distributions can change results. Enforce document permissions before exposing or combining retrieved text; evaluate Recall@K, nDCG@K, and grounded answer quality. Interview takeaway Relative score fusion normalizes each retriever's scores within its own result set and combines them using weights. Unlike RRF, it retains relative score differences, but requires careful normalization, missing result handling, and evaluation.

What is rank-based weighted fusion?

Rank based weighted fusion combines ranked results from multiple RAG retrievers while giving each retriever a different level of influence. Unlike score based fusion, it uses rank positions , so BM25 scores and vector similarity scores do not need the same numerical scale. How it works 1. Run two or more retrievers, such as BM25 and vector search. 2. Give each candidate a rank within each retriever's result list (rank 1 is best). 3. Assign retriever weights based on validated workload needs. 4. Calculate a combined rank based score for each candidate. 5. Merge duplicate chunk IDs, sort by the combined score, and optionally rerank the shortlist. A common implementation is weighted Reciprocal Rank Fusion (RRF) : fusion score(d) = Σ [w i / (k + rank i(d))] Here, w i is the weight for retriever i, and k is a rank damping constant. A document absent from one list contributes zero from that list. Practical numerical example Suppose the system favors exact product IDs and assigns BM25 weight = 0.7 and vector weight = 0.3 . For easy arithmetic, let k = 1 (an illustrative setting). Chunk A: BM25 rank 1, vector rank 3. 0.7/(1+1) + 0.3/(1+3) = 0.425 Chunk B: BM25 rank 3, vector rank 1. 0.7/(1+3) + 0.3/(1+1) = 0.325 Chunk A ranks first because the BM25 branch has more weight. For production, tune weights and the damping constant using evaluation data. How is it different from other fusion methods? Unweighted RRF: Gives every retriever equal influence while combining rank positions. Rank based weighted fusion: Still uses ranks, but assigns different influence to different retrievers. Relative score fusion: Normalizes and combines original relevance scores, preserving differences in score magnitude. Rank based fusion can work even when BM25 and dense retrieval scores are incompatible, but it discards information about how far apart the raw scores were. When to use it Different retrievers contribute unequally to real user questions. Exact identifiers are especially important, or semantic matches deserve higher priority. You want interpretable control over fusion without calibrating raw score scales. Key precautions Aggressive weights can suppress valuable results from a weaker weighted source. Candidate list depth and the rank damping constant affect fusion behavior. Deduplicate by stable chunk ID and preserve source references. Enforce tenant and document permissions in each retrieval branch. Compare against unweighted RRF with Recall@K, nDCG@K, grounded answer quality, and latency. Interview takeaway Rank based weighted fusion merges retrieval rankings using per retriever weights, often through weighted RRF. It lets you favor specific search methods without comparing raw scores, but weights and ranking parameters should be tuned on real queries.

What is learned fusion or learning-to-rank in RAG?

Learned fusion , also called learning to rank (LTR) in this setting, uses a trained model to combine retrieval signals and rank document chunks for a RAG question. Unlike fixed weighted fusion, the model learns which signals predict relevance from labeled examples. How does it work? 1. Retrieve candidates: Use BM25, dense vector search, or hybrid retrieval to collect a shortlist of authorized chunks. 2. Extract features: For each query–chunk pair, collect signals such as BM25 score, vector similarity, exact identifier match, source authority, and document freshness. 3. Prepare relevance judgments: Label example query–chunk pairs as relevant or irrelevant, or use graded relevance (for example, 0–3). 4. Train a ranking model: Learn to order the candidates using the features and judgments. Gradient boosted decision trees, including LambdaMART style models, are a common choice. 5. Apply at query time: Compute the same features for new candidates, score them with the trained model, and pass the best ranked chunks to the LLM. Query → candidate retrieval → features → trained ranker → relevant context → answer Practical example An employee asks: “What is the current reimbursement limit for remote work equipment?” The retrievers return: An archived policy with strong BM25 and vector scores. An active HR policy with a slightly weaker text match. An informal discussion about equipment purchasing. If training examples consistently show that current, approved policies are the best answers for policy questions, the learned ranker can give the active HR passage greater importance. It learns these patterns from data rather than relying only on manually chosen weights. Learned fusion versus fixed fusion RRF: Combines positions in result lists using a fixed formula; no labeled training data is required. Weighted score fusion: Combines calibrated retriever scores using chosen weights. Learned fusion / LTR: Learns a ranking function from examples, potentially capturing interactions among semantic scores, keyword matches, and document features. LTR is often implemented as second stage reranking . The broader term learning to rank also covers other learned ranking pipelines; it is not limited to fusing retrievers. When should you use it? Consider LTR when you have sufficient high quality relevance judgments, multiple useful ranking signals, and a measurable need for more customized ranking than RRF or simple reranking provides. Main limitations Training data quality: Poor, biased, or stale labels can teach incorrect priorities. Feature consistency: Training and production must compute features the same way. Added operational cost: Training, monitoring, retraining, and feature computation create maintenance overhead. Candidate recall: No learned reranker can recover a relevant document omitted by all first stage retrievers. Security: Apply tenant and document authorization before feature extraction or sharing passages with downstream models. Evaluate using held out queries, nDCG@K , MRR, evidence Recall@K, grounded answer quality, and latency against RRF or other simple baselines. Interview takeaway Learned fusion trains a ranking model on relevance labeled examples and signals such as BM25, vector similarity, and metadata. It can adapt ranking beyond fixed formulas, but needs reliable labels, consistent features, and measured production gains.

How should results from multiple retrievers be deduplicated?

Deduplication in multi retriever RAG means combining results from lexical, vector, graph, or other search methods without repeating the same source passage. The objective is to preserve useful evidence and ranking signals while avoiding redundant context for the LLM. How should it work? 1. Collect candidates: Run the authorized retrievers and collect each chunk's ID, document ID, score, rank, and source metadata. 2. Use stable identities: When the same indexed chunk appears in multiple lists, merge it using a canonical key such as tenant id + document id + document version + chunk id. 3. Keep retrieval signals: Record which retrievers returned the chunk and its score or rank in each list. Those signals can inform Reciprocal Rank Fusion (RRF) or calibrated score fusion. 4. Detect near duplicates carefully: For different chunk IDs containing substantially overlapping text, compare normalized text, content hashes, or similarity. Keep important differences such as exceptions, dates, and document versions. 5. Fuse and rerank: Combine eligible candidates, rerank the unique evidence, and select passages within the context budget. 6. Preserve provenance: Retain source IDs, citations, version information, and access restrictions in the merged record. Retriever A + Retriever B → canonical IDs → merge duplicates → fusion/reranking → unique evidence Practical example For “How do I fix PAY 409?”, BM25 returns chunks A, B, C and vector search returns B, D, A . The combined list should be A, B, C, D —not six separate chunks. Both A and B retain evidence that two retrievers selected them, so an RRF based fusion stage can reward their rankings appropriately. Exact versus near duplicate content Exact result duplicates: The same tenant, source version, and chunk ID. Merge deterministically. Duplicate text from different sources: Do not automatically erase provenance. Multiple sources may matter for authority or citation. Near duplicates: Similar passages or overlapping chunks. Consider context level redundancy removal, but avoid deleting unique conditions or facts. Common pitfalls Deduplicating solely by plain text can collapse different versions, sources, or security scopes. Removing duplicate list entries before recording their retriever ranks loses useful fusion information. Summing raw BM25 and vector scores ignores incompatible scales. Running similarity based deduplication across tenants can cross access boundaries; filter by authorization first. Aggressive similarity thresholds may drop the only passage containing a critical exception. Interview takeaway Deduplicate exact multi retriever hits by stable source and chunk identity, preserve each retriever's ranking signals for fusion, and remove near duplicate context cautiously. Always retain permissions, version information, and citation provenance.

What is context construction in RAG?

Context construction in RAG is the step that turns retrieved document chunks into a clear, compact, source attributed evidence package for the language model. It happens after retrieval and optional reranking , before answer generation. How does it work? 1. Collect eligible evidence: Start with chunks returned by vector, keyword, hybrid, or graph retrieval. Enforce user permissions before any text reaches the generator. 2. Filter and deduplicate: Remove irrelevant, stale, repeated, or superseded chunks while keeping unique facts and exceptions. 3. Select under a token budget: Choose enough evidence to address the question while leaving room for instructions, conversation history, and the answer. 4. Organize passages: Group related chunks or arrange them in a useful order; preserve section headings where needed for interpretation. 5. Attach provenance: Keep source title, document ID/version, page or section reference, and citation ID for each included passage. 6. Assemble the model input: Separate the user's question and trusted instructions from retrieved source text, which must be treated as untrusted evidence , not instructions. Query → retrieved candidates → filtering and selection → attributed context → LLM answer Practical example An employee asks: “Can I carry unused leave into next year?” Retrieval returns: A current HR policy stating the carry forward limit. A policy exception for employees in a particular country. An old policy with a different limit. Two overlapping copies of the current policy. Context construction keeps the current applicable rule and any relevant exception , drops obsolete and duplicate passages, and includes citation labels. The LLM can then answer using that concise evidence rather than a noisy bundle. What does a context package look like? For example: [Source 1 | Leave Policy v2026 | Section 4] Unused leave may be carried forward up to the applicable limit... [Source 2 | Regional Exception | Section 2] For employees in Region X, a different limit applies... The system includes the user's question separately. Source text is quoted or delimited as evidence; it is not allowed to override the application's instructions. Why does construction matter? Relevance: Irrelevant chunks can distract the generator. Coverage: Overaggressive trimming may exclude a crucial exception. Cost and latency: Longer prompts consume more tokens and can slow generation. Grounding: Clear source attribution helps the model cite support for individual claims. Security and robustness: Access controls and prompt injection defenses must be enforced by trusted application logic. Common pitfalls Passing raw Top K results without checking redundancy or applicability. Losing titles, tables, headings, or source citations that change a passage's meaning. Mixing old and current document versions without labeling them. Treating retrieved text as trusted commands. Assuming longer context always improves answer correctness. Interview takeaway Context construction selects, cleans, orders, and attributes retrieved evidence before generation. A good RAG context is relevant, sufficiently complete, within budget, permission safe, and clearly separated from trusted instructions.