Retrieval-Augmented Generation
Retrieval-augmented generation connects a generative model to a searchable knowledge source. Instead of relying only on model weights, the system fetches relevant documents and places them into the prompt or context.
How do you update or delete indexed documents in a RAG vector database?
When a source document changes or is deleted, its indexed chunks, embeddings, and metadata must be synchronized so the RAG system does not retrieve outdated or removed information. Updating a document A reliable update workflow is: 1. Identify the source document using a stable document id, along with its tenant or namespace. 2. Detect the change using a version number, content hash, file event, or updated at timestamp. 3. Parse and re chunk the latest document. If the content changed, regenerate embeddings for affected chunks using the currently configured embedding model. 4. Upsert updated records into the vector database using stable chunk IDs where possible. An upsert inserts missing records or updates existing ones. 5. Remove obsolete chunks that no longer exist, especially if editing changed the number or boundaries of chunks. 6. Verify retrieval returns the new content and no longer surfaces the old version. If chunk boundaries change significantly, replacing all indexed chunks belonging to the document may be simpler and safer than guessing which old chunks correspond to new ones. Example: updating a policy PDF Suppose leave policy.pdf originally produces five chunks. After revision, it produces four. Reprocess the updated PDF and create four new embeddings. Upsert the four current records, preserving document id and adding a new version. Delete the fifth old record, plus any other obsolete chunk IDs. Query a changed policy statement to confirm that only the latest version is available. Important: Updating four chunks does not automatically remove the old fifth chunk. That needs explicit cleanup or a document version replacement strategy. Deleting a document Deleting a source file does not necessarily delete vectors that were created from it. To remove a document from retrieval: 1. Identify all chunk IDs associated with its document id and the correct tenant or collection. 2. Delete those vectors by ID, or use a supported metadata filter delete operation. 3. Remove or invalidate separately stored chunk text, cached retrieval results, and related references as required. 4. Verify that queries no longer return the deleted content. Different databases expose different update/delete APIs. For example, some support deletion by record ID and by metadata filter; others require tracking the affected IDs. Avoid stale or partially updated results For production RAG, consider: Versioning: Index a new document version before switching a version filter or alias to make it active. Consistency: Some vector services apply writes asynchronously, so verify the change after it becomes visible. Idempotency: Design ingestion jobs so retrying an upsert or deletion does not produce duplicate records. Failure recovery: Track job status and retry partial failures rather than leaving old and new versions mixed. Access control: Scope update and delete operations to the verified tenant and authorized documents. Auditability: Record which document version was indexed, when, and by which ingestion job. Interview takeaway To update an indexed document, reprocess its changed content, regenerate the necessary embeddings, upsert current chunk records, and delete obsolete chunks. To delete a document, remove every associated vector and invalidate any related cached or stored content. Stable IDs, document versions, tenant scoping, and post update verification prevent stale retrieval results.
What is semantic search in RAG?
Semantic search retrieves information based on the meaning of a query rather than requiring documents to contain exactly the same words. In RAG, it helps identify relevant knowledge base chunks to send to the language model as evidence. How does it work? 1. Index documents: Split source documents into meaningful chunks. 2. Create embeddings: Convert each chunk into a vector using an embedding model. 3. Embed the question: Convert the user's query into a compatible query vector. 4. Find neighbors: Compare the query vector with indexed vectors using a metric such as cosine similarity or dot product. 5. Retrieve context: Return the most relevant candidate chunks, optionally filter and rerank them, then pass selected evidence to the generator. The document and query embeddings must share a compatible representation space; otherwise similarity scores are not meaningful. Example A company policy says: "Employees may work remotely up to three days per week." A user asks: "How often am I allowed to work from home?" A keyword only search may struggle if it relies heavily on literal overlap. Semantic search can retrieve the policy because working from home and working remotely express closely related ideas. The RAG system then supplies the original policy text to the LLM, which can answer using that evidence. Semantic search versus keyword search Semantic search: Strong at matching paraphrases, related concepts, and questions expressed in different wording. Keyword search (for example, BM25): Often better for exact identifiers, error codes, product names, and rare technical terms. Hybrid search: Combines semantic and keyword results when neither approach alone is reliable enough. Semantic similarity is not proof that a chunk answers the question. A close vector neighbor may still be irrelevant, outdated, or contradictory. What affects search quality? Embedding model: Does it represent the domain's vocabulary and intent accurately? Chunking: Are answer bearing facts kept together with enough context? Similarity metric and index: Are vectors compared consistently, and does approximate search retrieve strong candidates? Metadata filters: Does search stay within the correct documents, language, version, and authorized tenant? Reranking: Can a second stage improve the ordering of candidates? Evaluation: Do relevant evidence chunks appear in Top K across representative questions? When is semantic search useful in RAG? Use it especially when users naturally paraphrase documents, ask conceptual questions, or use different terms than the stored text. Consider hybrid retrieval when precise literal matches are just as important. Interview takeaway Semantic search in RAG embeds both document chunks and user queries into a compatible vector space, then retrieves chunks with similar meaning. It makes retrieval robust to paraphrasing, but it should be combined with appropriate filtering, evaluation, and sometimes keyword search or reranking to deliver reliable evidence.
How does dense retrieval work in RAG?
Dense retrieval is a method of finding relevant documents by comparing learned vector representations of a user query and stored document chunks. Unlike keyword matching, it can retrieve passages expressed in different words but with similar meaning. How dense retrieval works 1. Prepare the documents: Parse the knowledge base, split documents into chunks, and retain source identifiers and relevant metadata. 2. Embed the chunks: Pass each chunk through a document embedding model to produce a dense numerical vector. Store the vector alongside its chunk ID and metadata. 3. Embed the query: When someone asks a question, a compatible query encoder creates a query vector. 4. Search by vector similarity: Compare the query embedding with indexed vectors, typically using cosine similarity, dot product, or a compatible distance metric. Large collections often use approximate nearest neighbor (ANN) indexes to avoid exhaustive comparisons. 5. Return Top K candidates: Retrieve the highest scoring permitted chunks and their source text. 6. Build the RAG context: Optionally apply metadata filters, rerank candidates, and select evidence for the language model to answer with. Documents → embeddings → vector index User question → query embedding → similarity search → relevant chunks → LLM Example A policy document states: “Staff may perform their duties off site on three weekdays.” The user asks: “How many days can employees work from home?” Dense retrieval can surface the passage even though it does not literally contain the phrase “work from home.” The embedding model may represent the two expressions as semantically related. The LLM then uses the retrieved policy statement to formulate an answer. Why is it called dense? A dense embedding usually has many dimensions, and most values are nonzero. These learned numerical features encode useful relationships between phrases and concepts. This contrasts with sparse retrieval , where the representation emphasizes a relatively small number of vocabulary features and methods such as BM25 rely on term matching. Dense retrieval versus the generator The dense retriever selects evidence ; it does not produce the final answer. The generator reads the selected evidence and composes a response. If the retriever misses the correct passage, the generator may lack what it needs to answer accurately. Key limitations Exact identifiers: Product codes, error strings, names, and rare terms may be better matched by keyword search. Embedding quality: Domain specific terminology or unusual language can be poorly represented. Approximate indexing: ANN search can miss neighbors in exchange for lower latency. Chunking and filtering: Poor chunk boundaries or incorrect metadata filters can hide relevant evidence. Semantic similarity is not factual relevance: A passage can be topically close without answering the question. For these reasons, production RAG often combines dense retrieval with sparse search, metadata constraints, and reranking. Evaluate with questions that have known answer bearing source passages, using metrics such as Recall@K. Interview takeaway Dense retrieval embeds both document chunks and user queries into compatible vector spaces, finds semantically similar chunks through vector search, and supplies those chunks as context for generation. It handles paraphrasing well, but requires careful evaluation and may benefit from hybrid retrieval when exact wording matters.
What is sparse retrieval in RAG?
Sparse retrieval finds relevant documents using representations in which most features have zero weight. In RAG, it commonly means lexical retrieval with an inverted index and a ranking method such as BM25 , although learned sparse retrievers also exist. How does sparse retrieval work? 1. Index documents: Split knowledge base documents into chunks, tokenize their text, and build an inverted index that maps terms to the chunks containing them. 2. Process the query: Extract searchable terms from the user’s question. 3. Find candidates: Use the inverted index to locate chunks matching those terms. 4. Rank candidates: Score them using a method such as BM25, which considers term frequency, term rarity, and document length normalization. 5. Supply evidence: Return the top ranked chunks to the RAG pipeline for optional reranking and answer generation. Documents → terms and inverted index Question → matching terms → ranked chunks → LLM Simple example A user asks: “What does error code E CONN 204 mean?” A support manual contains the exact string E CONN 204 and its troubleshooting instructions. Sparse lexical retrieval is often effective here because the precise identifier matters more than a broad semantic resemblance. Sparse versus dense retrieval Sparse lexical retrieval (for example, BM25): Excels at exact words, identifiers, product names, and specialized terminology; it may miss relevant passages written using different words. Dense retrieval: Compares learned embeddings to find semantic matches, including paraphrases; it can underperform on rare strings or exact codes. Hybrid retrieval: Combines both candidate sets, often improving coverage when questions require a mix of meaning and precise terms. Important: Sparse does not always mean simple keyword matching. Learned sparse models can assign weights to expanded terms and capture semantic associations while keeping mostly zero valued features. What affects retrieval quality? Document parsing and tokenization Case normalization, stemming, and treatment of special characters Chunk boundaries and preserved identifiers Ranking function and domain vocabulary Metadata filters, tenant permissions, and source freshness For RAG, evaluate whether the answer bearing chunk appears in Top K—not merely whether the query terms appear in the result. Interview takeaway Sparse retrieval represents text with mostly zero valued term features and retrieves chunks efficiently using inverted indexes. BM25 is a common lexical approach, especially useful for exact terminology and identifiers. Learned sparse models can also capture semantic associations, and hybrid retrieval combines sparse and dense strengths.
What is BM25 and why is it used in RAG?
BM25 (Best Matching 25) is a lexical ranking algorithm that scores how well a document or chunk matches the words in a search query. In RAG, it is commonly used for sparse retrieval , especially when exact terminology matters. How does BM25 rank chunks? BM25 considers three main signals: Term frequency: A query term appearing more often can increase relevance, but with diminishing returns. Repeating a word many times does not increase its score indefinitely. Inverse document frequency (IDF): Terms that occur in fewer documents generally carry more weight than common terms. Document length normalization: BM25 accounts for chunk length so longer passages do not automatically score higher simply because they contain more words. Two familiar tuning parameters are k1 , which controls term frequency saturation, and b , which controls the strength of length normalization. Example in RAG A user asks: "How do I fix error E CONN 204?" A support document contains E CONN 204 and the exact troubleshooting steps. BM25 can rank that chunk highly because the rare error identifier matches directly. A dense embedding retriever might instead favor a generally related network troubleshooting passage. The RAG pipeline then passes the retrieved source chunk to the LLM to answer the question. Where BM25 fits 1. Parse and chunk source documents. 2. Build a searchable lexical index for those chunks. 3. Tokenize the user's query and find matching chunks using an inverted index. 4. Use BM25 scores to rank candidates. 5. Retrieve the best chunks and optionally combine them with dense search results or rerank them before generation. BM25 retrieves evidence; the LLM generates the answer. Why use BM25 in RAG? Exact matches: Useful for error codes, IDs, product names, technical terms, and uncommon keywords. No embedding required: Lexical scoring works without a neural embedding model. Efficient retrieval: Inverted indexes can narrow candidate documents quickly. Complementary signals: BM25 and dense retrieval often find different useful passages, making them a strong combination for hybrid search. Limitations BM25 primarily relies on query terms and text analysis. A passage may be highly relevant but use different wording. For example, a question about working from home may not match a policy that uses only remote work as strongly as a good dense retriever would. Tokenization, stopwords, stemming, chunk length, and indexing configuration also affect results. BM25 scores are not automatically comparable to vector similarity scores, so hybrid systems usually normalize or fuse ranked results rather than adding raw scores blindly. Interview takeaway BM25 is a term based ranking algorithm that combines term frequency, term rarity, and document length normalization. In RAG, it provides fast sparse retrieval and excels at exact terms and identifiers; combining it with dense retrieval can improve coverage across both lexical and semantic queries.
What is keyword search in RAG?
Keyword search in RAG retrieves document chunks based on the words or terms appearing in a user's question. It is a lexical retrieval technique: rather than converting every query into a semantic vector, it looks for matching indexed terms and ranks the matching chunks. How does it work? 1. Index the documents: Parse source material into chunks and build a searchable index of their words, typically an inverted index mapping terms to chunk IDs. 2. Process the question: Analyze the query text, often including tokenization and optional lowercasing, stemming, or synonym handling. 3. Find matching chunks: Look up chunks containing query terms or supported phrase matches. 4. Rank the matches: Use a lexical scoring method such as BM25 to favor informative matches, taking term rarity, frequency, and chunk length into account. 5. Pass evidence to RAG: Return the strongest matching chunks, optionally combine or rerank them, and supply the resulting context to the language model. Example A user asks: “What causes payment error PAY 409?” The support guide includes a section titled “PAY 409: Duplicate transaction.” Keyword search can retrieve that section by matching the precise error identifier. This is particularly valuable because a semantic embedding may not reliably distinguish similar looking error codes. When is keyword search useful? Exact identifiers: Error codes, invoice IDs, product SKUs, API endpoint names, or configuration keys. Rare technical terminology: Terms that must match closely. Named entities: Specific product names, legal clauses, or feature labels. Auditable retrieval: It can be easier to explain which search terms caused a chunk to rank highly. Keyword search versus semantic search Keyword search: Uses lexical matches and works especially well when the precise wording matters. Semantic search: Uses embeddings to retrieve conceptually similar text even when the words differ. Hybrid retrieval: Combines lexical and semantic candidate sets to benefit from both approaches. Keyword search can miss relevant passages that use synonyms or paraphrases. For example, a query about “work from home” may not strongly match a document that only says “remote employment.” Semantic retrieval can help bridge that wording gap. Is keyword search the same as BM25? Not exactly. Keyword search is the broader retrieval approach; BM25 is one ranking algorithm used within it. Other keyword search implementations may use Boolean matching, phrase searches, or other lexical scoring methods. Practical tips for RAG Preserve codes, punctuation sensitive identifiers, and important product terms during text processing. Tune analyzers and field boosts to suit the document types. Apply metadata and authorized access restrictions before returning chunks. Measure whether answer bearing evidence appears in Top K, especially for identifier heavy queries. Interview takeaway Keyword search retrieves RAG evidence by matching words or phrases in an indexed document collection. It is reliable for exact terms and identifiers, while semantic search handles paraphrases. In practice, many RAG systems combine both using hybrid retrieval.
What is hybrid search in RAG?
Hybrid search in RAG combines two complementary retrieval methods— lexical (sparse) search , often BM25, and semantic (dense) vector search —to find more useful evidence than either method may find alone. Keyword/BM25 search finds exact terms, codes, names, and uncommon terminology. Dense vector search finds conceptually similar passages, including paraphrases that use different wording. How does hybrid search work? 1. Index documents twice: Store searchable text in a lexical index and embeddings in a vector index. Keep a stable chunk ID to connect results. 2. Run both searches: Send the user's query to BM25 and embed it for dense vector search. 3. Collect candidates: Each retriever returns a ranked list of matching chunks. 4. Merge and deduplicate: Combine results that refer to the same chunk and calculate a fused ranking. 5. Select evidence: Optionally rerank the merged candidates with a stronger model, then pass the best chunks to the LLM. User query → BM25 + dense retrieval → fusion → optional reranking → grounded answer Practical example A user asks: “How do I resolve error PAY 409 when a payment is submitted twice?” BM25 is likely to recognize the exact identifier PAY 409 . Dense retrieval can surface a guide discussing duplicate payment submissions even if it uses different words. Hybrid search combines both evidence sources so the RAG assistant can answer with the most relevant documentation. How are the results combined? Because BM25 scores and vector similarity scores use different scales, they should not simply be added without adjustment. Two widely used approaches are: Reciprocal Rank Fusion (RRF): Combines results based on each chunk's position in the ranked lists. It does not require directly comparable raw scores. Weighted score fusion: Normalizes scores or otherwise calibrates them before combining, then applies weights to favor lexical or semantic relevance. An optional reranker can further reorder the merged shortlist based on the full query and document text. When is hybrid search most useful? Enterprise documents with both natural language descriptions and precise identifiers. Technical support knowledge bases containing error codes and troubleshooting explanations. Legal, medical, or product documents where terms matter but users may paraphrase their questions. RAG systems where either dense only or keyword only retrieval misses relevant evidence. Trade offs and best practices Running two retrieval methods increases compute, storage, and sometimes latency; parallel searches can reduce the latency penalty. Fusion quality depends on candidate counts, deduplication, and scoring strategy. Apply trusted tenant and permission constraints consistently to both retrievers. Measure Recall@K, retrieval latency, and downstream answer quality using real questions; hybrid retrieval is useful but not guaranteed to outperform either method for every workload. Interview takeaway Hybrid search combines exact term retrieval (BM25 or another sparse method) with semantic embedding search. It merges their candidates using methods such as RRF or calibrated weighted scores and may rerank them before generation. The main benefit is better coverage of both precise keywords and paraphrased meaning.
Why combine semantic search with keyword search?
Combining semantic search with keyword search helps a RAG system retrieve evidence that matches both the meaning of a question and its exact terms . This combination is often called hybrid retrieval . Why is one method not always enough? Semantic (dense) search uses embeddings to identify related ideas even when wording differs. It is useful for paraphrases but may overlook unusual identifiers, exact codes, or names. Keyword (lexical) search uses term matching, often ranked with BM25. It is strong at exact phrases and rare terms but can miss relevant passages that use synonyms. Together, they cover different failure cases. Practical example Suppose a user asks: “Why does API PAY 409 fail when I submit a payment twice?” Keyword search can find a troubleshooting guide containing the precise code PAY 409. Semantic search can find another passage explaining duplicate transaction requests , even if it never uses the same words as the question. Combining candidates gives the RAG assistant a better opportunity to retrieve the explanation and the exact reference. How are the results combined? 1. Run keyword search and embedding based search, often in parallel. 2. Retrieve a shortlist of document chunks from each. 3. Deduplicate chunks returned by both methods. 4. Fuse rankings , for example using Reciprocal Rank Fusion (RRF), which uses rank positions rather than assuming BM25 and vector similarity scores are comparable. 5. Optionally rerank the combined candidates against the full question. 6. Pass the strongest authorized, relevant chunks to the generator. Question → lexical + semantic retrieval → fusion → reranking → answer What improves in practice? Coverage: Better chance of finding answer bearing chunks across different writing styles. Robustness: Less dependence on one retriever's weaknesses. Technical accuracy: Exact identifiers stay discoverable alongside conceptually related explanations. Hybrid search is not guaranteed to be better for every query. It can require two indexes, more computation, and careful tuning. Validate the gain with labeled questions, Recall@K , p95 retrieval latency, and downstream grounded answer quality. Interview takeaway Combine keyword search with semantic search because BM25 style retrieval captures exact terms while embeddings capture meaning and paraphrases. Fuse and deduplicate their results, then optionally rerank. The aim is more reliable evidence retrieval—not simply a larger set of chunks.
What is Top-K retrieval?
Top K retrieval is the process of returning the K highest ranked document chunks for a user's query. In RAG, it controls how many search candidates are selected for further processing. Here, K is a number chosen by the application, such as 3, 5, or 10. How does Top K retrieval work? 1. Convert the question into a search query (and an embedding if using dense retrieval). 2. Search the document index using vector similarity, BM25, or hybrid retrieval. 3. Rank eligible chunks by the retriever's score. 4. Return the first K results, along with text, source IDs, and useful metadata. 5. Optionally rerank or filter these results before constructing the context sent to the LLM. Question → retrieval scores → ranked chunks → Top K candidates → LLM context Example Suppose a knowledge base has 50,000 chunks, and a user asks, “What is our refund policy?” If K = 5, the retriever returns up to five highest ranked eligible chunks. They might include the refund timeframe, exceptions, and return conditions. Important: Five retrieved chunks do not necessarily mean all five must be passed to the LLM. A later reranker or context selection step may keep fewer chunks. What happens when K changes? Smaller K (such as 3): Less context to process, but a higher chance that useful evidence is missed. Larger K (such as 20): More chances to include relevant evidence, but potentially more noise, reranking work, tokens, and latency. Increasing K can improve retrieval coverage, but it does not automatically improve answer quality. Irrelevant passages may distract the generator. Retrieval Top K versus final context count These numbers can differ. A system might: Retrieve Top 20 initial candidates. Rerank them using a cross encoder. Pass only the best Top 5 chunks to the LLM. It is helpful to distinguish candidate retrieval depth , reranking depth , and the final context budget . How do you choose K? Start with representative questions and evaluate several settings. Measure: Recall@K / Hit Rate: Is the answer bearing evidence present? Latency: How long does retrieval and optional reranking take? Context size and cost: How many tokens reach the LLM? Answer groundedness: Do responses stay supported by the retrieved passages? The best K depends on chunk size, document complexity, retriever quality, and the application. It should be tuned experimentally rather than selected as a universal constant. Interview takeaway Top K retrieval returns the K highest ranked candidate chunks for a query. Small K is cheaper but can miss evidence; large K improves candidate coverage at the cost of more noise and processing. Production RAG systems often retrieve more candidates than they finally send to the generator.
How do you choose Top-K in RAG?
Choose Top K in RAG by testing how many retrieved chunks are needed to find the right evidence without adding unnecessary latency, tokens, or irrelevant context. There is no universal best K. What does K control? Top K is the number of highest ranked results requested from a retriever. For example, K = 5 requests up to five candidate chunks. The number ultimately sent to the LLM can be different after filtering, deduplication, or reranking. How to choose K 1. Start with a small baseline: Try values such as K = 3, 5, and 10 on representative user questions. These are test settings, not fixed recommendations. 2. Measure evidence coverage: Use labeled query–document pairs to calculate Recall@K or Hit Rate. Check whether the answer bearing chunks are included. 3. Check final answer quality: Measure groundedness and answer relevance. More retrieved chunks do not necessarily produce better responses. 4. Track latency and cost: Larger K can increase reranking work, prompt tokens, and generation cost. Also observe p95 latency rather than just averages. 5. Account for chunk size: Five short chunks and five long chunks may consume very different context budgets. 6. Tune by query type: A straightforward factual question may need one or two supporting passages; comparison or multi hop questions may need evidence from multiple sources. 7. Keep access filters consistent: Only authorized chunks should be eligible, regardless of K. Practical example Suppose an HR RAG assistant answers questions using policy documents. With K = 3 , the leave policy definition is retrieved but an exception clause is often missed. With K = 8 , both definition and exception usually appear. With K = 20 , coverage may improve little while irrelevant policies add noise and context cost. You would compare those settings on a held out evaluation set and choose the smallest K that meets the required evidence retrieval quality and latency budget. The numbers above are illustrative; actual results require measurement. Initial retrieval K versus final context K A common two stage pattern is: Retrieve Top 20 candidates cheaply. Rerank those candidates with a stronger model. Pass only the best Top 5 chunks, subject to a token budget. This separates candidate coverage from the amount of context given to the LLM . The appropriate values depend on measured quality and cost. Common mistakes Assuming larger K always improves answers. Evaluating only vector similarity instead of answer bearing evidence. Ignoring near duplicate chunks, which can crowd out different relevant sources. Setting K without considering chunk length and context window limits. Choosing K from one example rather than a representative dataset. Interview takeaway Pick Top K empirically. Compare several values using evidence Recall@K, answer quality, p95 latency, and token cost. Use a larger candidate pool when reranking is available, but send only the strongest, nonredundant evidence to the generator.