How does dense retrieval work in RAG?

Dense retrieval is a method of finding relevant documents by comparing learned vector representations of a user query and stored document chunks. Unlike keyword matching, it can retrieve passages expressed in different words but with similar meaning.

How dense retrieval works

1. Prepare the documents: Parse the knowledge base, split documents into chunks, and retain source identifiers and relevant metadata.
2. Embed the chunks: Pass each chunk through a document embedding model to produce a dense numerical vector. Store the vector alongside its chunk ID and metadata.
3. Embed the query: When someone asks a question, a compatible query encoder creates a query vector.
4. Search by vector similarity: Compare the query embedding with indexed vectors, typically using cosine similarity, dot product, or a compatible distance metric. Large collections often use approximate nearest-neighbor (ANN) indexes to avoid exhaustive comparisons.
5. Return Top-K candidates: Retrieve the highest-scoring permitted chunks and their source text.
6. Build the RAG context: Optionally apply metadata filters, rerank candidates, and select evidence for the language model to answer with.

Documents → embeddings → vector index

User question → query embedding → similarity search → relevant chunks → LLM

Example

A policy document states: “Staff may perform their duties off-site on three weekdays.”

The user asks: “How many days can employees work from home?”

Dense retrieval can surface the passage even though it does not literally contain the phrase “work from home.” The embedding model may represent the two expressions as semantically related. The LLM then uses the retrieved policy statement to formulate an answer.

Why is it called dense?

A dense embedding usually has many dimensions, and most values are nonzero. These learned numerical features encode useful relationships between phrases and concepts.

This contrasts with sparse retrieval, where the representation emphasizes a relatively small number of vocabulary features and methods such as BM25 rely on term matching.

Dense retrieval versus the generator

The dense retriever selects evidence; it does not produce the final answer. The generator reads the selected evidence and composes a response. If the retriever misses the correct passage, the generator may lack what it needs to answer accurately.

Key limitations

• Exact identifiers: Product codes, error strings, names, and rare terms may be better matched by keyword search.
• Embedding quality: Domain-specific terminology or unusual language can be poorly represented.
• Approximate indexing: ANN search can miss neighbors in exchange for lower latency.
• Chunking and filtering: Poor chunk boundaries or incorrect metadata filters can hide relevant evidence.
• Semantic similarity is not factual relevance: A passage can be topically close without answering the question.

For these reasons, production RAG often combines dense retrieval with sparse search, metadata constraints, and reranking. Evaluate with questions that have known answer-bearing source passages, using metrics such as Recall@K.

Interview takeaway

Dense retrieval embeds both document chunks and user queries into compatible vector spaces, finds semantically similar chunks through vector search, and supplies those chunks as context for generation. It handles paraphrasing well, but requires careful evaluation and may benefit from hybrid retrieval when exact wording matters.