What is multi-stage retrieval?

Multi-stage retrieval in RAG is a search pipeline that uses two or more successive retrieval or ranking stages to narrow a broad candidate set into a small set of high-quality evidence. Each stage can use a different algorithm and cost budget.

Beginner-friendly example

Suppose a support knowledge base contains 100,000 document chunks. A user asks: “How can I rotate an expired API key without downtime?”

A production pipeline might:

1. Fast candidate search: Use BM25 and/or vector search to retrieve 100 potentially useful chunks.
2. Reranking: Use a more accurate but slower cross-encoder to reorder those 100 and retain the top 10.
3. Evidence validation: Remove duplicates, obsolete instructions, and results lacking relevant permissions; pass the best 5 supported passages to the language model.
4. Generation: Answer using those passages and cite their originating documents.

The numbers are illustrative, not universal settings.

Why use multiple stages?

A fast initial retriever can cheaply inspect a large collection, but semantic similarity or keyword overlap alone may rank a weak passage above the right answer. A slower reranker can examine query–passage interactions more carefully once the candidate set is small.

Common stages

• Candidate generation: Keyword search, dense vectors, hybrid search, or structured filtering.
• Candidate merging: Deduplicate overlapping results and combine source rankings if there are multiple retrievers.
• Reranking: Cross-encoders or other relevance models evaluate candidate evidence more thoroughly.
• Context selection: Apply source freshness, permissions, diversity, token limits, and sometimes claim-support checks.
• Generation: Pass the final grounded context to the LLM. The generator is downstream of retrieval, not a retrieval stage itself.

Multi-stage vs related concepts

• Single-stage retrieval: One search retrieves the final candidates directly.
• Hierarchical retrieval: Searches document or topic *levels* (document → section → passage); that is one possible multi-stage design.
• Parent-document retrieval: Retrieves small child chunks, then fetches their larger parent sections to give the generator more context.
• Hybrid retrieval: Combines multiple search techniques, possibly in the *same stage*. Hybrid search does not automatically mean multi-stage retrieval.

Practical trade-offs

Benefits

• Improves precision at the top of the ranked list when reranking works well.
• Reduces the number of items exposed to expensive models.
• Makes it easier to enforce evidence and context budgets.

Risks

• Lost recall: A relevant passage excluded by the first stage cannot be recovered by reranking.
• Latency: Extra ranking and validation calls take time.
• Cost: Complex rerankers and additional APIs consume resources.

How to evaluate it

Measure candidate recall@K before reranking, nDCG or MRR after reranking, grounded answer quality, total p95 latency, and per-query cost. Compare with a simpler baseline and inspect failures at every stage.

Interview takeaway: Multi-stage retrieval uses fast broad search → slower accurate ranking → evidence selection to balance coverage, precision, latency, and cost.