How many candidates should be retrieved before reranking?

Retrieve enough candidates to give the reranker a good chance of seeing the correct evidence, but not so many that reranking becomes unnecessarily slow or expensive. There is no universal candidate count.

Practical starting point

Try first-stage retrieval depths such as Top-10, Top-20, and Top-50 on a representative evaluation set. These are experimental settings, not fixed best practices. The right value depends on retriever recall, query complexity, reranker cost, and latency constraints.

How to choose the candidate count

1. Measure first-stage recall: With labeled questions, calculate whether an answer-bearing chunk appears in the retrieved pool (for example, Recall@20 versus Recall@50).
2. Test the reranker: Feed each candidate pool into the same reranker and measure how often relevant passages reach the final context (such as the best five chunks).
3. Check latency and cost: More candidates require additional query–passage scoring. Inspect p95 latency and infrastructure or API costs.
4. Consider query types: Exact factual questions may need fewer candidates than ambiguous comparisons or multi-document requests.
5. Deduplicate and filter: Remove duplicates and enforce access restrictions before spending reranker capacity on candidates.
6. Pick the smallest effective pool: Increase the depth only while it meaningfully improves evidence coverage or grounded answer quality.

Example

An HR assistant tests an initial retrieval pool of 10, 20, and 50 policy chunks before sending the top five reranked chunks to its LLM.

• Top-10: Occasionally misses country-specific exceptions entirely.
• Top-20: Includes those exceptions in most evaluation questions.
• Top-50: Adds more reranking work with little additional gain.

In this *illustrative* scenario, Top-20 is a reasonable choice. The result must be verified against real data.

Candidate count versus final context

These are different settings:

Retrieve Top-20 → rerank 20 → send Top-5 to the LLM

The first number controls potential evidence coverage. The final number controls the context selection and prompt-token budget.

A reranker cannot recover a correct chunk that initial retrieval omitted. If recall is poor, improve retrieval, chunking, hybrid search, or query transformation rather than simply increasing reranking complexity.

Operational considerations

• Cross-encoder inference usually grows with the number and length of candidate passages, although batching may improve throughput.
• LLM listwise reranking can become expensive or exceed input limits as the pool expands.
• Very large candidate sets may contain near-duplicates that crowd out diverse evidence.
• Different question categories may justify different candidate depths, provided the routing logic is evaluated.

Interview takeaway

Tune reranking candidate depth empirically: retrieve enough chunks to achieve strong first-stage evidence recall, then select a smaller high-quality context after reranking. Optimize Recall@N, final ranking quality, p95 latency, and cost together.