What are the trade-offs between retrieval accuracy and latency?

Retrieval accuracy and latency usually move in opposite directions.

The more work you do to find the best evidence, the better retrieval can become—but the query often takes longer.

The goal is not maximum accuracy at any cost. It is the best accuracy you can achieve within your latency budget.

1. Increasing Top-K can improve recall

If you retrieve more candidates, you reduce the chance of missing the correct chunk.

For example:

Top 3 → faster, but may miss useful evidence

Top 20 → better recall, but more retrieval work and more candidates to process

A larger Top-K can therefore improve recall, but it also increases downstream cost.

2. Hybrid retrieval adds extra work

Dense retrieval alone may be fast enough.

Hybrid retrieval runs:

• Vector search
• BM25 or sparse search
• Fusion

This can improve robustness, especially for exact terms and semantic matches, but it adds another retrieval path and a fusion step.

3. Reranking improves precision but adds latency

A reranker examines a smaller candidate set more deeply.

Example:

Retrieve 50 candidates → rerank 50 → keep best 5

This often improves final ranking quality, but cross-encoders or LLM-based rerankers are slower than basic vector similarity.

The more candidates you rerank, the greater the latency.

4. Exact search vs approximate search

Large vector databases often use approximate nearest-neighbor indexes.

Approximate search is much faster than comparing the query against every vector.

But aggressive speed settings can slightly reduce recall.

You typically tune the index to balance:

Speed ↔ Recall

5. More context can increase end-to-end latency

Retrieval latency is only part of the story.

If better retrieval returns many large chunks, the LLM receives more tokens.

That can increase:

• Prompt-processing time
• Generation latency
• API cost
• Noise in the final context

So retrieving more is not automatically better.

6. Filtering can improve both speed and precision

Good metadata filtering can reduce the number of candidates before vector search.

Example:

Search only:

tenant = A country = India status = active

instead of searching the full knowledge base.

This can reduce latency while also preventing irrelevant results.

7. Use stages instead of one expensive search

A common production pattern is:

Fast first-stage retrieval → moderate candidate set → rerank only those candidates → send a small final context to the LLM

This is usually more efficient than applying an expensive relevance model to the entire corpus.

How do you choose the balance?

Measure both retrieval quality and latency on realistic queries.

For example, compare:

Top 5 vs Top 20

Dense vs hybrid

No reranker vs reranking Top 20

Then track:

• Recall@K
• MRR or NDCG
• Retrieval latency
• End-to-end response time
• Token usage

Choose the simplest configuration that meets the required quality.

Interview takeaway

Higher retrieval accuracy often requires more candidates, multiple retrievers, reranking, or deeper search, which increases latency and cost. The practical solution is to retrieve broadly enough for good recall, narrow candidates with filtering and reranking, and measure both retrieval metrics and end-to-end latency rather than optimizing either one in isolation.