Better retrieval but more tokens and more latency?

Better retrieval often means retrieving more candidates, using stronger ranking methods, or adding extra retrieval stages.

That can improve answer quality, but it usually increases token usage and latency.

The key is to separate candidate retrieval from final context sent to the LLM.

Why better retrieval can cost more

Suppose you increase Top-K from 5 to 30.

You may improve recall because the correct chunk has a better chance of appearing.

But now you have:

• More chunks to score
• More candidates to rerank
• More text to potentially send to the LLM
• More input tokens
• Higher latency and cost

So retrieval quality can improve while system efficiency gets worse.

Do not send every retrieved chunk to the LLM

A strong pattern is:

Retrieve many candidates → rerank → remove duplicates → keep only the strongest few → send those to the LLM

For example:

Retrieve Top 30 → rerank Top 30 → keep Top 5 → generate answer

This gives the retrieval stage room to find the correct evidence without forcing the LLM to read all 30 chunks.

Why more context can hurt

More context is not automatically better.

Extra chunks may contain:

• Repeated information
• Weakly related content
• Conflicting versions
• Irrelevant details

That can distract the model and make the answer less accurate.

So the goal is not maximum context.

The goal is maximum useful evidence per token.

Where does latency come from?

Latency may increase at several stages:

• Running multiple retrievers
• Searching with a larger Top-K
• Reranking more candidates
• Processing a larger LLM prompt
• Generating from a longer context

Sometimes the retrieval itself is fast, but the larger prompt becomes the main source of latency.

How do you balance it?

Measure the system end to end.

Compare configurations such as:

Top 10 → rerank → keep 3

Top 30 → rerank → keep 5

Top 50 → rerank → keep 8

Then evaluate:

• Recall@K
• Final answer accuracy
• Input token count
• Retrieval latency
• Reranking latency
• Total response time
• Cost

Stop increasing retrieval depth when the quality gain becomes small compared with the extra latency and cost.

Interview takeaway

Better retrieval can require a larger candidate set, hybrid search, or reranking, which increases latency and computation. But the LLM should not receive every retrieved chunk. A common production strategy is to retrieve broadly for recall, rerank aggressively, and pass only a small high-quality context to the model so retrieval improves without unnecessarily increasing tokens and generation latency.