How many retrieved chunks should be passed to the generator?

Pass only as many retrieved chunks to the RAG generator as are needed to support a complete, grounded answer within the prompt budget. There is no universal count: chunk length, question complexity, source diversity, and model behavior all matter.

Retrieval count is not generation count

The retriever may return Top-20 candidates, a reranker may identify the best eight, and the context builder may send only four to the LLM. The final count is determined by evidence coverage and available tokens—not automatically by the retrieval K.

How to decide

1. Identify required evidence: A single-fact question may need one strong passage; a comparison may need several sources and exceptions.
2. Filter first: Remove unauthorized, obsolete, irrelevant, and duplicate results.
3. Estimate the token budget: Reserve space for system instructions, user input, conversation history, source labels, and the answer.
4. Select by value: Include high-relevance, nonredundant chunks that cover every important part of the question.
5. Preserve context: Retain headings, qualifications, dates, and citations necessary to interpret the evidence.
6. Evaluate: Compare different final context sizes using grounded answer quality, evidence coverage, latency, and token cost.

Numerical example

Suppose the model can accept 8,000 input tokens, but instructions, the conversation, and the user question require 2,000 tokens. The application reserves 1,000 input tokens as a safety margin, leaving approximately 5,000 input tokens for evidence.

If each passage plus its source label averages 800 tokens, about six passages could fit (6 × 800 = 4,800). That is an upper budget illustration, not proof that six passages are relevant. If three cover the answer completely, use three.

The model's output-token allowance must also be configured separately according to its API's context-window and token limits.

Why not send every retrieved chunk?

• More context can introduce contradictory, outdated, or irrelevant evidence.
• Repeated chunks waste tokens without adding new facts.
• Large prompts can increase latency and cost.
• The LLM may miss key details buried among less useful text.

Conversely, trimming too aggressively can remove a critical exception or one side of a comparison.

Practical recommendation

Start by measuring final-context sizes such as 3, 5, and 8 passages against representative questions. Report both number of passages and total evidence tokens, because passage sizes vary. Choose the smallest context that reliably preserves answer-bearing evidence and necessary qualifications.

Interview takeaway

The final RAG context count should be chosen based on evidence completeness and token budget, separately from retrieval Top-K. Retrieve broadly if needed, rerank and deduplicate, then send a compact set of well-attributed passages that supports the entire answer.