How does chunk size affect retrieval quality?

Chunk size affects retrieval quality because it changes how much information each indexed unit contains and how precisely that unit can match a user's query.

The main tradeoff is between retrieval precision and context preservation.

Smaller chunks can improve precision

Smaller chunks usually cover a narrower topic.

This can make it easier for the retriever to find the exact passage that matches a specific query.

For example, a small chunk containing only:

API timeout: 30 seconds

may rank very well for a query about API timeout.

If that same fact is buried inside a much larger chunk containing authentication, logging, deployment, and billing information, the retrieval signal may become less focused.

But very small chunks can lose context

If chunks are too small, important relationships may be broken apart.

For example:

Chunk 1: The token expires after 30 minutes.
Chunk 2: Refresh tokens remain valid for 7 days.

If a question requires understanding both statements together, retrieving only one chunk may produce incomplete context.

Very small chunks can also separate:

• Definitions from explanations
• Headings from their content
• Conditions from outcomes
• Steps in a workflow

Larger chunks preserve more context

Larger chunks can keep several related sentences or paragraphs together.

This is useful when answers depend on broader context, such as:

• Procedures
• Policies
• Multi-step workflows
• Technical explanations

However, larger chunks may include more irrelevant content.

Large chunks can reduce retrieval precision

If one chunk covers many topics, its embedding or retrieval representation must summarize all of them.

This can make the chunk less strongly associated with any one specific topic.

As a result, the retriever may:

• Rank the chunk lower than expected
• Return broad but less focused passages
• Send unnecessary text to the language model

Chunk size also affects ranking diversity

Very small chunks may create many highly similar neighboring chunks.

Retrieval can then return several nearly identical passages instead of diverse evidence.

Very large chunks create the opposite problem: fewer retrieval units exist, but each one contains more mixed information.

Chunk size interacts with overlap

Overlap can reduce the risk of losing context at chunk boundaries.

For example:

Chunk size: 500 tokens
Overlap: 50 tokens

But increasing both chunk size and overlap can create significant duplication.

Too much duplication may cause several similar chunks to rank together.

Chunk size affects prompt efficiency

Retrieved chunks eventually become part of the context sent to the language model.

If chunks are very large:

• Fewer chunks fit into the available context window.
• More irrelevant text may consume tokens.

If chunks are reasonably focused, the system can combine evidence from several relevant locations.

The effect depends on the query

Different queries may prefer different chunk sizes.

A factual query such as:

What is the maximum upload size?

may benefit from a small, focused chunk.

A broader query such as:

How does the document approval process work?

may require a larger chunk containing several related steps.

How should retrieval quality be tested?

Evaluate several candidate chunk sizes using representative queries.

For each size, measure or inspect:

• Whether the correct evidence appears in the top results
• Whether retrieved chunks contain enough context
• Whether irrelevant content dominates the result
• Whether multiple returned chunks are duplicates
• Whether downstream answers improve

Do not choose a chunk size only because it is commonly used elsewhere.

For the definition of chunk size itself, see “What is chunk size in RAG?”.

In short, smaller chunks usually improve retrieval precision, while larger chunks preserve more context. The best retrieval quality comes from finding a size that balances both and validating it with real queries.