What is late chunking in RAG?

Late chunking in RAG is a technique that creates chunk embeddings after a long-context embedding model has already processed the broader document context.

Traditional RAG pipelines usually split a document first and embed each chunk independently. Late chunking changes that order.

Traditional chunk-then-embed flow

A common pipeline is:

Document
   ↓
Split into chunks
   ↓
Embed Chunk 1 independently
Embed Chunk 2 independently
Embed Chunk 3 independently

Because each chunk is encoded separately, its embedding may not contain information that appears elsewhere in the document.

Late chunking flow

Late chunking instead works conceptually like this:

Long document
   ↓
Process the full supported context with the embedding model
   ↓
Obtain contextualized token representations
   ↓
Apply chunk boundaries
   ↓
Pool the token representations inside each chunk
   ↓
Create contextualized chunk embeddings

The chunk boundaries are applied after the transformer has contextualized the tokens and before the final pooling step.

That is why the method is called late chunking.

Why does this preserve more context?

Consider this passage:

Acme Cloud introduced a new enterprise plan.
...
It supports up to 10,000 users.

If the second sentence is embedded independently, the word "It" may not clearly identify what it refers to.

With late chunking, the token representations for that later sentence are created while the model can still attend to earlier parts of the document. The resulting chunk embedding can therefore carry more of the surrounding document context.

What is actually chunked late?

Late chunking does not mean that the application stops using chunks for retrieval.

The final index still contains chunk-level embeddings.

The difference is that the model first creates contextualized token representations over a longer span, and those representations are then pooled according to the desired chunk boundaries.

Main advantage

The main goal is to create context-aware chunk embeddings without simply making every retrieval chunk larger.

This can help when a chunk contains:

• Pronouns whose subject appeared earlier
• Product names mentioned in a previous paragraph
• Section-level context not repeated in every paragraph
• Terms whose meaning depends on surrounding text

Late chunking vs contextual chunking

Both techniques try to preserve broader context, but they do it differently.

Contextual chunking typically adds explanatory context to the chunk text before indexing.

Late chunking instead changes how the embedding is computed: the broader text is processed first, and chunk embeddings are produced from contextualized token representations.

The two ideas are related but not identical.

Late chunking vs parent-child chunking

Parent-child chunking retrieves a small child and then expands to a larger parent for generation.

Late chunking focuses on making the child embedding itself more context-aware during indexing.

A system could potentially combine these approaches.

Requirements and limitations

Late chunking depends on an embedding model that can process sufficiently long input context.

It can also require:

• More embedding-time computation for long documents
• Careful alignment between original chunk boundaries and token positions
• A model and implementation that expose or support token-level representations before pooling

It therefore may be more complex than standard chunk-then-embed pipelines.

Does it eliminate normal chunking?

No.

The system still needs meaningful chunk boundaries for retrieval.

Late chunking changes when chunk-level embeddings are formed, not the need to decide what text belongs to each retrieval chunk.

The original late-chunking work, “Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models,” describes the method as processing the long text first and applying chunking after the transformer and before mean pooling.

In short, late chunking lets a long-context embedding model understand the broader document before chunk embeddings are produced. This gives individual retrieval chunks embeddings that can retain information from their surrounding document context.