How does late chunking work in RAG?

Late chunking works by letting a long-context embedding model understand the broader document before the final chunk embeddings are created.

In a normal RAG pipeline, text is usually split first and each chunk is embedded independently. Late chunking changes the order: the longer text is encoded first, and chunk-level pooling happens afterward.

1. Decide the chunk boundaries

The document still needs normal chunk boundaries.

For example:

Chunk 1 → Introduction
Chunk 2 → Authentication
Chunk 3 → Token expiration

These boundaries identify which tokens should later belong to each retrieval chunk.

2. Tokenize the larger document context

Instead of sending each chunk separately to the embedding model, the system tokenizes a much larger portion of the document, potentially the full document if it fits within the model's supported context length.

Conceptually:

Document
   ↓
Token 1, Token 2, Token 3, ... Token N

The system also keeps track of which token positions belong to each original chunk.

3. Run the long context through the transformer

The full supported token sequence is passed through the embedding model's transformer.

During this step, each token representation can incorporate information from surrounding tokens in the larger context.

For example, consider:

Acme Cloud introduced a new enterprise plan.
...
It supports up to 10,000 users.

When the later sentence is processed together with the earlier context, the representation of "It" can reflect information about the enterprise plan rather than being encoded in isolation.

4. Obtain contextualized token embeddings

After the transformer, the system has a contextualized vector for each token.

Conceptually:

Token 1 → contextualized vector
Token 2 → contextualized vector
...
Token N → contextualized vector

At this stage the broader document context has already influenced the token representations.

5. Apply the chunk boundaries after contextualization

Now the system uses the previously determined chunk boundaries to group the contextualized token vectors.

For example:

Chunk 1 → token vectors 1–220
Chunk 2 → token vectors 221–510
Chunk 3 → token vectors 511–760

This is the key difference from ordinary chunk-then-embed processing: chunk separation happens after transformer contextualization for the purpose of producing the final chunk representations.

6. Pool the token vectors for each chunk

Each group of contextualized token vectors is reduced to a single chunk embedding, commonly through mean pooling.

Conceptually:

Contextualized token vectors for Chunk 2
                 ↓
             Mean pooling
                 ↓
      Context-aware Chunk 2 embedding

The resulting vector represents the chunk while retaining information learned from the larger surrounding text.

7. Store the chunk embeddings normally

After the contextualized chunk embeddings are created, they can be stored in a vector index just like ordinary chunk embeddings.

A stored record may contain:

• Chunk text
• Contextualized embedding
• Document ID
• Section heading
• Page or source metadata

At query time, retrieval can then work like a standard vector-search RAG pipeline.

What changes at query time?

Usually, the major change is during ingestion and embedding creation, not during the basic retrieval step.

The user query is embedded, vector similarity is calculated, and relevant chunks are returned in the usual way.

The difference is that the indexed chunk vectors were created with awareness of their broader document context.

What if the document is longer than the embedding model's context window?

Late chunking can only contextualize the amount of text that the embedding model can process at once.

If a document exceeds that limit, the ingestion pipeline still needs to divide it into larger processing windows or document sections before applying late chunking within those supported spans.

Why can this improve retrieval?

Independent chunk embeddings may lose information that appears outside the chunk, such as:

• The subject of a pronoun
• A product name introduced earlier
• A section-level topic
• Definitions established in previous paragraphs

Late chunking allows those relationships to influence the token representations before the final chunk vector is produced.

The original paper “Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models” describes this as applying chunking after the transformer and just before mean pooling.

For the introductory explanation, see “What is late chunking in RAG?”.

In short, late chunking first contextualizes the long document with a long-context embedding model, then groups the contextualized token vectors according to chunk boundaries, pools each group into a chunk embedding, and stores those context-aware embeddings for normal RAG retrieval.