Retrieval-Augmented Generation

Retrieval-augmented generation connects a generative model to a searchable knowledge source. Instead of relying only on model weights, the system fetches relevant documents and places them into the prompt or context.

What is hierarchical chunking?

Hierarchical chunking is a RAG chunking approach that represents a document at multiple levels of granularity instead of treating every chunk as an independent flat piece of text. A hierarchy might contain: Each smaller chunk knows where it belongs in the larger document structure. Why use a hierarchy? A user query may match a very specific paragraph, while the language model may need the surrounding subsection or section to answer correctly. Hierarchical chunking lets the system preserve both: Fine grained units for precise retrieval Larger units for broader context How hierarchical chunking works During ingestion, the document is parsed into structural levels such as: Document Chapter Section Subsection Paragraph Each node can store identifiers that connect it to its parent and children. For example: If Chunk C7 matches a query, the system can use its hierarchy to recover the surrounding subsection or section. Retrieval can happen at different levels Different implementations can use the hierarchy differently. A system may: Search only the smallest chunks and expand upward. Search both small and large levels. Retrieve a section first and then search within that section. Use the hierarchy to add neighboring or parent context after retrieval. The hierarchy gives the retrieval pipeline more flexibility than a flat list of chunks. Hierarchical chunking vs parent child chunking Parent child chunking commonly uses two main levels: Hierarchical chunking can extend this idea across several levels: So parent child chunking can be viewed as a simpler form of hierarchical organization, while hierarchical chunking may preserve a deeper document tree. Metadata is important Useful hierarchical metadata may include: document id parent id section id heading path level page number This allows the system to reconstruct where a chunk came from and navigate upward or downward through the document. Advantages Hierarchical chunking can: Preserve document organization Improve retrieval precision at lower levels Provide broader context from higher levels Support section aware retrieval Improve traceability and citations Work well for long structured documents Limitations It requires reliable document parsing and more complex storage. The system must preserve relationships between levels and decide: Which levels to index Which level to retrieve first When to expand to parents How much higher level context to include If the document has poor or inconsistent structure, building a useful hierarchy can be difficult. When is hierarchical chunking useful? It is especially useful for: Technical manuals Policies Research reports Legal documents Long documentation sites Books and multi section reports For a simpler two level version, see “What is parent child chunking in RAG?” . In short, hierarchical chunking preserves multiple levels of document structure so RAG can retrieve precise lower level chunks while still using their parent sections or higher level context when needed.

How does hierarchical chunking work in RAG?

Hierarchical chunking works by representing a document at several connected levels of granularity and letting the RAG pipeline use those levels differently during retrieval and context building. Instead of storing every chunk as an unrelated flat block, the system preserves relationships such as document → section → subsection → paragraph. 1. Parse the document structure The ingestion pipeline first identifies logical levels in the source. For example: The exact levels depend on the document type. A technical manual might use chapters, sections, and paragraphs, while a website might use pages, headings, and content blocks. 2. Create chunks at multiple levels The system can create representations for several levels rather than only the smallest text units. For example: Each lower level chunk remains connected to the higher level unit it came from. 3. Store hierarchy metadata Every chunk should carry enough information to reconstruct its position in the document. Useful fields may include: document id parent id section id heading path level page number For example: This makes it possible to move upward or downward through the hierarchy after retrieval. 4. Index the levels needed for retrieval Not every implementation indexes every level. A system may: Index only the smallest chunks. Index both small chunks and section summaries. Maintain separate indexes for different hierarchy levels. The right design depends on the retrieval strategy. 5. Retrieve relevant units When the user submits a query, retrieval finds the most relevant indexed chunks. For example: The retrieved chunk tells the system exactly where the relevant evidence exists. 6. Expand through the hierarchy when more context is needed The system can then follow the parent relationship. For example: If the small paragraph is sufficient, it may be used directly. If the answer requires more context, the subsection or section can be added. This allows context size to be adjusted dynamically. 7. Hierarchical retrieval can also work top down Some systems retrieve a higher level section first and then search within that section. For example: This can reduce the search space and keep retrieval focused within the correct part of a large document. 8. Deduplicate overlapping hierarchy results Several matching lower level chunks may belong to the same section. If the system expands each one independently, the same parent content could be added repeatedly. The pipeline should therefore deduplicate parent or section IDs before constructing the final prompt. 9. Build the final context After retrieval and hierarchy expansion, the selected evidence is assembled for the language model. A simplified flow is: Why hierarchical chunking helps It separates two goals: Precise retrieval from smaller units Context preservation from larger units It also lets the system use the document's structure rather than forcing every question to work with one fixed chunk size. Important design decisions A hierarchical system still needs tuning. You need to decide: Which hierarchy levels to create Which levels to index Which level retrieval should start from When to expand to parent context How much higher level context to include How to rank or deduplicate results across levels For the underlying concept, see “What is hierarchical chunking?” . In short, hierarchical chunking works by preserving multiple connected document levels, retrieving the most relevant level or leaf chunks, navigating through parent child relationships when more or less context is needed, and assembling the best sized evidence for generation.

What is late chunking in RAG?

Late chunking in RAG is a technique that creates chunk embeddings after a long context embedding model has already processed the broader document context . Traditional RAG pipelines usually split a document first and embed each chunk independently. Late chunking changes that order. Traditional chunk then embed flow A common pipeline is: Because each chunk is encoded separately, its embedding may not contain information that appears elsewhere in the document. Late chunking flow Late chunking instead works conceptually like this: The chunk boundaries are applied after the transformer has contextualized the tokens and before the final pooling step . That is why the method is called late chunking . Why does this preserve more context? Consider this passage: If the second sentence is embedded independently, the word "It" may not clearly identify what it refers to. With late chunking, the token representations for that later sentence are created while the model can still attend to earlier parts of the document. The resulting chunk embedding can therefore carry more of the surrounding document context. What is actually chunked late? Late chunking does not mean that the application stops using chunks for retrieval. The final index still contains chunk level embeddings. The difference is that the model first creates contextualized token representations over a longer span, and those representations are then pooled according to the desired chunk boundaries. Main advantage The main goal is to create context aware chunk embeddings without simply making every retrieval chunk larger. This can help when a chunk contains: Pronouns whose subject appeared earlier Product names mentioned in a previous paragraph Section level context not repeated in every paragraph Terms whose meaning depends on surrounding text Late chunking vs contextual chunking Both techniques try to preserve broader context, but they do it differently. Contextual chunking typically adds explanatory context to the chunk text before indexing. Late chunking instead changes how the embedding is computed : the broader text is processed first, and chunk embeddings are produced from contextualized token representations. The two ideas are related but not identical. Late chunking vs parent child chunking Parent child chunking retrieves a small child and then expands to a larger parent for generation. Late chunking focuses on making the child embedding itself more context aware during indexing . A system could potentially combine these approaches. Requirements and limitations Late chunking depends on an embedding model that can process sufficiently long input context. It can also require: More embedding time computation for long documents Careful alignment between original chunk boundaries and token positions A model and implementation that expose or support token level representations before pooling It therefore may be more complex than standard chunk then embed pipelines. Does it eliminate normal chunking? No. The system still needs meaningful chunk boundaries for retrieval. Late chunking changes when chunk level embeddings are formed , not the need to decide what text belongs to each retrieval chunk. The original late chunking work, “Late Chunking: Contextual Chunk Embeddings Using Long Context Embedding Models,” describes the method as processing the long text first and applying chunking after the transformer and before mean pooling. In short, late chunking lets a long context embedding model understand the broader document before chunk embeddings are produced. This gives individual retrieval chunks embeddings that can retain information from their surrounding document context.

How does late chunking work in RAG?

Late chunking works by letting a long context embedding model understand the broader document before the final chunk embeddings are created. In a normal RAG pipeline, text is usually split first and each chunk is embedded independently. Late chunking changes the order: the longer text is encoded first, and chunk level pooling happens afterward. 1. Decide the chunk boundaries The document still needs normal chunk boundaries. For example: These boundaries identify which tokens should later belong to each retrieval chunk. 2. Tokenize the larger document context Instead of sending each chunk separately to the embedding model, the system tokenizes a much larger portion of the document, potentially the full document if it fits within the model's supported context length. Conceptually: The system also keeps track of which token positions belong to each original chunk. 3. Run the long context through the transformer The full supported token sequence is passed through the embedding model's transformer. During this step, each token representation can incorporate information from surrounding tokens in the larger context. For example, consider: When the later sentence is processed together with the earlier context, the representation of "It" can reflect information about the enterprise plan rather than being encoded in isolation. 4. Obtain contextualized token embeddings After the transformer, the system has a contextualized vector for each token. Conceptually: At this stage the broader document context has already influenced the token representations. 5. Apply the chunk boundaries after contextualization Now the system uses the previously determined chunk boundaries to group the contextualized token vectors. For example: This is the key difference from ordinary chunk then embed processing: chunk separation happens after transformer contextualization for the purpose of producing the final chunk representations. 6. Pool the token vectors for each chunk Each group of contextualized token vectors is reduced to a single chunk embedding, commonly through mean pooling. Conceptually: The resulting vector represents the chunk while retaining information learned from the larger surrounding text. 7. Store the chunk embeddings normally After the contextualized chunk embeddings are created, they can be stored in a vector index just like ordinary chunk embeddings. A stored record may contain: Chunk text Contextualized embedding Document ID Section heading Page or source metadata At query time, retrieval can then work like a standard vector search RAG pipeline. What changes at query time? Usually, the major change is during ingestion and embedding creation , not during the basic retrieval step. The user query is embedded, vector similarity is calculated, and relevant chunks are returned in the usual way. The difference is that the indexed chunk vectors were created with awareness of their broader document context. What if the document is longer than the embedding model's context window? Late chunking can only contextualize the amount of text that the embedding model can process at once. If a document exceeds that limit, the ingestion pipeline still needs to divide it into larger processing windows or document sections before applying late chunking within those supported spans. Why can this improve retrieval? Independent chunk embeddings may lose information that appears outside the chunk, such as: The subject of a pronoun A product name introduced earlier A section level topic Definitions established in previous paragraphs Late chunking allows those relationships to influence the token representations before the final chunk vector is produced. The original paper “Late Chunking: Contextual Chunk Embeddings Using Long Context Embedding Models” describes this as applying chunking after the transformer and just before mean pooling. For the introductory explanation, see “What is late chunking in RAG?” . In short, late chunking first contextualizes the long document with a long context embedding model, then groups the contextualized token vectors according to chunk boundaries, pools each group into a chunk embedding, and stores those context aware embeddings for normal RAG retrieval.

What are the advantages of late chunking in RAG?

Late chunking can improve RAG retrieval because it lets each chunk embedding retain information from the broader document context instead of representing the chunk in isolation. Traditional chunking usually splits first and embeds each chunk independently. Late chunking lets a long context embedding model contextualize the larger text first, then creates chunk embeddings from those contextualized token representations. 1. Better preservation of cross chunk context A small chunk may contain references that depend on earlier text. For example: If the second passage is embedded independently, the meaning of "It" may be unclear. With late chunking, the later tokens are contextualized while the earlier text is still visible to the embedding model. The resulting chunk embedding can therefore better represent what the passage refers to. 2. More informative embeddings for small chunks Small chunks are useful because they usually provide focused retrieval units. The problem is that small chunks can lose surrounding information. Late chunking helps preserve the precision benefits of smaller chunks while making their embeddings more aware of the larger document. This reduces the need to make every retrieval chunk large simply to preserve context. 3. Better representation of ambiguous references Late chunking can help when a chunk contains: Pronouns such as "it", "they", or "this" Product names introduced earlier Abbreviations defined in previous paragraphs Section level topics not repeated in every chunk Terms whose meaning depends on earlier context Because the transformer processes the broader span before pooling, these dependencies can influence the final chunk representation. 4. Improved retrieval quality on long documents The original Late Chunking paper reports stronger results across multiple retrieval tasks compared with conventional chunking approaches. The improvement comes from creating chunk embeddings that capture information from the larger text instead of encoding each chunk as an isolated passage. 5. Retrieval chunks can remain focused One way to preserve more context is to create very large chunks. But large chunks can mix topics and reduce retrieval precision. Late chunking offers another approach: This separates the amount of context used to understand the text from the amount of text stored as each retrieval unit . 6. No special query time retrieval process is required Once contextualized chunk embeddings have been created and stored, retrieval can still use a normal vector search workflow. Conceptually: The main change happens during ingestion when the document embeddings are created. This means the technique can improve the indexed representations without requiring an entirely different RAG retrieval architecture. 7. It can work without additional model training The original late chunking method is designed to work with long context embedding models without requiring additional training. This makes the technique potentially applicable to existing suitable embedding models, although the paper also describes dedicated fine tuning as a way to improve it further. 8. It is a general embedding technique Late chunking is not tied to one specific document type or retrieval domain. The method operates on contextualized token representations and can, in principle, be applied across different long context embedding models that support the required processing and pooling behavior. What late chunking does not solve Late chunking does not remove the need for: Good chunk boundaries Appropriate chunk sizes Useful metadata Retrieval evaluation It improves how chunk embeddings capture surrounding context, but the rest of the RAG pipeline still needs to be designed carefully. For the mechanics of the method, see “How does late chunking work in RAG?” . In short, the main advantage of late chunking is that it creates focused chunk embeddings that retain information from the broader document context. This can improve retrieval of context dependent passages without forcing the system to use overly large retrieval chunks.

What are the limitations of late chunking in RAG?

Late chunking can preserve more document context inside chunk embeddings, but it also introduces practical and architectural limitations. It is not automatically the best choice for every RAG system. 1. It requires a long context embedding model Late chunking works only when the embedding model can process a sufficiently large span of the document before chunk level pooling. If the model supports only short inputs, it cannot contextualize tokens across a long document. This makes model context length an important constraint. 2. Very long documents can still exceed the model context window Late chunking does not remove context window limits. If a document is longer than the embedding model can process at once, the ingestion pipeline must still divide it into larger processing windows or sections. Context can then be preserved only within those supported spans, not necessarily across the entire original document. 3. Ingestion can require more computation Traditional chunking embeds each small chunk separately. Late chunking runs the transformer over a much longer sequence before producing the chunk embeddings. Longer transformer sequences generally require more memory and computation during ingestion. For large document collections, this can affect: Embedding throughput GPU or accelerator memory Ingestion latency Infrastructure cost 4. Implementation is more complex than standard chunk then embed A standard pipeline can simply: Late chunking needs additional coordination: The implementation therefore needs access to the model's token level representations and control over the pooling step. 5. Chunk boundaries are still required Late chunking does not eliminate chunking decisions. The original method still requires boundary cues so the contextualized token representations can be grouped into retrieval chunks. You still need to decide: Where chunks begin and end How large chunks should be Whether document structure should guide the boundaries Poor chunk boundaries can still reduce retrieval quality. 6. Token to chunk alignment must be handled correctly The text boundaries chosen by a parser must be aligned with the token positions produced by the embedding model. This can become harder when: Tokenization changes character boundaries Documents contain unusual formatting Parsed text differs from the original source Large documents are processed in several windows Incorrect alignment can pool the wrong token representations into a chunk. 7. Benefits depend on the retrieval task Late chunking is designed to help when chunk meaning depends on broader context. But not every dataset has that problem. If chunks are already self contained, the improvement may be small. For example, independent FAQ entries, short records, or product attributes may already contain enough context without long document contextualization. Recent comparative research also indicates that contextualized chunking is task dependent rather than universally superior: it can improve some in corpus retrieval settings while performing worse in some in document retrieval settings. 8. The embedding model must actually use long context well A model supporting a long token limit does not automatically mean it represents distant context equally well everywhere. Late chunking depends on the quality of the underlying long context representations. If the model does not effectively propagate useful information across the long sequence, the expected benefit can be limited. 9. It does not replace the rest of RAG optimization Late chunking improves how chunk embeddings are created, but it does not solve every retrieval problem. A production RAG system may still need: Good parsing Useful metadata Hybrid search Reranking Query rewriting Retrieval evaluation Context deduplication Late chunking should therefore be treated as one retrieval optimization, not a complete RAG strategy. When may late chunking be unnecessary? It may add little value when: Documents are already short. Each chunk is naturally self contained. The embedding model cannot process useful long context. Ingestion cost or implementation simplicity is more important. Evaluation shows no meaningful retrieval improvement. For the benefits of the technique, see “What are the advantages of late chunking in RAG?” . In short, late chunking can improve contextual chunk representations, but it requires suitable long context embedding models, more complex ingestion, careful token boundary alignment, and additional computation. Its value should be confirmed with retrieval evaluation because the gains depend on the documents and retrieval task.

When should late chunking be used in RAG?

Late chunking should be used in RAG when individual retrieval chunks need information from the broader document to be represented correctly . It is most useful when normal chunk then embed pipelines lose important context because each chunk is embedded independently. 1. Use it for long documents with cross chunk dependencies Late chunking is a strong candidate when later passages frequently depend on information introduced earlier. Examples include: Technical manuals Research papers Long reports Policies Legal documents Product documentation For example: If the second passage is embedded alone, the meaning of "It" may be weak or ambiguous. Late chunking lets the embedding model process the broader context before creating the final chunk embedding. 2. Use it when you want small retrieval chunks without losing broader context Small chunks are useful for precise retrieval, but they can lose surrounding information. Making chunks much larger can preserve context, but large chunks may mix several topics and reduce retrieval precision. Late chunking is useful when you want both: 3. Use it when references depend on earlier text It can be valuable when chunks contain: Pronouns such as "it", "they", or "this" Abbreviations defined earlier Product or company names introduced in previous paragraphs Section level topics not repeated in every chunk Statements whose meaning depends on preceding paragraphs These are situations where independent chunk embeddings can miss important relationships. 4. Use it when documents are long enough for contextualization to matter Jina AI's published evaluation of late chunking found larger improvements as average document length increased. This makes intuitive sense: very short documents have less surrounding information to lose, while longer documents create more opportunities for important context to fall outside an individual chunk. Late chunking is therefore more compelling for substantial documents than for tiny standalone records. 5. Use it when a suitable long context embedding model is available The embedding model must be able to process a useful amount of the document in one forward pass. If a model can only accept short inputs, late chunking cannot preserve context across a much larger span. For documents exceeding the model's context limit, the system may need larger processing windows and apply late chunking within each window. 6. Use it when retrieval evaluation shows context loss Late chunking should solve an observed retrieval problem, not be enabled only because it is newer. Useful symptoms include: Correct passages rank poorly when the subject is mentioned in an earlier chunk. Retrieved chunks contain ambiguous references. Increasing chunk size improves retrieval but creates too much irrelevant text. Adding document context manually improves retrieval. These indicate that context aware embeddings may help. When simpler chunking may be better Late chunking may provide little value when: Documents are short. Each record is naturally independent. FAQ answers are already self contained. Product attributes contain their own identifying context. A long context embedding model is unavailable. Standard chunking already performs well in evaluation. Ingestion simplicity and throughput are more important than possible retrieval gains. Compare it against alternatives Depending on the problem, you should also evaluate: Structure based chunking Contextual chunking Parent child retrieval Hierarchical chunking Metadata enrichment For example, if the missing context is simply a section heading, including that heading in every chunk may be simpler than changing the embedding pipeline. A practical decision rule Late chunking is a good candidate when all three are true: 1. The documents contain meaningful long range context. 2. Small independently embedded chunks lose that context. 3. A suitable long context embedding model and ingestion budget are available. Then compare late chunking with the existing approach using representative retrieval queries before adopting it in production. For its tradeoffs, see “What are the limitations of late chunking in RAG?” . In short, use late chunking when long documents contain cross chunk dependencies and you want focused retrieval chunks whose embeddings still reflect the broader document. Use it when evaluation demonstrates that independent chunk embeddings are losing useful context—not as a universal replacement for simpler chunking.

When should late chunking not be used in RAG?

Late chunking should not be used automatically in every RAG system. It is most useful when broader document context improves the meaning of smaller retrieval chunks; when that condition is absent, simpler chunking may be better. 1. Avoid it when chunks are already self contained If each retrieval unit already includes the information needed to understand it, there may be little benefit from contextualizing it with the rest of the document. Examples include: FAQ question answer pairs Product attributes Error code definitions Independent database records Short knowledge base entries In these cases, normal chunk then embed pipelines are often simpler and sufficient. 2. Avoid it for very short documents Late chunking is designed to preserve context that would otherwise fall outside an independently embedded chunk. If the entire document is already short, there may be little surrounding context to recover. Jina AI's early evaluation found that the improvement over naive chunking tended to increase with document length, while extremely short texts showed little or no difference. 3. Avoid it when queries target isolated facts inside documents with lots of irrelevant context Broader context is not always helpful. If a long document contains many unrelated details and users usually search for one narrow fact, contextualizing a small chunk with the broader document can introduce distracting information into its representation. Jina AI's later comparison found that late chunking could regress on some fact retrieval datasets such as NarrativeQA and 2WikiMultiHopQA, where additional document context was more distracting than useful. That means late chunking is not automatically better simply because a document is long. 4. Avoid it when a suitable long context embedding model is unavailable Late chunking depends on processing a meaningful span of the document before chunk level pooling. If the embedding model supports only a short context window, the technique cannot preserve much information beyond ordinary chunk boundaries. In that case, alternatives such as structure based chunking, contextual prefixes, parent child retrieval, or better metadata may be more practical. 5. Avoid it when documents greatly exceed the usable context window Late chunking can only contextualize the text that fits into the embedding model's supported input. If documents are much longer, they still need to be divided into larger processing windows. This does not make late chunking impossible, but it reduces the amount of truly document wide context available and increases pipeline complexity. 6. Avoid it when ingestion simplicity and throughput are priorities A conventional pipeline is straightforward: Late chunking requires a more specialized ingestion flow: If standard retrieval already performs well, this extra complexity may not be justified. 7. Avoid it when you cannot control token level pooling The original late chunking method applies chunk boundaries after the transformer and before mean pooling. An embedding service or model interface that only returns one final vector per input may not expose enough control to implement the method directly. In that situation, use a provider with explicit late chunking support or choose another retrieval strategy. 8. Avoid it when evaluation shows no measurable gain Late chunking should be adopted because it improves the actual retrieval workload, not because it is newer. Compare it against the current baseline using representative queries. Measure or inspect: Whether relevant chunks rank higher Whether fact retrieval improves or regresses Whether answer quality improves Whether broader context introduces noise Whether the ingestion cost is justified If the baseline performs as well or better, keep the simpler system. Consider simpler alternatives first If the missing context is easy to restore, simpler methods may solve the problem. For example: Add the heading path to each chunk. Preserve document titles in the indexed text. Use contextual chunk prefixes. Retrieve a small child chunk and expand to its parent. Improve structural chunk boundaries. These approaches may provide enough context without changing how embeddings are pooled. A practical decision rule Late chunking is usually a poor fit when one or more of these are true: Chunks are already self contained. Documents are very short. Queries target isolated facts amid large amounts of unrelated context. Long context embedding support is unavailable. The embedding interface does not support the required token level processing. Ingestion complexity or compute cost matters more than potential retrieval gains. Evaluation shows no meaningful improvement. For the positive use cases, see “When should late chunking be used in RAG?” . In short, do not use late chunking by default. Prefer it when broader context genuinely helps chunk meaning; prefer simpler chunking when content is already self contained, fact retrieval benefits from isolation, implementation constraints are significant, or evaluation does not show a clear gain.

What is chunk overlap?

Chunk overlap is the amount of content intentionally repeated between two neighboring chunks when a document is split for RAG. It is usually configured together with chunk size. Simple example Suppose a document is split into chunks of 500 tokens with an overlap of 50 tokens. The last 50 tokens of one chunk are repeated at the beginning of the next chunk. What can be overlapped? Overlap does not have to be measured only in tokens. It can also be defined using: Characters Words Sentences Paragraph fragments For example, sentence based chunking may repeat one complete sentence between neighboring chunks. Chunk size and overlap are different Chunk size defines how much content belongs to one chunk . Chunk overlap defines how much content is repeated between neighboring chunks . For example: means each chunk targets about 500 tokens while 50 tokens are shared with the next chunk. Overlap is usually a configuration, not a separate chunking strategy Overlap can be applied to several chunking approaches, including: Fixed size chunking Recursive chunking Sentence based chunking Some structure based chunking pipelines The underlying strategy decides where chunks are created, while overlap decides how much neighboring content is repeated. What happens to overlapped content? The repeated content becomes part of both chunks. This means it may be: Embedded more than once Stored in multiple chunk records Retrieved from more than one neighboring chunk Because of this duplication, overlap affects index size and retrieval behavior. Is overlap always required? No. If chunks already follow strong natural boundaries and each chunk is understandable on its own, little or no overlap may be necessary. Whether overlap should be used depends on the chunking strategy and the document structure. The reason overlap is often added is covered separately in “Why is chunk overlap used?” . In short, chunk overlap is the repeated portion shared by adjacent RAG chunks. It is configured alongside chunk size and can be measured in tokens, words, sentences, or other text units.

Why is chunk overlap used?

Chunk overlap is used in RAG to reduce the amount of useful context lost at the boundaries between neighboring chunks. When a document is split, an important sentence, explanation, or relationship can fall across two chunks. Repeating a small amount of text between them increases the chance that at least one chunk contains enough context to be useful on its own. Why chunk boundaries can cause problems Imagine this sentence is split between two chunks: Neither chunk contains the complete statement. If a user asks about token expiration, retrieval may return only one half of the information. Overlap keeps boundary context together With overlap, some text from the end of one chunk is repeated at the beginning of the next. For example: Now both chunks carry more of the complete idea. What kinds of information can overlap protect? Overlap can help preserve: A sentence and its continuation A heading and the first explanation below it A definition and its example A condition and its result Consecutive steps in a process Related sentences that span a chunk boundary Why this can improve retrieval Retrieval works on individual chunks. If an important concept is split across two chunks, each chunk may contain only a weak signal for the query. Repeating a small boundary region can make one or both chunks more complete and easier to match. Overlap is especially useful with artificial boundaries Overlap is often more useful when chunks are created by fixed character or token counts because those boundaries may cut through sentences or ideas. If chunks already follow strong natural boundaries such as headings, paragraphs, or complete sentences, less overlap may be needed. Why not use very large overlap? Overlap creates duplication. Too much overlap can: Increase the number of repeated tokens stored Increase embedding and storage cost Cause nearly identical chunks to appear together in retrieval results Waste context window space when repeated chunks are passed to the language model So overlap should be large enough to protect context but not so large that duplication dominates retrieval. Overlap does not replace good chunking Overlap is a safeguard around chunk boundaries. It should not be used as a substitute for preserving meaningful document structure whenever that structure is available. For the definition of overlap itself, see “What is chunk overlap?” . In short, chunk overlap is used to preserve continuity across chunk boundaries so important information is less likely to be split into incomplete retrieval units. Its benefit must be balanced against the duplication it introduces.