Retrieval-Augmented Generation

Retrieval-augmented generation connects a generative model to a searchable knowledge source. Instead of relying only on model weights, the system fetches relevant documents and places them into the prompt or context.

What is chunking in RAG?

Chunking in RAG is the process of splitting a large document into smaller pieces of content called chunks before those pieces are indexed for retrieval. Instead of searching an entire document every time a user asks a question, the RAG system searches among these smaller chunks and retrieves the ones that are most relevant. Why documents are chunked Large documents often contain many unrelated topics. For example, a 100 page product manual may include: Installation instructions Configuration options Troubleshooting steps Security guidance API references If the entire manual were stored as one retrieval unit, the system would have difficulty identifying the exact section needed for a specific question. Chunking creates smaller searchable units. Simple example A document might contain: If a user asks about authentication, the retriever can return the authentication chunk instead of sending the whole document to the language model. What a chunk may contain A chunk can contain: One or more paragraphs A section or subsection Several sentences A table with its surrounding context A code block with explanation The exact boundaries depend on the chunking strategy. Chunking happens before indexing A common flow is: Each chunk usually keeps metadata such as its document ID, section heading, page number, and source so it can still be traced back to the original document. Why chunk size matters If chunks are too large : They may contain several unrelated topics. Retrieval can become less precise. More irrelevant text may be sent to the language model. If chunks are too small : Important context may be split across chunks. A retrieved passage may become difficult to understand on its own. More chunks may need to be retrieved to reconstruct the answer. The goal is therefore to create chunks that are small enough for precise retrieval but large enough to preserve useful context . Chunking is not only about length Good chunking can also respect document structure. For example, instead of splitting after an arbitrary number of characters, a system may prefer boundaries such as: Paragraph ends Section headings Sentences Logical document units Different approaches are covered in the upcoming questions on fixed size, recursive, sentence based, semantic, and structure based chunking. In short, chunking in RAG breaks large documents into smaller, meaningful retrieval units so the system can find and send the most relevant pieces of information to the language model.

Why is chunking necessary in RAG?

Chunking is necessary in RAG because large documents usually contain more information than a retriever or language model needs for a single question . By splitting documents into smaller units, the system can retrieve only the parts that are most relevant to the user's query. 1. It improves retrieval precision If an entire document is stored as one retrieval unit, the retriever has to judge the relevance of that whole document. A long document may contain only one small section that answers the question. Chunking lets the retriever match the query against that specific section instead. 2. It reduces irrelevant context Without chunking, a retrieved document may contain large amounts of unrelated information. Sending all of that text to the language model can: Distract the model from the important evidence Increase token usage Increase latency and cost Make the final answer less focused Chunking helps send a smaller and more relevant context. 3. It makes embeddings more meaningful When a very large piece of text covers many topics, a single embedding has to represent all of them at once. This can make similarity matching less precise. A smaller chunk usually represents a narrower topic, which can make semantic retrieval more useful. 4. It helps fit retrieved evidence into the context window Language models can only process a limited amount of input at one time. If every retrieved item is very large, only a few sources may fit into the prompt. Smaller chunks allow the system to combine relevant evidence from several parts of one or more documents. 5. It allows fine grained ranking Retrievers and rerankers can score individual chunks instead of treating an entire document as equally relevant. For example, a manual might contain sections on: Installation Authentication Billing Troubleshooting If the user asks about billing, the billing chunk can rank highly while unrelated sections remain out of the retrieved context. 6. It supports targeted citations When chunks preserve metadata such as page number and section heading, the system can cite the specific location that supports an answer instead of pointing only to a large document. 7. It makes updates more manageable If only one section of a document changes, a chunk based system may be able to update the affected retrieval units instead of rebuilding a single giant representation for the whole document. But chunking introduces a tradeoff Chunking is necessary, but poor chunking can also hurt retrieval. If chunks are too large, retrieval becomes less precise. If chunks are too small, important context may be separated. The goal is therefore not simply to create small pieces, but to create meaningful retrieval units that balance precision and context . For the basic definition, see “What is chunking in RAG?” . The different ways to create chunks are covered next in “What are different types of chunking?” . In short, chunking is necessary in RAG because it turns large documents into focused retrieval units, making search more precise, reducing irrelevant context, improving representation quality, and helping the language model receive only the evidence it needs.

What are different types of chunking?

There are several ways to split documents into chunks for RAG. The main difference is how each strategy decides where one chunk should end and the next should begin . 1. Fixed size chunking Fixed size chunking splits text after a set number of characters, words, or tokens. For example: It is simple and predictable, but it can split sentences, paragraphs, or logical sections at awkward points. 2. Recursive chunking Recursive chunking tries larger natural boundaries first and falls back to smaller ones only when needed. For example, it may try to split by: Sections Paragraphs Sentences Words This usually preserves more natural structure than a purely fixed size approach. 3. Sentence based chunking Sentence based chunking groups complete sentences together until a target size is reached. This avoids breaking a sentence in the middle and can make each chunk easier to understand. However, sentence boundaries alone do not always preserve broader topic structure. 4. Semantic chunking Semantic chunking uses the meaning of the text to decide where topic boundaries occur. For example, when the discussion shifts from installation to authentication, the system may start a new chunk even if the previous chunk has not reached a fixed size. This can produce more coherent chunks, but it usually requires more processing than simple rule based methods. 5. Structure based chunking Structure based chunking follows the organization already present in the source document. Possible boundaries include: Chapters Headings Subheadings Markdown sections HTML sections Slides Table boundaries This works especially well when the source has reliable structure. Other variations Real systems often combine these strategies. For example, a pipeline might: Chunk overlap can also be added so neighboring chunks share a small amount of text. Overlap is usually a configuration choice applied to a chunking strategy rather than a separate core chunking type. Which type is best? There is no single best method for every RAG system. The right choice depends on: Document structure Typical query size Embedding model limits Retrieval behavior Whether context spans several paragraphs The precision required from retrieval The upcoming questions examine fixed size, recursive, sentence based, semantic, and structure based chunking individually. In short, the main RAG chunking strategies differ in whether they split by fixed length, natural language boundaries, semantic topic changes, or the source document's existing structure.

What is fixed-size chunking?

Fixed size chunking is a chunking method that splits text into pieces of a predefined length, such as a fixed number of characters, words, or tokens. The split happens based mainly on size rather than on the meaning or structure of the document. How fixed size chunking works Suppose the target chunk size is 500 tokens. The text may be divided like this: Each chunk is then indexed separately for retrieval. What can define the size? Fixed size chunks can be measured using: Characters Words Tokens Token based limits are often useful because embedding models and language models work with tokens and have token limits. What is chunk overlap? Fixed size chunking is often combined with overlap . For example, with a chunk size of 500 tokens and an overlap of 50 tokens: The repeated region helps preserve context when important information crosses a chunk boundary. Advantages Fixed size chunking is: Simple to implement Fast Predictable Easy to configure Useful as a baseline strategy It can work reasonably well when documents have fairly uniform text and strong structural parsing is not available. Limitations The main weakness is that the split points do not understand meaning. A boundary may occur: In the middle of a sentence Between a heading and its explanation Inside a list Between related paragraphs In the middle of a logical topic For example: Neither chunk contains the complete statement by itself. Overlap can reduce this problem, but it does not guarantee that chunks follow natural semantic boundaries. When is fixed size chunking useful? It can be a practical choice when: A simple baseline is needed Documents are mostly plain text Processing speed is important More advanced structure is unavailable Chunk sizes need to be predictable More structure aware approaches may work better when documents contain clear headings, paragraphs, or topic boundaries. For an approach that tries natural separators before falling back to smaller ones, see “What is recursive chunking?” . In short, fixed size chunking splits text according to a predefined length. It is simple and predictable, but because it does not understand document meaning, it can break related information across chunk boundaries.

What is recursive chunking?

Recursive chunking is a chunking method that tries to split text using larger natural boundaries first , and only uses smaller boundaries when a piece is still too large. Instead of cutting text strictly after a fixed number of characters or tokens, recursive chunking attempts to preserve document structure as much as possible. How recursive chunking works A recursive splitter may try separators in an order such as: Section breaks Paragraph breaks Sentence boundaries Spaces Individual characters as a last resort If a section is already small enough, it can remain intact. If it is too large, the splitter tries the next smaller boundary. Simple example Suppose a document contains: If each section fits within the target size, the sections may become separate chunks. If Section A is too large, it can be split into paragraphs. If one paragraph is still too large, the splitter can continue down to sentences or smaller units. Why is it called recursive? The same splitting logic is applied repeatedly to pieces that remain too large. Conceptually: Advantages Recursive chunking can: Preserve paragraphs and sections when possible Reduce awkward mid sentence splits Produce more readable chunks Keep related text together better than purely fixed size splitting Still enforce a maximum chunk size Limitations Recursive chunking still relies on predefined separators. It does not necessarily understand whether two paragraphs discuss the same topic or whether a topic changes in the middle of a section. It can also produce chunks of different sizes depending on the source structure. Recursive chunking and overlap Overlap can still be added between neighboring chunks. This can help preserve context when information spans a chunk boundary, although excessive overlap can create duplicate content in the index. When is recursive chunking useful? It is a practical choice for documents that contain natural text boundaries such as: Paragraphs Headings Markdown sections Documentation pages Articles It is often a stronger default than arbitrary fixed size splitting when the source text has reasonably clean structure. For the simpler length based approach, see “What is fixed size chunking?” . In short, recursive chunking repeatedly splits oversized text using progressively smaller natural boundaries, helping RAG preserve document structure while still keeping chunks within a desired size.

What is sentence-based chunking?

Sentence based chunking is a chunking method that uses complete sentences as the basic building blocks for creating chunks. Instead of splitting text at an arbitrary character or token position, the system first detects sentence boundaries and then groups one or more complete sentences together. How sentence based chunking works A typical process is: For example: The exact number of sentences in each chunk can vary because sentence lengths are different. Why use sentence boundaries? Complete sentences usually represent meaningful units of language. Keeping them intact helps avoid awkward splits such as: Sentence based chunking would normally keep that full statement together. Advantages Sentence based chunking can: Preserve complete thoughts Improve readability of retrieved passages Reduce mid sentence splits Work well for articles, documentation, reports, and prose Produce chunks that are easier for a language model to interpret Limitations Sentence boundaries do not always match topic boundaries. Several consecutive sentences may discuss different ideas, while one idea may span many sentences. For example, a long section on authentication may contain ten related sentences. Splitting them only according to a size limit can still separate important context. Sentence size can vary Some sentences are very short while others are extremely long. A practical implementation often combines sentence boundaries with a maximum token or character limit so that chunks do not become too large. Can sentence based chunking use overlap? Yes. A system can repeat one or more sentences between neighboring chunks. For example: This can preserve context across chunk boundaries, but excessive overlap increases duplication in the index. When is sentence based chunking useful? It is useful when: The source is mostly natural language prose Complete sentences are important for interpretation Paragraphs are too large or inconsistent A simple structure aware method is preferred For documents with strong headings, sections, tables, or other explicit structure, structure based chunking may preserve even more context. In short, sentence based chunking creates RAG chunks from complete sentences, helping preserve readable and meaningful statements while still grouping them into retrieval sized units.

What is semantic chunking?

Semantic chunking is a chunking method that splits text based on changes in meaning or topic , rather than using only a fixed number of characters, words, or tokens. The goal is to keep semantically related content together and create a new chunk when the discussion shifts to a different idea. How semantic chunking works A common approach is: Break the text into smaller units such as sentences or paragraphs. Represent those units using embeddings or another semantic similarity method. Compare neighboring units. Keep similar units together. Start a new chunk when the semantic similarity drops enough to suggest a topic change. For example, a document might move from installation requirements and troubleshooting into authentication methods and token expiration. A semantic chunker may keep the installation content together and begin a new chunk when the document shifts to authentication. Why semantic chunking can help It can produce chunks that are more internally coherent because the boundaries are based on meaning rather than only size. This can improve retrieval when users ask topic specific questions, because a retrieved chunk is less likely to contain several unrelated subjects. Semantic chunking still needs size controls A semantically consistent section can still become too large. For that reason, semantic chunking is often combined with minimum or maximum chunk size rules. For example: Keep related sentences together. Start a new chunk when the topic changes. Force a split if the chunk exceeds the allowed token size. Advantages Semantic chunking can: Keep related ideas together. Reduce arbitrary boundaries. Improve chunk coherence. Help topic focused retrieval. Adapt to documents whose section structure is weak or inconsistent. Limitations Semantic chunking usually requires more processing than simple rule based methods. It may involve: Creating embeddings during chunking. Comparing many neighboring text units. Choosing a similarity threshold. Tuning behavior for different document types. The result can also vary depending on the embedding model and threshold used. It is not the same as structure based chunking Semantic chunking finds boundaries from the meaning of the content . Structure based chunking uses explicit document signals such as headings, chapters, HTML sections, or slide boundaries. A production pipeline can combine both approaches. For example, it may first split by document sections and then apply semantic chunking inside sections that are still too large. When is semantic chunking useful? It can be useful when: Documents contain long sections with several topic shifts. Explicit headings are missing or unreliable. Retrieval quality is more important than the cheapest ingestion process. Fixed or sentence based chunking produces mixed topic chunks. For the strategy that follows explicit document structure, see “What is structure based chunking?” . In short, semantic chunking creates RAG chunks by detecting changes in meaning, helping related information stay together while starting new chunks when the topic changes.

What is structure-based chunking?

Structure based chunking is a chunking method that splits content according to the explicit organization already present in the source document . Instead of deciding boundaries only by length or semantic similarity, it uses structural signals such as headings, sections, slides, table boundaries, or HTML elements. How structure based chunking works A parser first identifies the document's structural elements. Possible boundaries include: Chapters Headings Subheadings Paragraph groups Markdown sections HTML sections Slides Tables FAQ entries Code sections Each logical unit can then become a chunk or a starting point for further splitting. Simple example Suppose a technical document contains: A structure based chunker can keep the authentication content separate from billing because the headings already define meaningful boundaries. Large sections may still need further splitting A document section can be structurally correct but too large for efficient retrieval. In that case, a common approach is: This means structure based chunking is often combined with recursive or size based splitting. Preserve the heading path When a subsection becomes a chunk, keep its parent hierarchy as metadata. For example: This gives the retrieved chunk context even when it is separated from the rest of the document. Advantages Structure based chunking can: Preserve the author's original organization Keep related content together Reduce arbitrary splits Improve chunk interpretability Preserve useful hierarchy for retrieval and citations Work naturally with structured formats such as HTML, Markdown, DOCX, PPTX, and well parsed PDFs Limitations It depends on the quality of the source structure and parsing. Problems can occur when: Headings are missing or inconsistent PDF structure is not detected correctly Sections are extremely large Formatting does not reflect real semantic boundaries For poorly structured documents, recursive or semantic chunking may provide better fallback behavior. Structure based vs semantic chunking Structure based chunking follows explicit document organization . Semantic chunking detects changes in meaning , even when no heading or structural marker exists. The two approaches can also be combined. When is structure based chunking useful? It is especially useful for: Technical documentation Policies and manuals Markdown files Web documentation Presentations Reports with clear headings Documents containing tables or distinct sections For the meaning based alternative, see “What is semantic chunking?” . In short, structure based chunking uses the source document's existing organization to create meaningful retrieval units, while applying smaller fallback splits when a structural section is still too large.

How do you choose the right chunking strategy for RAG?

Choosing the right chunking strategy for RAG means selecting a method that creates retrieval units that are small enough to search precisely but large enough to preserve the context needed to answer questions correctly . There is no single chunking strategy that works best for every dataset. 1. Start with the structure of the source If documents have reliable structure, use it. For example: Technical documentation → headings and sections HTML pages → semantic page sections PPTX files → slides FAQs → one question answer pair Tables → rows or logical table sections Plain prose → paragraphs or sentences Structure based chunking often works well when the source already contains meaningful boundaries. 2. Consider the questions users will ask Chunking should match the expected retrieval task. If users ask very specific questions, smaller chunks may improve precision. If answers usually require several related paragraphs, larger chunks may preserve more useful context. For example, a query such as: may need only a small configuration chunk. A query such as: may require a broader section containing several related steps. 3. Match the chunk size to the content Chunks that are too large may: Mix unrelated topics Reduce retrieval precision Add irrelevant prompt context Chunks that are too small may: Lose important context Separate definitions from explanations Require many chunks to reconstruct an answer The right size should be tested rather than chosen only from a generic rule. 4. Use document structure before arbitrary splitting A practical default is often: This preserves natural boundaries while still controlling chunk size. 5. Consider semantic chunking when topic shifts matter Semantic chunking can help when: Long sections contain several topics Headings are missing or unreliable Fixed size or recursive splitting produces mixed topic chunks However, it adds processing cost and tuning complexity, so it should be justified by retrieval improvements. 6. Decide whether overlap is necessary Overlap can help when information frequently crosses chunk boundaries. But too much overlap can: Duplicate indexed content Increase storage Return several nearly identical chunks Use enough overlap to protect context, not as a substitute for good boundaries. 7. Preserve metadata regardless of strategy Chunks should usually keep metadata such as: Document ID Section heading Heading path Page number Source URL This helps restore context even when a small chunk is retrieved independently. 8. Evaluate with real queries The best chunking strategy should be selected using retrieval evaluation rather than intuition alone. Test representative questions and check: Did retrieval find the correct evidence? Is the retrieved chunk understandable by itself? Is important context missing? Are irrelevant chunks ranking highly? Are too many duplicate chunks returned? Compare different chunk sizes and strategies using the same query set. 9. Use different strategies for different document types when needed A single knowledge base can contain PDFs, web pages, presentations, spreadsheets, and FAQs. These do not all need to use the same chunking method. A production ingestion pipeline can route each document type to an appropriate chunking strategy. A practical starting point For many structured text documents, a reasonable starting approach is: Preserve headings and hierarchy. Split at natural structural boundaries. Recursively split sections that are too large. Add limited overlap if needed. Measure retrieval quality and adjust. In short, choose a RAG chunking strategy based on document structure, expected queries, context requirements, and measured retrieval quality. Start with natural boundaries, control size carefully, and validate the choice using real search questions.

What is chunk size in RAG?

Chunk size in RAG is the amount of content placed inside one chunk before that chunk is indexed and made available for retrieval. It is usually measured in: Tokens Characters Words Sentences Token based measurement is common because embedding models and language models operate on tokens and have token limits. Simple example If a document is split using a target chunk size of 500 tokens, the result may look like: The exact boundaries may still depend on the chunking strategy. Why chunk size matters Chunk size directly affects what the retriever sees as one searchable unit. If chunks are too large : Several topics may be mixed together. Retrieval may become less precise. More irrelevant text may be sent to the language model. Fewer retrieved chunks may fit into the prompt. If chunks are too small : Important context may be separated. Definitions may be split from explanations. A single answer may require several chunks. Very similar small chunks may increase retrieval noise. Chunk size and retrieval precision Smaller chunks often make it easier to retrieve a very specific fact because each chunk covers a narrower topic. For example, a small chunk containing only API timeout configuration may rank more precisely for a timeout related query than a large chunk containing timeout, authentication, logging, and deployment information together. Chunk size and context preservation Larger chunks preserve more surrounding information. This can help when the answer depends on several related sentences or steps. For example, an authentication workflow may require multiple paragraphs to explain the full sequence. The goal is therefore to balance: Chunk size is different from chunking strategy Chunk size controls how much content is allowed in a chunk. Chunking strategy controls where the boundaries are chosen . For example, two systems may both target 500 token chunks while one uses recursive chunking and the other uses structure based chunking. What about overlap? Chunk overlap repeats some content between neighboring chunks. For example: Overlap can help preserve context across boundaries, but too much overlap creates duplicate indexed content and may cause nearly identical chunks to appear in retrieval results. Is there a best chunk size? There is no universal chunk size that works best for every RAG system. The right size depends on: Document type Typical question complexity Embedding model Retrieval method Amount of context needed for an answer Document structure Reranking and context building strategy How should chunk size be selected? Start with a reasonable candidate size, then test it using representative queries. Check whether: The correct evidence is retrieved. Retrieved chunks contain enough context. Chunks contain too much unrelated material. Important information is frequently split across boundaries. Too many overlapping or duplicate chunks are returned. Then compare different sizes using the same evaluation set. For choosing the broader strategy around size, see “How do you choose the right chunking strategy for RAG?” . In short, chunk size defines how much content each retrieval unit contains. Smaller chunks usually improve precision, larger chunks preserve more context, and the best size should be chosen through retrieval evaluation rather than a fixed universal rule.