What is chunking in RAG?

Chunking in RAG is the process of splitting a large document into smaller pieces of content called chunks before those pieces are indexed for retrieval.

Instead of searching an entire document every time a user asks a question, the RAG system searches among these smaller chunks and retrieves the ones that are most relevant.

Why documents are chunked

Large documents often contain many unrelated topics.

For example, a 100-page product manual may include:

• Installation instructions
• Configuration options
• Troubleshooting steps
• Security guidance
• API references

If the entire manual were stored as one retrieval unit, the system would have difficulty identifying the exact section needed for a specific question.

Chunking creates smaller searchable units.

Simple example

A document might contain:

Document
 ├─ Chunk 1: Installation
 ├─ Chunk 2: Configuration
 ├─ Chunk 3: Authentication
 └─ Chunk 4: Troubleshooting

If a user asks about authentication, the retriever can return the authentication chunk instead of sending the whole document to the language model.

What a chunk may contain

A chunk can contain:

• One or more paragraphs
• A section or subsection
• Several sentences
• A table with its surrounding context
• A code block with explanation

The exact boundaries depend on the chunking strategy.

Chunking happens before indexing

A common flow is:

Document
   ↓
Parse and clean
   ↓
Split into chunks
   ↓
Attach metadata
   ↓
Create embeddings or other retrieval representations
   ↓
Index

Each chunk usually keeps metadata such as its document ID, section heading, page number, and source so it can still be traced back to the original document.

Why chunk size matters

If chunks are too large:

• They may contain several unrelated topics.
• Retrieval can become less precise.
• More irrelevant text may be sent to the language model.

If chunks are too small:

• Important context may be split across chunks.
• A retrieved passage may become difficult to understand on its own.
• More chunks may need to be retrieved to reconstruct the answer.

The goal is therefore to create chunks that are small enough for precise retrieval but large enough to preserve useful context.

Chunking is not only about length

Good chunking can also respect document structure.

For example, instead of splitting after an arbitrary number of characters, a system may prefer boundaries such as:

• Paragraph ends
• Section headings
• Sentences
• Logical document units

Different approaches are covered in the upcoming questions on fixed-size, recursive, sentence-based, semantic, and structure-based chunking.

In short, chunking in RAG breaks large documents into smaller, meaningful retrieval units so the system can find and send the most relevant pieces of information to the language model.