What is hierarchical chunking?

Hierarchical chunking is a RAG chunking approach that represents a document at multiple levels of granularity instead of treating every chunk as an independent flat piece of text.

A hierarchy might contain:

Document
  ├─ Section
  │   ├─ Subsection
  │   │   ├─ Paragraph chunk
  │   │   └─ Paragraph chunk
  │   └─ Subsection
  └─ Section

Each smaller chunk knows where it belongs in the larger document structure.

Why use a hierarchy?

A user query may match a very specific paragraph, while the language model may need the surrounding subsection or section to answer correctly.

Hierarchical chunking lets the system preserve both:

• Fine-grained units for precise retrieval
• Larger units for broader context

How hierarchical chunking works

During ingestion, the document is parsed into structural levels such as:

• Document
• Chapter
• Section
• Subsection
• Paragraph

Each node can store identifiers that connect it to its parent and children.

For example:

Document D1
  ↓
Section S2: Authentication
  ↓
Subsection SS3: Token Expiration
  ↓
Chunk C7: Access tokens expire after 30 minutes.

If Chunk C7 matches a query, the system can use its hierarchy to recover the surrounding subsection or section.

Retrieval can happen at different levels

Different implementations can use the hierarchy differently.

A system may:

• Search only the smallest chunks and expand upward.
• Search both small and large levels.
• Retrieve a section first and then search within that section.
• Use the hierarchy to add neighboring or parent context after retrieval.

The hierarchy gives the retrieval pipeline more flexibility than a flat list of chunks.

Hierarchical chunking vs parent-child chunking

Parent-child chunking commonly uses two main levels:

Parent
  └─ Child

Hierarchical chunking can extend this idea across several levels:

Document
  └─ Section
      └─ Subsection
          └─ Chunk

So parent-child chunking can be viewed as a simpler form of hierarchical organization, while hierarchical chunking may preserve a deeper document tree.

Metadata is important

Useful hierarchical metadata may include:

• document_id
• parent_id
• section_id
• heading_path
• level
• page_number

This allows the system to reconstruct where a chunk came from and navigate upward or downward through the document.

Advantages

Hierarchical chunking can:

• Preserve document organization
• Improve retrieval precision at lower levels
• Provide broader context from higher levels
• Support section-aware retrieval
• Improve traceability and citations
• Work well for long structured documents

Limitations

It requires reliable document parsing and more complex storage.

The system must preserve relationships between levels and decide:

• Which levels to index
• Which level to retrieve first
• When to expand to parents
• How much higher-level context to include

If the document has poor or inconsistent structure, building a useful hierarchy can be difficult.

When is hierarchical chunking useful?

It is especially useful for:

• Technical manuals
• Policies
• Research reports
• Legal documents
• Long documentation sites
• Books and multi-section reports

For a simpler two-level version, see “What is parent-child chunking in RAG?”.

In short, hierarchical chunking preserves multiple levels of document structure so RAG can retrieve precise lower-level chunks while still using their parent sections or higher-level context when needed.