What metadata should be preserved while parsing documents for RAG?

Metadata preserved during RAG document parsing should help the system understand where a piece of content came from, where it belongs in the source, how current it is, and who is allowed to access it.

The exact metadata depends on the application, but several categories are especially useful.

1. Source identity

Preserve information that uniquely identifies the source document, such as:

• Document ID
• File name
• Source URL or storage location
• Document type
• Repository or knowledge-base name

This makes retrieved chunks traceable back to their original source.

2. Position inside the document

Keep metadata that identifies where the content appeared, for example:

• Page number
• Slide number
• Sheet name
• Row or record ID
• Section title
• Subsection title

This is important for citations, debugging, and reconstructing context.

3. Document hierarchy

Preserve structural relationships such as:

• Parent section
• Heading path
• Chapter
• Subchapter
• Table or figure number

For example:

Installation > Windows > Troubleshooting

This hierarchy can help a retrieved chunk remain understandable even when it is separated from the rest of the document.

4. Version and freshness information

When available, preserve:

• Document version
• Publication date
• Last-modified time
• Effective date
• Ingestion time

This helps the system distinguish current information from older content and supports re-indexing when documents change.

5. Access and permission metadata

For private or multi-user knowledge bases, metadata may need to include:

• Tenant or organization ID
• User or group permissions
• Confidentiality level
• Access-control labels

Retrieval should respect these permissions so users do not receive content they are not authorized to access.

6. Content-type metadata

Useful labels may include:

• Paragraph
• Table
• Code block
• Image description
• FAQ
• Product record

This can help the retriever apply filters or choose different retrieval strategies for different content types.

7. Processing metadata

In some systems it is useful to keep information such as:

• Parser version
• OCR usage
• OCR confidence
• Chunk ID
• Parent document ID

This is especially valuable for debugging ingestion problems and reprocessing documents later.

Do not preserve metadata blindly

More metadata is not always better.

Keep metadata that supports:

• Retrieval
• Filtering
• Access control
• Citations
• Freshness
• Debugging

Metadata that has no practical use can increase storage and operational complexity without improving retrieval.

For a deeper explanation of positional metadata, see “Why should page numbers and section headings be preserved in RAG ingestion?”.

In short, RAG parsing should preserve enough metadata to identify the source, locate the content within that source, retain hierarchy and version information, enforce permissions, and support reliable retrieval and citations.