How do you process charts and diagrams for RAG?

Charts and diagrams in RAG should be processed by extracting both their visible text and the relationships represented by the visual structure.

OCR alone is often not enough because a chart or diagram may communicate meaning through position, arrows, axes, colors, grouping, or connections.

1. Identify the visual type

First determine what kind of visual is present, for example:

• Bar chart
• Line chart
• Pie chart
• Flowchart
• Architecture diagram
• Network diagram
• Process diagram
• Annotated figure

Different visual types require different extraction strategies.

2. Extract visible text

Use OCR or native extraction to capture:

• Titles
• Axis labels
• Legends
• Data labels
• Node labels
• Annotations
• Captions

This text is useful, but it should stay connected to the visual element it describes.

3. Preserve relationships

For charts, preserve information such as:

• Which value belongs to which category
• Axis meaning and units
• Series names
• Trends
• Relative comparisons

For diagrams, preserve:

• Nodes or components
• Connections
• Arrow direction
• Parent-child relationships
• Processing order
• Grouping

For example, a diagram may show:

Client → API Gateway → Service → Database

Extracting only the labels Client, API Gateway, Service, and Database would lose the actual architecture.

4. Use visual understanding when structure matters

When relationships cannot be reliably reconstructed from text and layout rules alone, a vision-capable model or specialized visual parser can generate a structured description.

The description should focus on facts that could help answer questions rather than generic appearance.

For example:

The API Gateway routes requests to two backend services, and both services connect to the same PostgreSQL database.

5. Preserve captions and surrounding context

Charts and diagrams are often explained by nearby text.

Keep useful context such as:

• Figure title
• Caption
• Section heading
• Page number
• Nearby explanatory paragraph

This can make the extracted visual information easier to interpret during retrieval.

6. Store structured data when possible

If a chart can be converted into reliable structured values, preserve them.

For example:

2024 revenue: 12.4M
2025 revenue: 15.1M

This can support more precise retrieval than a vague description such as "revenue increased."

7. Keep source traceability

Image-derived content should remain linked to:

• Document ID
• Page number
• Figure number
• Image or region ID
• Original source

This allows users and developers to verify the retrieved evidence.

8. Validate important visuals

Before indexing, check that:

• Labels were read correctly
• Values and units are accurate
• Connections were interpreted correctly
• Arrow direction was preserved
• Legends were matched to the right series
• The description does not invent information that is not visible

9. Avoid processing decorative visuals

Logos, backgrounds, icons, and stock images usually do not improve retrieval and can be ignored unless they carry useful information.

Chart / diagram
      ↓
Classify visual
      ↓
Extract text
      ↓
Recover values / relationships
      ↓
Add caption + page context
      ↓
Validate
      ↓
Index structured data and/or description

For the broader image-handling workflow, see “How should images inside documents be handled in RAG?”.

In short, charts and diagrams should be converted into retrieval-friendly representations that preserve not only text, but also values, labels, connections, flow direction, and surrounding document context.