How do you handle PDFs containing both text and images?

When a PDF contains both text and images, do not treat the whole file as plain text.

A good RAG pipeline processes the text and the visual content separately, then keeps the relationship between them.

1. Extract the normal text

Start by parsing the PDF text layer.

Keep useful structure such as:

• Page number
• Heading
• Section
• Paragraph order
• Figure references

This text can be chunked and embedded normally.

2. Detect and extract important images

Images may contain information that does not exist in the text layer.

Examples:

• Architecture diagrams
• Screenshots
• Flowcharts
• Product images
• Infographics

Extract these images together with their page number and surrounding text.

3. Turn the image into something searchable

There are two common approaches.

If the image mainly contains text, use OCR to extract that text.

If the image contains visual meaning, use a vision model to create a description or structured summary.

For example, an architecture diagram might become:

"Diagram showing API Gateway sending requests to three microservices, which write to PostgreSQL and Redis."

That description can then be embedded and searched like normal text.

4. Preserve the connection between text and image

This part is important.

Do not store an image description without knowing where it came from.

Keep metadata such as:

• Document ID
• Page number
• Figure number
• Image path or object ID
• Nearby heading
• Related text chunk

For example:

Text chunk: "The system architecture is shown in Figure 4."

Figure 4: architecture image

Image description: "API Gateway routes traffic to authentication, payment, and order services."

All three should remain linked.

5. Retrieve the right modality

If the user asks:

"Which service connects to Redis in the architecture?"

The answer may depend on the diagram rather than the surrounding paragraph.

The retriever can search both:

• Text chunks
• Image descriptions or multimodal embeddings

Then the RAG system can pass the relevant text and, when supported, the original image to a multimodal LLM.

6. Do not flatten everything into one text blob

A common mistake is:

PDF → extract all text → ignore images

That works only when the images are decorative.

If important information appears only in diagrams or screenshots, the RAG system will never retrieve it.

Practical pipeline

PDF → layout-aware parsing

Then:

Text → chunk → embed → index

Images → extract → OCR or vision description → embed → index

Both should share metadata so retrieval can reconstruct the original context.

Interview takeaway

For PDFs containing both text and images, use layout-aware parsing, process text and images separately, convert visual information into searchable text or multimodal embeddings, and preserve page and figure relationships. At retrieval time, return the relevant text, image, or both instead of assuming the PDF is only text.