Can text and images be stored in the same RAG retrieval system?

Yes. Text and images can be stored in the same RAG retrieval system, but the design depends on whether their embeddings are directly comparable.

Option 1: Shared multimodal embedding space

A multimodal embedding model can encode text and images into the same vector space.

That means the system can store:

• Text chunk embeddings
• Image embeddings

inside the same vector collection.

A text query can then retrieve either text or images based on similarity.

Example:

User asks:

"Show the diagram for the cooling system."

The query is converted into a multimodal embedding.

The vector search may return:

• A text paragraph describing the cooling system
• A diagram of the cooling system

because both live in the same compatible embedding space.

Option 2: Separate indexes

You can also keep:

Text embeddings → text index Image embeddings → image index

At query time, the system searches both indexes and combines the results.

This is useful when:

• Different embedding models are used
• Text and image vectors have different dimensions
• Their similarity scores are not directly comparable
• You want different retrieval logic for each modality

The final results can then be merged using ranking or fusion logic.

Why compatibility matters

You should not put arbitrary text embeddings and arbitrary image embeddings into one collection and assume they can be compared.

If they were generated by unrelated models, the vectors may represent completely different spaces.

Similarity scores would have no useful meaning.

The embeddings must either:

• Come from a model designed for cross-modal comparison

or

• Be searched separately and combined later

Store modality metadata

Each indexed item should identify what it represents.

Useful metadata includes:

• modality = text or image
• document_id
• page_number
• section
• figure_number
• caption
• source URI or file ID

This helps the retriever and generator understand what was returned.

Keep text and image relationships

Suppose page 12 contains:

• A paragraph describing a circuit
• Figure 4 showing the circuit

Both should be linked through metadata.

Then if the image is retrieved, the system can also fetch nearby explanatory text.

Similarly, retrieving the paragraph can bring the associated figure.

This often produces better answers than treating each modality as completely independent.

Do you store the actual image in the vector database?

Usually, the vector database stores:

• Image embedding
• Metadata
• A reference to the original image

The image itself may live in object storage, a document store, or another file system.

The vector record helps find it.

Unified vs separate retrieval

A shared multimodal index is convenient when one model supports text-image comparison well.

Separate indexes give more control when different models or retrieval strategies are needed.

Neither architecture is automatically better.

The right choice depends on the models, scale, and types of queries your application needs to support.

Interview takeaway

Text and images can be part of the same RAG retrieval system. If they are encoded by a compatible multimodal model, they can be stored and searched in one shared vector space. If different embedding models are used, keep separate indexes and combine results later. In both cases, preserve modality and source metadata so retrieved text and images can be linked back to the same document context.