Why should document and query embeddings use compatible embedding models?

Document and query embeddings should use compatible embedding models because vector similarity is only meaningful when both sides are represented in the same or intentionally aligned vector space.

A RAG system compares the query vector with stored document vectors. If those vectors come from unrelated embedding models, the numerical coordinates usually do not have the same meaning.

Think of embeddings as coordinates

An embedding model maps text into a multidimensional vector space.

For example:

Document chunk
   ↓
Embedding model A
   ↓
[0.12, -0.44, 0.83, ...]

and:

User query
   ↓
Embedding model A
   ↓
[0.09, -0.39, 0.79, ...]

Because both vectors were produced in the same space, their distance or similarity can be interpreted.

What happens if unrelated models are mixed?

Suppose document chunks were embedded with Model A, but the query is embedded with unrelated Model B.

Documents → Model A → Vector space A
Query     → Model B → Vector space B

Even if the vectors happen to have the same number of dimensions, dimension 1 in Model A does not necessarily represent the same learned feature as dimension 1 in Model B.

Comparing those vectors can therefore produce meaningless similarity scores.

Matching dimensions alone is not enough

Two embedding models may both output, for example, 1,024-dimensional vectors.

That does not make them compatible.

Compatibility requires the vectors to be trained or designed so their coordinates belong to the same semantic space.

The safest approach

For most RAG systems, the simplest approach is:

Document chunks → same embedding model
User queries    → same embedding model

This ensures that both document and query vectors are directly comparable.

Some models use different query and document encoders

Compatibility does not always mean the exact same input path.

Some retrieval models are intentionally trained with:

• A query encoder
• A document or passage encoder

These encoders may use different instructions, prefixes, or even separate model components while still producing vectors in a compatible space.

For example:

Query → query encoder ───────┐
                             ├→ compatible vector space
Document → passage encoder ──┘

In that case, using the model's recommended query/document encoding behavior is correct.

Why compatibility matters for similarity search

Vector databases rank document vectors using functions such as:

• Cosine similarity
• Dot product
• Euclidean distance

These calculations assume the vectors are comparable.

If query and document embeddings come from incompatible spaces, the ranking can become effectively arbitrary.

What if you change the embedding model?

If a RAG system switches to a new embedding model, existing document vectors generally need to be regenerated with that new model.

Otherwise, new query vectors would be compared against vectors created in the old embedding space.

This migration topic is covered separately in “How should embeddings be regenerated when the embedding model changes?”.

Model-specific input instructions matter too

Even when the same embedding model is used, some models expect different input formatting for queries and documents.

For example, a model may require conceptually:

query: reset password
passage: Users can reset their password from account settings.

Ignoring the model's recommended query/document format can reduce retrieval quality even though the underlying model is compatible.

Compatibility across an index

All vectors searched together in the same index should normally use a consistent embedding configuration.

That includes:

• Model or compatible encoder pair
• Output dimensionality
• Recommended normalization
• Query/document instructions

This keeps similarity scores meaningful across the indexed collection.

In short, document and query embeddings must be compatible because vector retrieval depends on comparing them inside the same semantic coordinate system. Using unrelated embedding models can make similarity scores meaningless, even when their output dimensions happen to match.