How do embeddings work in a RAG pipeline?

Embeddings are the bridge between raw text and semantic retrieval in a RAG pipeline.

They convert document chunks and user queries into numerical vectors so the system can compare them by meaning.

During ingestion

Each document is split into chunks.

Example:

"Employees can carry forward up to 10 unused leave days."

That chunk is passed through an embedding model:

Chunk → embedding model → vector

Conceptually:

[0.12, -0.44, 0.81, ...]

The vector, original chunk, and metadata are stored together in a vector index.

At query time

Suppose the user asks:

"How many unused leave days can I keep for next year?"

The query is embedded using the same or a compatible embedding model:

Query → embedding model → query vector

Now the system can compare that query vector with the stored chunk vectors.

Why this works

The user and the document may use different words:

Query: "How many unused leave days can I keep?"

Document: "Employees can carry forward up to 10 unused leave days."

Keyword overlap is limited, but the meanings are closely related.

A good embedding model places semantically similar text near each other in vector space.

Retrieval step

The system calculates similarity between the query vector and document vectors.

Common measures include:

• Cosine similarity
• Dot product
• Euclidean distance

The closest chunks become retrieval candidates.

Conceptually:

Query embedding ↓ Compare with stored chunk embeddings ↓ Rank by similarity ↓ Return Top-K chunks

What embeddings do not do

Embeddings do not generate the answer.

Their main job is to help find relevant evidence.

The full flow is:

Documents → chunks → embeddings → vector index

Then:

User query → query embedding → retrieve relevant chunks → LLM → answer

Important limitation

Embeddings are strong at semantic similarity, but they can be weaker for exact identifiers, codes, numbers, or rare terms.

That is why production RAG systems often combine dense embedding search with lexical methods such as BM25.

Interview takeaway

Embeddings convert document chunks and user queries into comparable vectors. During ingestion, document vectors are stored in an index. At query time, the user query is embedded and compared with those stored vectors so the system can retrieve semantically relevant chunks before the LLM generates the answer.