What is a vector database in RAG?

A vector database is a database designed to store, index, and search vector embeddings efficiently.

In RAG, it is commonly used to find document chunks that are semantically similar to a user's query.

Why RAG needs it

During ingestion, each document chunk is converted into an embedding.

For example:

Chunk: "Enterprise customers can request a refund within 45 days."

Embedding: [0.14, -0.37, 0.81, ...]

The vector database stores that embedding together with information about the chunk.

Later, the user asks:

"What is the refund period for enterprise customers?"

The query is also converted into an embedding.

The vector database compares the query vector with stored document vectors and returns the closest matches.

What does it usually store?

A vector record commonly contains:

• The embedding vector
• The original chunk or a reference to it
• Document ID
• Chunk ID
• Page or section
• Source information
• Metadata such as tenant, product, language, date, or permissions

The vector helps with semantic retrieval.

The metadata helps with filtering, traceability, and access control.

How is this different from a normal database?

A relational database is very good at exact structured queries.

Example:

customer_id = 48291

or:

status = 'active'

A vector database is designed for similarity queries such as:

"Find content whose meaning is closest to this query."

That is useful when the query and document do not use exactly the same words.

Example

User query:

"Can I move unused vacation to next year?"

Document chunk:

"Employees may carry forward up to 10 days of unused annual leave."

A keyword-only lookup may struggle if the vocabulary differs.

A vector search can retrieve the chunk because the embeddings capture semantic similarity.

What role does indexing play?

If you have only a few hundred vectors, you could compare the query against every vector.

But with millions of chunks, that becomes expensive.

Vector databases use specialized indexes such as:

• HNSW
• IVF
• IVF-PQ

These make nearest-neighbor search much faster, often by trading a small amount of recall for much lower latency.

Is a vector database always required?

No.

For small datasets, a normal database with vector support may be enough.

Some relational databases can store vectors and perform similarity search directly.

The important capability is not the product label "vector database."

The important capabilities are:

• Vector storage
• Similarity search
• Efficient indexing
• Metadata filtering
• Updating and deleting vectors

Where it sits in the RAG flow

A simplified flow is:

Documents → chunking → embeddings → vector database

Then at query time:

User query → query embedding → vector search → relevant chunks → LLM

The vector database belongs mainly to the retrieval layer.

It does not generate the final answer.

Interview takeaway

A vector database stores embeddings and enables efficient nearest-neighbor search over them. In RAG, it is used to retrieve document chunks that are semantically similar to a query, often together with metadata for filtering and source tracking. It is part of the retrieval layer, while the LLM uses the retrieved evidence to generate the final answer.