What is vector indexing?

Vector indexing is the process of organizing embedding vectors so similarity searches can run much faster.

Without an index, the system may have to compare the query vector with every stored vector.

That is called a brute-force or exact scan.

For a small collection, this may be acceptable.

For millions of vectors, it can become too slow.

Why indexing is needed

Suppose your RAG system contains 5 million document chunks.

A user query becomes one embedding.

If the system compares that query with all 5 million vectors every time, retrieval may become expensive and slow.

A vector index creates a structure that helps the database search only the most promising parts of the vector space.

Simple idea

Without an index:

Query vector → compare with every vector → sort by similarity → return Top-K

With an index:

Query vector → navigate the index → inspect likely nearby candidates → return Top-K

The index reduces how much work must be done per query.

Exact vs approximate search

Some vector search methods return the mathematically exact nearest neighbors.

But many production vector indexes use Approximate Nearest Neighbor, or ANN, search.

ANN intentionally avoids checking every vector.

That makes retrieval much faster, but it introduces a trade-off:

Higher speed → possibly slightly lower recall

The true nearest vector may occasionally be missed.

Common vector indexes

Popular approaches include:

• HNSW
• IVF
• IVF-PQ

They organize vectors in different ways.

HNSW builds a graph that lets search move efficiently toward nearby vectors.

IVF groups vectors into regions or clusters so the system searches only selected regions.

PQ compresses vectors to reduce memory and speed up large-scale search.

The best index depends on the workload.

Does an index change the embeddings?

No.

The embedding model still produces the same vectors.

The index is an additional data structure built around those vectors to make searching them efficient.

Think of it like an index in a book:

The information does not change.

The index helps you reach the relevant information faster.

What affects index performance?

Important factors include:

• Number of vectors
• Embedding dimension
• Similarity metric
• Required recall
• Query latency target
• Memory available
• Update frequency

A configuration optimized for maximum recall may use more memory or take longer to search.

A configuration optimized for very low latency may accept slightly lower recall.

When is an index unnecessary?

For a small dataset, exact search can be perfectly reasonable.

An approximate vector index becomes more valuable as the number of vectors grows and full scans become too expensive.

So indexing is mainly a scalability optimization.

RAG impact

If the vector index is poorly configured, the correct document chunk may exist in the database but never appear in the retrieved Top-K.

That means index configuration directly affects RAG retrieval quality.

You should therefore evaluate both:

• Search latency
• Retrieval recall

not latency alone.

Interview takeaway

Vector indexing organizes embeddings so nearest-neighbor retrieval can be performed efficiently without scanning every stored vector. At large scale, indexes such as HNSW or IVF often use approximate search to reduce latency, trading a small amount of recall for much faster retrieval. The embeddings themselves do not change—the index is the structure that makes searching them scalable.