Retrieval-Augmented Generation

Retrieval-augmented generation connects a generative model to a searchable knowledge source. Instead of relying only on model weights, the system fetches relevant documents and places them into the prompt or context.

What is ANN?

ANN stands for Approximate Nearest Neighbor search. It is a family of techniques used to find vectors that are very close to a query vector without comparing the query against every vector in the database. The key word is approximate . ANN is designed to return highly likely nearest neighbors much faster than exhaustive search, while accepting that it may occasionally miss the mathematically exact nearest result. Why ANN is useful Suppose a RAG system contains 10 million document embeddings. With exact search, the system may need to compare the query embedding with all 10 million vectors. That can be expensive and slow. ANN uses an index to narrow the search to promising regions or candidates. So instead of: Query → compare with every vector → find nearest results ANN works more like: Query → navigate an index → inspect likely candidates → return near nearest results Why is it called approximate? Because ANN does not guarantee that every returned result is the exact mathematical nearest neighbor. For example, imagine the true closest chunks are ranked: 1. Chunk A 2. Chunk B 3. Chunk C An ANN search might return: 1. Chunk A 2. Chunk C 3. Chunk D It found very close results quickly, but missed Chunk B. That is the trade off. Speed vs recall ANN introduces an important balance: More aggressive search → lower latency → possibly lower recall More thorough search → higher recall → more latency and computation The goal is usually to find a configuration where retrieval is fast while Recall@K remains high enough for the RAG application. Common ANN approaches Popular vector indexing methods include: HNSW IVF IVF PQ They achieve approximate search differently. HNSW uses a graph structure. IVF divides vectors into groups or regions. PQ compresses vector representations and is often combined with IVF for large scale search. ANN in a RAG pipeline A typical flow is: User question → query embedding → ANN search → Top K candidate chunks → optional reranking → best context → LLM ANN usually handles the fast first stage retrieval. A reranker can then inspect the smaller candidate set more carefully. This combination is common because ANN gives speed and recall, while reranking improves precision. Is ANN always necessary? No. If the collection is small, exact nearest neighbor search may be simple and fast enough. ANN becomes valuable when the corpus grows large enough that scanning every vector creates unacceptable latency or cost. So ANN is mainly a scalability technique. Important RAG implication Fast retrieval is not useful if ANN misses the answer bearing chunk too often. That is why ANN should be evaluated using both: Retrieval latency Recall@K If recall drops too much, the final LLM answer may fail even though the vector search itself is very fast. Interview takeaway Approximate Nearest Neighbor search finds vectors that are likely to be among the closest matches without scanning the entire vector collection. It greatly reduces retrieval latency at large scale, but may occasionally miss the exact nearest neighbor. In RAG, ANN is commonly used for fast candidate retrieval, with parameters tuned to balance recall, latency, memory, and cost.

Why is Approximate Nearest Neighbor search used in RAG?

Approximate Nearest Neighbor search is used in RAG because exact vector search becomes expensive as the number of stored embeddings grows. The goal is simple: Retrieve highly relevant chunks quickly without comparing the query against every vector in the database. Why exact search becomes expensive Suppose your RAG system contains 20 million chunk embeddings. With exact nearest neighbor search, every query may require comparing the query vector with all 20 million vectors. That gives mathematically exact results, but the latency and compute cost can become too high for production systems. ANN reduces that work. How ANN helps ANN uses a vector index to search only promising areas or candidates. Instead of: Query → compare with every vector → sort all results → return Top K ANN works more like: Query → navigate index → inspect likely neighbors → return Top K candidates This can reduce retrieval time dramatically. Why approximation is acceptable in RAG RAG usually does not require the single mathematically closest vector. It needs a set of sufficiently relevant chunks that contains the answer bearing evidence. For example, suppose the exact nearest neighbors are: 1. Chunk A 2. Chunk B 3. Chunk C 4. Chunk D 5. Chunk E ANN may return: 1. Chunk A 2. Chunk C 3. Chunk B 4. Chunk F 5. Chunk D The ordering is not perfectly exact, but the relevant evidence may still be present. For many RAG workloads, that is a very good trade. ANN makes large scale RAG practical ANN is especially useful when you have: Millions of chunks High query traffic Strict latency targets High dimensional embeddings Limited compute budget Without ANN, vector retrieval can become the bottleneck. The trade off: recall vs latency ANN is faster because it does less search work. But searching fewer candidates can reduce recall. If ANN misses the chunk containing the correct answer, the LLM cannot recover that evidence later. So you normally tune ANN parameters to balance: Recall@K Search latency Memory usage Throughput Example Imagine the correct chunk is the 4th nearest vector. A very aggressive ANN configuration may search too narrowly and miss it. A more thorough configuration may inspect more candidates and retrieve it, but take longer. So the right ANN settings depend on how much retrieval quality you are willing to trade for speed. ANN works well with reranking A common production design is: ANN search → retrieve 30 candidates → rerank them → keep best 5 → send to LLM ANN handles fast broad retrieval. The reranker then improves precision on the smaller candidate set. This is often more efficient than performing extremely expensive exact search across the entire database. Common ANN indexes Examples include: HNSW IVF IVF PQ Different indexes offer different trade offs in: Recall Query latency Memory Build time Update behavior When ANN may not be needed If your RAG corpus contains only a few thousand vectors, exact search may already be fast enough. ANN becomes more valuable as the vector collection grows. Interview takeaway Approximate Nearest Neighbor search is used in RAG to make vector retrieval scalable. Instead of comparing every query with every stored embedding, ANN searches a smaller set of promising candidates, greatly reducing latency and compute cost. The trade off is that some true nearest neighbors may be missed, so ANN should be tuned and evaluated using both retrieval recall and latency.

What is exact nearest-neighbor search?

Exact nearest neighbor search finds the true closest vectors to a query by comparing the query against every eligible vector in the collection. It is also called exhaustive search or brute force vector search. How it works Suppose your RAG system has 100,000 document embeddings. A user query is converted into an embedding. Exact search then: Query embedding → compare against all 100,000 vectors → calculate similarity or distance → sort the results → return the true Top K nearest vectors Because every candidate is checked, the returned nearest neighbors are exact for the chosen similarity metric. Why exact search is useful Its biggest advantage is accuracy. There is no approximation caused by skipping parts of the vector space. So if Chunk A is mathematically the closest vector, exact search will find it. This makes exact search useful for: Small vector collections Offline evaluation Benchmarking ANN indexes Workloads where maximum retrieval recall matters more than latency Example Suppose the true nearest chunks are: 1. Chunk A 2. Chunk B 3. Chunk C Exact search evaluates every stored vector and returns those true nearest results. An approximate method might instead return: 1. Chunk A 2. Chunk C 3. Chunk D because it did not inspect every candidate. The main drawback Exact search becomes more expensive as the vector collection grows. If you have: 10,000 vectors → usually manageable 10 million vectors → every query may require millions of vector comparisons 100 million vectors → exhaustive comparison can become a serious latency and compute bottleneck The cost also increases with embedding dimensionality. Exact search vs indexing Exact search is often associated with a flat or brute force index. A flat index does not use approximation to avoid large parts of the search space. It performs the necessary distance calculations across the eligible vectors and returns exact nearest neighbors. Approximate indexes such as HNSW or IVF are designed to reduce that work. Why exact search matters in RAG evaluation Exact search is useful as a reference point. For example: Exact search Recall@10 = ground truth nearest neighbors ANN search → compare its Top 10 against the exact Top 10 This helps measure how much recall an approximate index loses in exchange for lower latency. Without an exact baseline, it is harder to know whether poor retrieval comes from: The embedding model The ANN index ANN parameter settings Does exact nearest neighbor mean the answer is correct? No. Exact search guarantees the nearest vectors according to the embedding space and similarity metric. It does not guarantee that those vectors are actually relevant to the user's intent. If the embedding model represents the query poorly, exact search can still retrieve the wrong semantic content. So exact vector search guarantees search correctness, not RAG answer correctness. When should you use it? Exact search makes sense when: The dataset is small Query volume is low Maximum recall is required You need an evaluation baseline Latency is less important As the corpus grows, ANN often becomes more practical. Interview takeaway Exact nearest neighbor search compares a query vector with every eligible stored vector and returns the mathematically closest Top K results for the chosen similarity metric. It provides maximum search recall and is valuable for small datasets and ANN benchmarking, but its compute and latency grow with collection size, which is why large RAG systems often use approximate search instead.

What is the difference between exact search and ANN search?

Exact search and ANN search both try to find vectors that are closest to a query vector, but they differ in how much of the vector collection they inspect. The main trade off is: Exact search maximizes search accuracy. ANN search maximizes speed and scalability. Exact search Exact nearest neighbor search compares the query vector against every eligible vector in the collection. Then it calculates the chosen similarity or distance metric and returns the true nearest Top K results. For example: Query → compare with all 1,000,000 vectors → sort by similarity → return exact Top 5 Because every candidate is checked, exact search does not lose recall because of indexing shortcuts. ANN search Approximate Nearest Neighbor search avoids comparing the query with every vector. Instead, it uses an index such as HNSW or IVF to quickly locate promising candidates. For example: Query → navigate ANN index → inspect a much smaller candidate set → return approximate Top K This dramatically reduces search latency, but the index may occasionally miss a true nearest neighbor. Example Assume the true nearest vectors are: 1. A 2. B 3. C 4. D 5. E Exact search returns: A, B, C, D, E An ANN search might return: A, B, D, C, F The ANN result is still very close, but E was missed and F appeared instead. Which one is faster? ANN is usually much faster for large vector collections because it avoids exhaustive comparisons. Exact search becomes increasingly expensive as: Number of vectors grows Embedding dimension grows Query traffic increases For small collections, however, exact search may already be fast enough. Which one has better recall? Exact search provides the true nearest neighbors for the chosen distance metric. So, from a vector search perspective, its nearest neighbor recall is effectively the reference baseline. ANN can have slightly lower recall because it searches only part of the space. How much recall it loses depends on: Index type Index parameters Search parameters Dataset structure Why ANN is common in RAG A production RAG system may contain millions or tens of millions of chunks. Scanning every vector for every query can make retrieval too slow. ANN gives a practical balance: High retrieval recall + low latency + better scalability That is usually more useful than mathematically perfect search that is too expensive to serve. Why exact search is still important Exact search is useful for: Small datasets Offline evaluation Creating an ANN baseline Measuring ANN Recall@K Debugging retrieval quality For example, if exact search retrieves the correct evidence but ANN does not, the issue is likely related to the ANN index or its configuration rather than the embedding model alone. ANN often works with reranking A common RAG design is: ANN → retrieve 20–50 candidates → reranker → select best 5 → send context to LLM ANN provides speed and broad recall. Reranking provides a more precise ordering of the smaller candidate set. Interview takeaway Exact search compares the query against every eligible vector and returns the true nearest neighbors, giving maximum search recall but increasing latency as the dataset grows. ANN searches only promising candidates using an index, making retrieval much faster and more scalable at the cost of occasionally missing a true nearest neighbor. In production RAG, ANN is usually preferred at scale, while exact search is valuable for small datasets and as an evaluation baseline.

What is the HNSW algorithm?

HNSW stands for Hierarchical Navigable Small World . It is a graph based Approximate Nearest Neighbor (ANN) algorithm used to search high dimensional vectors quickly. In RAG, HNSW is commonly used to retrieve semantically similar document chunks from large embedding collections. Core idea HNSW represents vectors as nodes in a graph. Each node is connected to a small number of nearby nodes. Instead of comparing the query with every vector, the search starts from an entry point and moves through the graph toward vectors that are progressively closer to the query. Think of it like navigating a map: Start far away Follow roads toward a promising region Move to increasingly closer neighbors Stop when no better nearby candidates are found Why is it hierarchical? HNSW uses multiple graph layers. The upper layers contain fewer nodes and longer range connections. The lower layers contain more nodes and more detailed local connections. Search usually works like this: Query → start at a high layer → make large jumps toward the right region → move down one layer → search more precisely → continue until the bottom layer → return nearest candidates This hierarchy helps HNSW avoid exploring the entire graph. Simple example Imagine 1 million document embeddings. A flat exact search may compare the query with all 1 million vectors. HNSW may instead: Enter the graph through a starting node Make large jumps at upper layers Reach the correct neighborhood Explore only nearby candidates at lower layers This can dramatically reduce query latency. Why “Small World”? The graph is designed so that most nodes can be reached through a relatively small number of hops. Some connections link nearby vectors. Others act like shortcuts across distant parts of the graph. These shortcut like connections help search move quickly across the vector space. Important HNSW parameters Two commonly discussed parameters are: M : controls how many graph connections each node can have efSearch : controls how many candidates are explored during a query Higher efSearch usually means: Better recall More computation Higher latency Lower efSearch usually means: Faster search Possibly lower recall Another build time parameter, often called efConstruction , controls how much effort is used while building the graph. Higher values can improve index quality but increase index build time. Why HNSW is popular in RAG HNSW is attractive because it often provides: High recall Low query latency Good performance on large vector collections No need to scan every vector That makes it a strong default choice for many semantic search and RAG workloads. What are the trade offs? HNSW is fast, but it is not free. It can require significant memory because the graph stores connections between vectors. Index construction can also be relatively expensive. So the trade off is often: Higher memory + more index build cost → fast search with strong recall Where HNSW fits in RAG A typical flow is: User question → query embedding → HNSW search → Top K candidate chunks → optional reranking → final context → LLM HNSW is responsible for fast candidate retrieval, not for generating the answer. Interview takeaway HNSW is a graph based ANN algorithm that organizes vectors into multiple layers of navigable connections. Search begins in sparse upper layers for fast long range movement, then descends into denser lower layers for precise local search. It is popular in RAG because it usually provides high recall and low latency, with the main trade offs being memory usage and index build cost.

What does 'Small World' mean in HNSW?

In HNSW, “Small World” means the graph is structured so that even very distant regions can usually be reached through only a small number of hops. That matters because vector search becomes much faster when the algorithm can move across the graph through shortcuts instead of checking every node. The basic idea Imagine every vector is a node in a graph. Each node is connected to some nearby nodes. If the graph had only local connections, searching could be slow because the algorithm might need to move through many intermediate nodes. A small world graph also contains longer range connections. These act like shortcuts. So search can move: Far region → shortcut → closer region → local neighbors → nearest candidates Simple analogy Think about traveling between cities. If you only had local roads, you might need to pass through dozens of towns. If you also have highways, you can travel quickly to the right region and then use local roads near the destination. HNSW uses a similar idea. The upper layers act like highways. The lower layers provide more detailed local navigation. Why this helps vector search Suppose the query vector is currently near node A, but the best matches are in a distant region. If the graph contains useful long range links, the search can jump toward that region quickly. Without such shortcuts, it may need to explore many more nodes. That is why the small world property helps reduce search latency. How this connects to HNSW’s hierarchy HNSW does not use just one graph. It uses multiple layers. Upper layers: Fewer nodes Longer range connections Faster movement across the vector space Lower layers: More nodes More local connections More precise search The search starts high, moves toward a promising area, then descends layer by layer. This combines: Small world connectivity + hierarchical navigation Is every connection long range? No. Most useful connections are still between relatively nearby vectors. The graph works because it combines: Local links for precision Some longer range links for fast navigation That balance is what gives the graph its small world behavior. Why it matters in RAG In a RAG system with millions of embeddings, the retriever needs to reach relevant regions of the vector space quickly. Small world connectivity helps HNSW find strong candidates without scanning the entire collection. That gives HNSW its practical advantage: Low latency High recall Good scalability Interview takeaway “Small World” in HNSW means the graph is connected so that distant regions can usually be reached in only a few hops. HNSW achieves this through a mix of local neighbor links and longer range shortcut connections, organized across multiple layers. This lets vector search move quickly toward the right region and then refine the search locally.

How does HNSW search work?

HNSW search works by navigating a multi layer graph from a coarse global search to a more detailed local search. The goal is to reach vectors close to the query without comparing the query with every vector in the index. 1. Start from an entry point HNSW keeps an entry point in one of the highest graph layers. The query begins there. At these upper layers, the graph contains relatively few nodes and longer range connections, so the search can move quickly across the vector space. 2. Greedily move toward a closer node At the current layer, HNSW compares the query with the current node and its connected neighbors. If one of those neighbors is closer to the query, the search moves there. This continues until none of the connected neighbors is closer. At that point, the algorithm has reached a local best position for that layer. 3. Move down one layer The best node found in the current layer becomes the starting point for the next lower layer. Then the same process repeats. So the search gradually changes from: Large jumps across the space → smaller regional movement → detailed local search This is why the hierarchy is useful. 4. Search more broadly at the bottom layer At the lowest layer, HNSW usually explores more than just one greedy path. It keeps a candidate set of promising nodes and continues exploring nearby connections. This broader search helps avoid stopping too early at a locally good but not truly strong neighborhood. The amount of exploration is controlled by a search parameter commonly called efSearch . What does efSearch do? efSearch controls how many candidate neighbors HNSW is willing to explore during a query. Higher efSearch usually gives: Better recall More distance calculations Higher latency Lower efSearch usually gives: Faster retrieval Less computation Greater chance of missing a true nearest neighbor So efSearch is one of the main knobs for tuning HNSW search quality. Simple example Suppose your index contains 5 million embeddings. Instead of checking all 5 million vectors, HNSW may: Start from a sparse upper layer Follow long range links toward the query region Descend through several layers Explore a relatively small candidate set at the bottom Return the closest Top K candidates it found That is what makes it an Approximate Nearest Neighbor algorithm. Why does this work well? The graph combines: Local neighbor connections Longer range shortcut connections Multiple layers The upper layers help the search reach the right region quickly. The lower layer helps refine the result. This gives HNSW its typical combination of high recall and low query latency. Where HNSW fits in RAG A RAG retrieval flow may be: User query → query embedding → HNSW graph search → Top K chunks → optional reranking → final context → LLM HNSW is responsible for quickly finding promising vector candidates. It does not judge whether the final answer is correct. Interview takeaway HNSW search starts from an entry point in a sparse upper layer, greedily moves toward vectors closer to the query, then descends layer by layer. At the bottom layer it explores a broader candidate set, controlled by efSearch, before returning the best Top K matches. This hierarchical graph navigation avoids scanning the full vector collection while still achieving high recall.

What are the advantages and disadvantages of HNSW?

HNSW is popular because it usually gives an excellent balance of high recall and low query latency , but that performance comes with costs—especially memory usage and index building overhead. Advantages of HNSW 1. Fast query performance HNSW can navigate toward relevant vectors without scanning the full collection. That makes it well suited for interactive RAG systems where retrieval latency matters. 2. High recall With good parameter settings, HNSW often retrieves results very close to exact nearest neighbor search. Increasing parameters such as efSearch can improve recall further, although latency also increases. 3. Works well for high dimensional embeddings Modern embedding models can produce hundreds or thousands of dimensions. HNSW is widely used because graph navigation can remain efficient even in these high dimensional spaces. 4. Good fit for RAG retrieval RAG usually needs a small Top K candidate set quickly. HNSW works well for this pattern: Query embedding → HNSW search → Top K candidates → optional reranking → LLM 5. Tunable quality vs latency You can trade search speed for retrieval quality. For example: Higher efSearch → explore more candidates → better recall → higher latency Lower efSearch → faster search → possibly lower recall This makes HNSW flexible for different workloads. Disadvantages of HNSW 1. High memory usage HNSW stores not only the vectors but also graph connections between nodes. These connections add memory overhead. For very large collections, the graph can become expensive to keep in memory. This is one reason HNSW may be less attractive for tens or hundreds of millions of vectors when memory is constrained. 2. Index construction can be expensive Building a high quality HNSW graph takes work. Parameters such as M and efConstruction affect: Build time Memory Graph quality Retrieval recall Higher quality construction usually requires more computation. 3. More tuning parameters HNSW has several important parameters. Common ones include: M efConstruction efSearch Poor settings can either waste memory and compute or hurt retrieval recall. So production systems should benchmark these values rather than rely on defaults blindly. 4. It is still approximate HNSW is an ANN algorithm. Even with strong recall, it does not guarantee that every true nearest neighbor will be returned. If the answer bearing chunk is missed, the downstream LLM cannot use it. That is why HNSW should be evaluated against an exact search baseline where possible. 5. Updates and deletions can be more complex than flat search Adding new vectors is supported by many HNSW implementations, but maintaining a graph under frequent changes requires extra work. Deletion behavior, graph cleanup, and rebuilding strategies depend on the database or library being used. For highly dynamic datasets, update characteristics should be tested carefully. When is HNSW a good choice? HNSW is often a strong choice when you need: Low retrieval latency High recall Moderate to large vector collections Enough memory for the graph Interactive semantic search or RAG When might another index be better? You may consider other approaches when: Memory is severely constrained The collection is extremely large Compression is important Bulk/static indexing is more important than online search flexibility Indexes such as IVF or IVF PQ can sometimes be more memory efficient at very large scale. Interview takeaway HNSW is popular because it delivers fast vector search with high recall and works very well for RAG style Top K retrieval. Its main disadvantages are memory overhead, relatively expensive index construction, parameter tuning, and the fact that it remains approximate. It is often a strong default when low latency and recall matter more than minimizing memory usage.

What is Inverted File Index (IVF)?

Inverted File Index, or IVF , is an Approximate Nearest Neighbor indexing method that speeds up vector search by dividing the vector space into groups and searching only the most relevant groups for each query. Instead of comparing a query with every stored vector, IVF first asks: Which cluster is this query closest to? Then it searches vectors mainly inside a small number of nearby clusters. Core idea Suppose a RAG system contains 10 million embeddings. A flat search may compare the query with all 10 million vectors. IVF first partitions those vectors into many clusters. For example: 10 million vectors → divided into 1,000 clusters → each cluster contains vectors from a similar region of the embedding space At query time, IVF finds the cluster centroids closest to the query and searches only those selected clusters. How is the index created? IVF typically uses a clustering step, commonly based on k means. The process is roughly: Embeddings → learn cluster centroids → assign each vector to its nearest centroid → maintain a list of vectors for each centroid Each centroid therefore acts like an entry point to a region of the vector space. The lists associated with those centroids are the “inverted lists.” How does search work? For a query: Query → create query embedding → compare query with cluster centroids → choose the nearest clusters → search vectors inside those clusters → return Top K results Because only part of the database is searched, retrieval can be much faster than exhaustive search. Why is IVF approximate? A relevant vector may exist in a cluster that was not selected. If IVF searches only one cluster, it is fast but may miss good candidates near cluster boundaries. Searching more clusters improves recall, but also increases latency. This trade off is controlled by a parameter commonly called nprobe . Higher nprobe: Searches more clusters Usually improves recall Increases query cost Lower nprobe: Searches fewer clusters Reduces latency May miss relevant neighbors Important index parameter: number of clusters IVF also depends on how many clusters are created. Too few clusters: Each cluster becomes large More vectors must be scanned inside each selected cluster Too many clusters: Clusters become smaller Index training and tuning become more sensitive You may need to probe more clusters to maintain recall So cluster count and nprobe should be tuned together. Why IVF is useful in RAG IVF is useful when: The vector collection is large Exact search is too slow Memory efficiency matters Some approximation is acceptable You want explicit control over how much of the index is searched It can also be combined with Product Quantization to create IVF PQ , which reduces memory usage further. IVF vs HNSW at a high level HNSW navigates a graph. IVF narrows the search using clusters. HNSW is often chosen for strong low latency recall when memory is available. IVF can be attractive for very large collections, especially when combined with compression techniques such as PQ. The better choice depends on scale, latency targets, recall requirements, memory, and update patterns. Interview takeaway IVF is a vector index that partitions embeddings into clusters and stores vectors in inverted lists associated with cluster centroids. At query time, it searches only the nearest clusters instead of the full vector collection. This reduces latency, but recall depends on how well the vectors are clustered and how many clusters are searched, usually controlled by nprobe.

How does IVF work?

IVF works by first dividing the vector space into clusters, then searching only the clusters that are most likely to contain the nearest vectors. The goal is to avoid scanning the entire vector collection for every query. 1. Train cluster centroids During index creation, IVF learns a set of representative cluster centers, usually with a clustering algorithm such as k means. For example: 1,000,000 embeddings → learn 1,000 centroids Each centroid represents one region of the vector space. 2. Assign each vector to a cluster Every stored vector is compared with the centroids and assigned to its nearest one. So the index becomes a set of inverted lists: Centroid 1 → vectors assigned to cluster 1 Centroid 2 → vectors assigned to cluster 2 Centroid 3 → vectors assigned to cluster 3 and so on. These lists are what IVF searches later. 3. Embed the user query At retrieval time, the user's question is converted into a query embedding using the same compatible embedding space as the stored vectors. 4. Find the nearest centroids Instead of immediately searching all stored vectors, IVF first compares the query with the cluster centroids. It selects the centroids closest to the query. For example: 1,000 total clusters → query is closest to clusters 17, 204, and 611 5. Search only selected clusters IVF then compares the query with vectors inside those selected inverted lists. So instead of scanning 1 million vectors, the system might search only a much smaller candidate set. This is where most of the speedup comes from. 6. Return the Top K matches The candidate vectors are scored using the configured similarity or distance metric. The closest Top K vectors are returned to the RAG pipeline. The flow becomes: Query → query embedding → nearest centroids → selected inverted lists → candidate vector search → Top K chunks What controls how many clusters are searched? A parameter commonly called nprobe determines how many inverted lists are searched for each query. For example: nprobe = 1 → search only the nearest cluster nprobe = 10 → search the 10 nearest clusters Higher nprobe usually means: Better recall More vectors examined Higher latency Lower nprobe usually means: Faster search Less computation Greater risk of missing relevant vectors Why can IVF miss a relevant vector? Consider a query located near the boundary between two clusters. The true nearest document vector may be stored in the neighboring cluster. If IVF searches only one cluster, that vector may never be considered. Searching multiple nearby clusters reduces this problem. Example in RAG Suppose a knowledge base contains 5 million chunk embeddings divided into 5,000 IVF clusters. A user asks: "What is the refund period for enterprise customers?" The query embedding is closest to a few clusters related to policies and customer terms. IVF searches only those clusters, retrieves the strongest candidate chunks, and passes them to the next RAG stage. IVF is a two stage search A useful way to remember IVF is: Stage 1: coarse search Find the nearest cluster centroids. Stage 2: fine search Search the vectors inside those selected clusters. That is why IVF is often described as a coarse to fine retrieval method. Interview takeaway IVF first clusters the vector space and assigns every vector to an inverted list associated with a centroid. At query time, it finds the nearest centroids, searches only their lists, and returns the best Top K candidates. This makes retrieval much faster than scanning the full collection, while nprobe controls the trade off between recall and latency.