Retrieval-Augmented Generation
Retrieval-augmented generation connects a generative model to a searchable knowledge source. Instead of relying only on model weights, the system fetches relevant documents and places them into the prompt or context.
What is nprobe in IVF search?
In IVF search, nprobe controls how many inverted lists, or clusters, are searched for a query. It is one of the most important parameters for balancing retrieval quality and speed. How it works Suppose an IVF index contains 1,000 clusters. A query embedding is first compared with the cluster centroids. If: nprobe = 1 the search looks only inside the single nearest cluster. If: nprobe = 10 the search looks inside the 10 nearest clusters. So nprobe determines how widely IVF searches around the query. Why does nprobe matter? IVF is approximate because it does not search every cluster. A relevant vector may be stored in a nearby cluster rather than the single closest cluster. If nprobe is too low, that cluster may never be searched. Increasing nprobe gives the search more chances to find those relevant vectors. Low nprobe A low value means: Fewer clusters searched Fewer vector comparisons Lower latency Lower compute cost Greater risk of missing true nearest neighbors This can work well when latency is the highest priority and the clusters are well separated. High nprobe A higher value means: More clusters searched More candidate vectors examined Better recall in many cases Higher latency More compute As nprobe approaches the total number of IVF clusters, the search becomes less approximate because more of the vector space is examined. Example Assume the true answer bearing chunk is stored in the third nearest cluster to the query. With: nprobe = 1 IVF searches only the nearest cluster, so the correct chunk is missed. With: nprobe = 3 the third cluster is also searched, so the chunk can now be retrieved. This is why nprobe directly affects Recall@K. nprobe is not Top K These two settings control different things. nprobe controls how many IVF clusters are searched. Top K controls how many final vectors are returned. For example: nprobe = 20 Top K = 5 means: Search 20 nearby clusters Score candidate vectors from those clusters Return the best 5 results Increasing Top K does not help if the correct vector's cluster was never searched in the first place. How should nprobe be chosen? There is no universal best value. Tune it using representative queries and measure: Recall@K Retrieval latency Throughput Resource usage A practical process is: Start with a small nprobe → increase it gradually → observe recall improvement → stop when the additional recall is no longer worth the extra latency nprobe also depends on the number of clusters An nprobe value should not be interpreted in isolation. For example: Searching 10 clusters out of 100 is very different from searching 10 clusters out of 10,000. So the right value depends on: Total number of IVF lists Dataset size Vector distribution Latency target Required recall RAG impact If nprobe is too low, IVF may miss the answer bearing chunk even when the embedding model represented it correctly. If nprobe is unnecessarily high, retrieval may become slower without meaningfully improving answer quality. So nprobe should be tuned as part of the RAG retrieval system, not treated as a fixed default. Interview takeaway nprobe is the number of IVF clusters searched for each query. A higher nprobe usually improves retrieval recall because more regions of the vector space are examined, but it also increases latency and compute. A lower nprobe is faster but increases the chance of missing relevant vectors. The right value should be chosen by benchmarking recall and latency on the actual RAG workload.
What is Product Quantization (PQ)?
Product Quantization, or PQ , is a vector compression technique used to reduce the memory required to store large embedding collections and to make similarity search more efficient. Instead of storing every vector using all of its original floating point values, PQ represents each vector using a compact set of learned codes. Core idea Suppose an embedding has hundreds of dimensions. PQ first splits the vector into smaller parts called subvectors . For example: 768 dimensional vector → split into 96 subvectors → each subvector has 8 dimensions Each subvector is then approximated by one representative vector from a small learned codebook. Instead of storing the full subvector, PQ stores only the ID of the selected codebook entry. So the original vector becomes something like: Full floating point vector → [code 12, code 87, code 4, code 201, ...] These integer codes require much less memory. How are the codebooks created? For each subvector position, PQ learns a separate codebook, commonly using clustering such as k means. Conceptually: Training vectors → split into subvectors → cluster each subspace → learn representative centroids → encode each subvector using its nearest centroid If a codebook contains 256 centroids, one centroid can be represented by an 8 bit code. This is where much of the compression comes from. How does search work with compressed vectors? At query time, the query is compared with the learned codebook centroids. The system can precompute lookup tables containing distances between the query's subvectors and each centroid. Then, for every stored PQ code, the approximate vector distance can be calculated by combining those lookup values. This avoids reconstructing and comparing every original full precision vector. Why is PQ approximate? PQ replaces each original subvector with the nearest codebook centroid. That replacement introduces quantization error . For example: Original subvector → [0.42, 0.71, 0.18, ...] PQ approximation → nearest learned centroid The approximation is close, but not identical. Therefore the resulting similarity or distance is approximate as well. More aggressive compression usually saves more memory but can reduce nearest neighbor recall. Why is PQ useful in RAG? A RAG system with millions or billions of chunk embeddings can require a large amount of memory. PQ can substantially reduce the storage footprint of those vectors. That can allow: More embeddings to fit in RAM Larger indexes on the same hardware Lower storage cost Faster scanning of compressed candidates The trade off is some loss in retrieval precision. PQ is different from IVF These techniques solve different problems. IVF reduces how many regions of the vector space are searched. PQ reduces how much memory each vector requires and allows approximate distance computation on compressed representations. They are often combined as IVF PQ : IVF → narrow the search to promising clusters PQ → store and compare the vectors inside those clusters efficiently When is PQ most useful? PQ becomes especially attractive when: The vector collection is very large Memory is a major constraint Full precision vectors are expensive to keep in memory A small reduction in recall is acceptable For smaller collections where memory is not a problem, full precision vectors may be simpler and provide better accuracy. Interview takeaway Product Quantization compresses high dimensional vectors by splitting them into subvectors and replacing each subvector with the ID of a nearby learned centroid. This greatly reduces memory usage and enables efficient approximate distance computation, but it introduces quantization error and can reduce retrieval recall. PQ is especially useful for very large vector indexes and is commonly combined with IVF as IVF PQ.
Why is Product Quantization useful at large scale?
Product Quantization is useful at large scale because it can reduce the memory required to store vector embeddings by a very large factor. That matters when a RAG system grows from thousands of vectors to millions or billions of vectors. The memory problem A full precision embedding may contain hundreds or thousands of floating point values. For example, a 768 dimensional vector stored as 32 bit floats requires roughly: 768 × 4 bytes ≈ 3 KB per vector At small scale, that is manageable. At 100 million vectors, however, raw vector storage alone can reach hundreds of gigabytes before adding index structures, metadata, replication, or database overhead. This makes memory one of the biggest constraints in large scale vector search. How PQ helps Product Quantization replaces the original high dimensional vector with a compact sequence of learned codes. Instead of storing hundreds of floating point numbers, the system stores a much smaller set of integer code IDs. Conceptually: Full vector → split into subvectors → replace each subvector with a codebook centroid ID → store compact PQ codes This can dramatically reduce the bytes required per vector. Why that matters operationally Smaller vector representations can provide several advantages: More vectors can fit in RAM. Fewer machines may be required to host the same index. Memory cost can drop substantially. More of the index may remain in fast memory instead of slower storage. Less memory bandwidth is needed while scanning candidate vectors. These improvements become increasingly important as the collection grows. Search can also become more efficient PQ does not always reconstruct every original vector during search. Distance calculations can often use precomputed lookup tables between the query subvectors and PQ codebook centroids. So candidate scoring becomes something like: PQ code → lookup approximate distances → combine them → estimate vector distance This is much cheaper than repeatedly loading and comparing full precision vectors. What is the trade off? PQ is lossy compression. Each original subvector is replaced by the nearest codebook centroid, so some information is lost. That can cause: Approximate distances Slightly different rankings Lower nearest neighbor recall More aggressive compression usually saves more memory but introduces more quantization error. So PQ does not give free compression. It trades some retrieval accuracy for much better memory efficiency and scalability. Why it is especially useful with IVF At very large scale, PQ is often combined with IVF. IVF reduces the number of vectors that need to be searched. PQ reduces the memory and computation required for the vectors inside those searched clusters. The combination becomes: IVF → select promising clusters PQ → store and score vectors inside those clusters compactly This is the idea behind IVF PQ. RAG example Suppose a production RAG system contains 200 million chunk embeddings. Keeping every vector in full precision may require a large distributed memory footprint. With PQ, the system can store compact vector codes, search a larger corpus using the same hardware budget, and still retrieve strong candidates quickly. A reranker can then be used on the smaller candidate set to recover some precision lost during approximate retrieval. When is PQ worth using? PQ is most attractive when: The vector collection is very large RAM or storage cost is a major constraint Search throughput matters Some loss in nearest neighbor recall is acceptable A second stage reranker can refine candidates For smaller datasets, full precision search may be simpler and more accurate. Interview takeaway Product Quantization is useful at large scale because it compresses high dimensional vectors into compact codes, allowing far more embeddings to fit in memory and reducing storage and memory bandwidth costs. It also enables efficient approximate distance computation. The trade off is quantization error, which can slightly reduce retrieval recall, so PQ is most valuable when scalability and memory efficiency matter more than exact vector precision.
What is IVF-PQ?
IVF PQ (Inverted File Index with Product Quantization) is an approximate nearest neighbor search index that combines two techniques: IVF narrows the search to promising groups of vectors. PQ compresses the vectors in those groups into small codes. Together, they make large vector collections more practical to search when storing every full precision embedding is expensive. How IVF PQ is built 1. Learn IVF centroids: Cluster the embedding space into regions, commonly using k means. 2. Assign vectors to inverted lists: Each vector belongs to the list associated with its nearest coarse centroid. 3. Compress the assigned vectors: PQ divides each vector representation into smaller subvectors and replaces them with learned codebook IDs. Many implementations apply PQ to the residual —the difference between a vector and its assigned IVF centroid—to improve approximation quality. 4. Store compact entries: An inverted list contains vector identifiers and their PQ codes instead of needing the entire full precision vector for approximate search. The index now has both a coarse map of where vectors are located and a compact way to represent them. How a query is searched For a RAG query: 1. Convert the user question into an embedding. 2. Find the nearest IVF centroids. 3. Select a limited number of their inverted lists, controlled by nprobe . 4. Score the compressed PQ codes in those lists using efficient approximate distance calculations, often based on precomputed lookup tables. 5. Return the best Top K candidates; optionally rerank them using original vectors or a separate reranking model if those resources are available. Query → nearest IVF clusters → PQ code distance estimates → Top K chunks → optional reranking → LLM Simple example Imagine a RAG knowledge base with millions of document chunk embeddings. An uncompressed flat index may need to compare the query with every full vector. IVF PQ searches only selected clusters. Within those clusters, compact PQ codes let it evaluate candidates without reading each full precision vector. This can reduce both the search workload and the memory footprint. What are the trade offs? IVF PQ introduces two potential sources of recall loss : IVF pruning: A relevant vector may lie in a cluster that was not searched. PQ approximation: A compressed representation may slightly distort distances and change the ranking. Important tuning parameters include: nlist: Number of IVF clusters created during indexing. nprobe: Number of clusters searched at query time. PQ subquantizer count and code size: Control compression strength and distance accuracy. More searched clusters or less aggressive compression can improve recall, but may increase latency or memory usage. When should you use IVF PQ? It is especially useful when the vector corpus is very large, memory is constrained, and some approximation is acceptable. It is not automatically better than HNSW or exact search: compare Recall@K, latency, index size, build time, and update behavior on the actual workload. Interview takeaway IVF PQ combines IVF's cluster based search pruning with PQ's compact vector representation. IVF decides where to search; PQ makes the vectors within those selected regions cheaper to store and compare. The result is memory efficient ANN retrieval at scale, with recall affected by both cluster selection and quantization error.
How does IVF-PQ reduce memory usage?
IVF PQ reduces memory usage primarily by replacing full precision embedding vectors with compact Product Quantization (PQ) codes. IVF organizes those compressed vectors into searchable clusters; it is PQ, not clustering alone, that provides most of the per vector memory savings. How compression reduces storage A standard embedding often stores every dimension as a 32 bit floating point value. For example, a 768 dimensional vector needs: 768 × 4 bytes = 3,072 bytes (about 3 KB) per vector. 10 million vectors ≈ 30.72 GB for the raw embeddings alone, excluding index and database overhead. Now consider a common illustrative PQ configuration: Split each 768 dimensional embedding into 96 subvectors , each containing 8 dimensions. Learn a codebook of 256 representative centroids for each subvector position. Store an 8 bit centroid ID for each subvector instead of its original eight floats. The compressed representation contains 96 bytes per vector rather than 3,072 bytes. That is 32× smaller for the vector representation . Across 10 million vectors, the codes alone require roughly 0.96 GB , rather than 30.72 GB. These figures are illustrative—not a promise of the same total index size reduction in every implementation. What IVF adds IVF assigns vectors to inverted lists based on nearby coarse centroids. In many IVF PQ implementations, PQ encodes the residual vector (the difference between a vector and its assigned centroid), which can improve compression accuracy. The index stores compact PQ codes and identifiers inside those lists. At search time: Query embedding → choose nearby IVF lists → estimate distances using PQ codes → return the closest candidates This avoids keeping and scanning every original full precision vector in the active search index. What still consumes memory? The full IVF PQ index is larger than its PQ codes alone because it may also hold: Vector identifiers and inverted list bookkeeping. IVF centroids and PQ codebooks . Lookup tables or temporary search structures. Additional metadata, filtering indexes, and database overhead. Optionally, original vectors stored separately for reconstruction or accurate reranking. For example, an 8 byte vector ID would add another 8 bytes per entry in an implementation using that ID width. The exact total depends on the library and configuration. What is the trade off? PQ is lossy: the stored codes approximate the original vectors. This introduces quantization error, which can change similarity rankings and reduce Recall@K . IVF introduces another source of approximation when a query searches only some inverted lists. You therefore tune both: PQ code length: More bits generally preserve more information but use more memory. nprobe: Searching more IVF lists generally improves recall but increases latency. If recall is critical, some systems retrieve a larger candidate pool with IVF PQ and rerank it using full precision vectors or another ranking method. Interview takeaway IVF PQ saves memory because it stores short PQ codes instead of full floating point embeddings. IVF narrows which clusters to search; PQ compresses the vectors in those clusters. A 768 dimensional FP32 vector (3,072 bytes) can, for example, become a 96 byte PQ code—32× smaller for the vector representation—while index metadata and optional original vectors add overhead. The main cost is approximate distances and possible retrieval recall loss.
What is the difference between Flat, HNSW, IVF, PQ, and IVF-PQ indexes?
Flat, HNSW, IVF, PQ, and IVF PQ are approaches used in vector search, but they solve different problems. Flat, HNSW, and IVF primarily describe how vectors are searched; PQ describes how vectors are compressed. IVF PQ combines search space pruning with compression. 1. Flat index — exhaustive search A Flat index compares the query embedding with every stored vector and ranks them by the chosen similarity or distance metric. Strength: Exact Top K nearest neighbors under that metric, assuming the same candidate set and full precision scoring. Limitation: Work grows with the number of vectors, so latency and compute can become expensive at scale. Best fit: Small collections, accuracy critical searches, or establishing an exact search benchmark. 2. HNSW — graph based approximate search Hierarchical Navigable Small World (HNSW) connects vectors in a multilayer graph. Search follows promising neighbors instead of examining every vector. Strength: Often excellent recall at low query latency. Limitation: Graph links add memory overhead, and index construction can be expensive. Key settings: M, efConstruction, and query time efSearch. Best fit: Interactive RAG workloads where fast responses and high recall matter and sufficient memory is available. 3. IVF — cluster based approximate search Inverted File Index (IVF) partitions vectors into groups around coarse centroids. At query time, it searches vectors in only the nearest selected groups. Strength: Reduces the number of candidates scored. Limitation: Relevant neighbors can be missed in clusters that were not searched. Key settings: Number of clusters (nlist) and clusters searched per query (nprobe). Best fit: Large datasets where scanning every vector is impractical. Important: IVF alone does not necessarily compress the stored vector values; an IVF Flat index can keep full precision vectors. 4. PQ — compressed vector representation Product Quantization (PQ) splits a vector into subvectors and represents each one using an index into a learned codebook. Strength: Significantly reduces vector storage and enables efficient approximate distance calculations. Limitation: Quantization error can distort distances and change the nearest neighbor ranking. Key settings: Number of subvectors and bits per code. Best fit: Collections where vector memory is a primary constraint. Unlike HNSW or IVF, PQ is fundamentally a compression method . It can be used alone in a search index or combined with other indexing methods. 5. IVF PQ — clustering plus compression IVF PQ combines two stages: 1. IVF: Choose a small number of promising vector clusters. 2. PQ: Score compact vector codes stored within those clusters. Strength: Reduces both the search scope and per vector memory footprint. Limitation: Approximation comes from both unsearched clusters and quantized distances. Best fit: Very large RAG indexes where storing full embeddings in memory is too costly. Some implementations also use residual PQ, compressing the difference between each vector and its assigned IVF centroid. Practical RAG example Suppose a knowledge base holds 10 million chunk embeddings. Flat: Compare the query against all 10 million vectors. HNSW: Navigate a graph to find promising neighbors. IVF: Search vectors in selected nearby clusters. PQ: Represent vectors as compact codes and estimate distances. IVF PQ: Search selected clusters and score their compact codes. After retrieval, a RAG pipeline may rerank candidate chunks before passing the most useful evidence to the LLM. How should you choose? Consider these priorities: Need an exact benchmark or have a small dataset? Start with Flat . Need fast interactive search with strong recall and can afford memory? Benchmark HNSW . Need cluster pruned search with tunable query breadth? Consider IVF . Need to shrink vector storage ? Consider PQ based compression . Need both large scale pruning and compression ? Consider IVF PQ . There is no universally best index. Evaluate Recall@K, p95 retrieval latency, memory, build cost, and update behavior on representative queries. Interview takeaway Flat scans every vector and is exact; HNSW navigates a graph; IVF searches selected clusters; PQ compresses vectors into codes; and IVF PQ combines IVF's cluster pruning with PQ's compression. The central trade off is retrieval recall versus latency, memory, and operational cost.
How do you choose a vector index for a RAG workload?
Choosing a vector index for a RAG workload means finding the best balance between retrieval quality, query latency, memory usage, and operational complexity . There is no universally best index: the right choice depends on the data and how the application is queried. 1. Start with the workload, not the index Before selecting an algorithm, identify: Corpus size: How many chunk embeddings exist now, and how many will exist later? Latency target: What are the acceptable p50 and p95 retrieval times? Recall requirement: How often must the correct evidence appear in the retrieved Top K? Memory budget: Can the complete index fit in RAM? Update frequency: Are documents mostly static, or frequently inserted, updated, and deleted? Filtering and access control: Must retrieval enforce tenant IDs, permissions, dates, or other metadata? Query traffic: How many simultaneous searches must the system support? For enterprise RAG, permission correctness is a hard requirement—not a performance trade off. 2. Match the index to the constraints Flat / exact search Searches all eligible vectors and provides an exact nearest neighbor baseline for the selected distance metric. Useful for small corpora and as a ground truth benchmark. Becomes expensive as the searchable collection grows. HNSW A strong candidate for low latency, high recall interactive retrieval. Often performs well on medium to large collections that fit comfortably in memory. Requires additional graph memory; build and update costs should be measured. IVF (including IVF Flat) Partitions vectors into clusters and searches only selected lists. Useful when scanning everything is too expensive. Tune the number of lists (nlist) and probed lists (nprobe) together. IVF Flat still stores full precision vectors; clustering alone does not provide PQ style compression. PQ and IVF PQ Use compact codes to reduce vector memory. IVF PQ additionally narrows the search to selected clusters. Attractive at very large scale or under strict memory budgets. Compression introduces quantization error, so recall needs careful validation. 3. Evaluate on the actual RAG task Do not choose based only on vector search benchmarks or advertised throughput. Build an evaluation set of representative user questions with known answer bearing document chunks. Compare candidate indexes using: Recall@K / Hit Rate: Was the evidence needed for an answer retrieved? p50/p95 latency: Is retrieval consistently fast enough? Memory and index size: What is the real cost after IDs, graph edges, centroids, and metadata? Build and update cost: How expensive is reindexing or keeping data fresh? Filtered search recall and speed: Does performance hold when access control or metadata filters are applied? End to end answer quality: Do retrieved chunks lead to accurate, grounded responses after reranking and generation? Exact search can serve as the nearest neighbor reference, but exact vector neighbors are not automatically the most useful documents for answering a question. 4. Practical example Suppose a RAG system has 8 million document chunk embeddings , needs low latency interactive answers, and has enough memory for HNSW. A sensible approach is: 1. Build a Flat baseline on representative queries. 2. Benchmark HNSW for recall and p95 latency. 3. Benchmark IVF Flat if build cost, query throughput, or memory pressure warrants it. 4. Consider IVF PQ if the full precision index is too expensive to keep in memory. 5. Test the complete retrieval → optional reranking → generation pipeline. If the application later grows to 100 million embeddings, revisit the decision; the best choice can change with scale. 5. Avoid common mistakes Optimizing speed alone: A fast index that misses answer bearing chunks can make RAG worse. Ignoring filters: A high recall score on unfiltered queries may not reflect real multi tenant traffic. Treating PQ as exact: Compression can change distances and result order. Assuming more recall always improves answers: Extra candidates can increase reranking cost and noise. Choosing once forever: Growth, new embedding models, and changing query patterns may require retuning or rebuilding. Interview takeaway Start with corpus size, recall target, p95 latency, memory, update rate, and filtering requirements. Use Flat as an exact baseline; evaluate HNSW for fast high recall retrieval, IVF for cluster pruned search, and IVF PQ when memory savings are essential. Make the final decision using real RAG queries and end to end grounded answer quality—not index speed alone.
What trade-offs exist between recall, latency, and memory in vector indexing?
In vector indexing, recall, latency, and memory are interconnected design trade offs. Optimizing one often increases the cost of another. For a RAG application, the goal is not simply the fastest vector search. It is to retrieve the answer bearing evidence reliably within the application's response time and infrastructure budgets. What does each metric mean? Recall: How many of the true nearest neighbors an approximate index retrieves compared with an exact search reference. Recall@K measures overlap with the exact Top K; it is different from relevance based retrieval recall against labeled evidence. Latency: Time needed to return search results, commonly assessed with p50 and p95 or p99 latency under realistic traffic. Memory: RAM occupied by the vectors, index structures, codes, identifiers, and associated search data. 1. Higher recall often costs more latency Approximate nearest neighbor indexes deliberately avoid examining every vector. Expanding the search tends to find more of the true nearest neighbors but requires additional work. For example: HNSW: Raising efSearch examines more graph candidates; recall often improves, but queries take longer. IVF: Raising nprobe searches more inverted lists; recall often improves, but more vectors are scored. Flat: Examines every eligible vector and provides exact search under the selected metric, but its work grows with corpus size. The improvement is not necessarily linear: after a point, scanning additional candidates may deliver little extra recall. 2. Better recall can require more memory Some indexes use extra data structures or less aggressive compression to improve search quality. HNSW: More graph connections (larger M) may support stronger recall but increase memory and build cost. Uncompressed vectors: Preserve distances more faithfully but consume more storage than compressed codes. PQ / IVF PQ: Smaller codes can drastically reduce memory, but quantization may distort distances and lower nearest neighbor recall. Not every recall improvement requires more RAM—some tuning parameters mainly increase query time compute instead. 3. Lower memory can affect both recall and latency Compression enables much larger collections to fit in memory and reduces memory bandwidth demand. But it can also reduce accuracy: Quantized vectors approximate the original embeddings. Two candidates that were correctly ordered in full precision may be reversed after compression. Recovering quality with larger candidate pools or exact reranking can add latency and sometimes require original vectors to be stored elsewhere. So the memory savings must be evaluated together with retrieval quality and overall search time. Example in a RAG system Suppose a RAG application contains 20 million chunk embeddings and must retrieve supporting evidence quickly. Possible approaches: Flat: Useful as an exact search baseline; exhaustive comparisons may exceed the query budget. HNSW: Often offers strong recall with low latency but requires additional graph memory. IVF Flat: Saves comparisons by searching selected clusters, while still storing full precision vectors. IVF PQ: Reduces vector memory and scans compressed candidates, accepting additional approximation. The best option depends on actual measurements, not just the dataset size. How should you tune the trade off? Use a representative evaluation set and: 1. Establish exact nearest neighbor results as an ANN recall baseline. 2. Measure Recall@K , p95 latency , memory consumption, and throughput for candidate indexes. 3. Separately measure whether the correct answer bearing chunks are retrieved. 4. Sweep relevant parameters, such as efSearch, nprobe, and PQ code size. 5. Choose a configuration that meets minimum retrieval quality while staying within memory and latency budgets. 6. Check end to end RAG results after reranking and generation, not only vector search scores. Be careful with filters, concurrent requests, and frequent updates: all can change the observed performance. Interview takeaway Higher ANN recall usually requires broader search, increasing latency; greater search quality may also require additional index memory. Compression reduces memory and can improve scale, but introduces distance approximation that can hurt recall. The right balance is the configuration that meets RAG evidence retrieval quality targets within measured p95 latency and memory limits.
What is a vector database collection?
A vector database collection is a logical container that groups related vectors and their associated records so they can be stored, indexed, and searched together. Think of a collection as somewhat similar to a table in a relational database, although the exact features and terminology vary by vector database. What does a collection contain? In a RAG system, each record commonly includes: Vector embedding: A numerical representation of a document chunk. Unique ID: Identifies the vector or chunk. Text or document reference: The original chunk text or a pointer to where it is stored. Metadata: Such as document ID, title, source, page number, tenant ID, or creation date. The collection also has search settings or indexes that determine how vectors are compared and retrieved. Some databases specify vector dimensions and similarity metrics at collection creation; others configure these through individual vector fields or indexes. Example in RAG Suppose a company builds a RAG assistant over its internal documentation. It might create a collection named company knowledge containing: A vector for each document chunk. The corresponding chunk ID and source document ID. Metadata such as department, document type, and updated at. When a user asks, "What is the employee leave policy?", the query is embedded and searched against this collection. The database returns relevant vector records, which the RAG system uses to retrieve supporting text. Why use separate collections? Collections can help organize data that should be managed or searched differently. For example: hr policies: HR document embeddings. product manuals: Product support documentation. technical docs: Engineering knowledge. Separate collections may make sense when datasets need different embedding models, dimensions, index configurations, retention policies, or access arrangements. However, you do not need a separate collection for every department, user, or tenant . A shared collection with metadata filtering can be simpler and more efficient when the vectors are compatible and authorization is correctly enforced. Collection vs vector index These two concepts are not the same: Collection: Organizes vector records and related metadata. Index: Speeds up similarity search over vectors in that collection. A collection may support one or more indexes, depending on the database. For instance, the same logical collection may hold document embeddings while an HNSW or IVF index accelerates nearest neighbor retrieval. Important design considerations When creating a collection for production RAG: Keep vectors in a compatible embedding space for meaningful similarity comparisons. Choose an appropriate similarity metric and indexing method. Preserve stable document and chunk IDs for updates and deletions. Store metadata required for source citations and filtering. Enforce tenant permissions securely; collection boundaries or filters alone are not automatically sufficient access control. Interview takeaway A vector database collection is a logical grouping of vector records, identifiers, and metadata that can be searched and managed together. In RAG, it usually represents a searchable body of document chunks. Collections organize data, while indexes such as HNSW or IVF accelerate retrieval within that data.
How do metadata filters work in a vector database?
Metadata filtering lets a vector database restrict search results using structured information attached to each vector, such as a document's department, language, creation date, or tenant ID. Vector similarity answers "Which chunks are semantically close to this question?" Metadata filtering adds "Which chunks are eligible to be searched or returned?" What metadata is stored? A RAG chunk might have: document id: "policy 42" department: "HR" language: "en" updated at: "2026 09 15" tenant id: "company a" Its embedding represents the content; these fields describe the chunk and help narrow retrieval. How does a filter work? Suppose an employee asks, "What is our maternity leave policy?" The application might search for semantically similar chunks with conditions such as: department = "HR" AND language = "en" Only records satisfying those conditions should be eligible for the final results. The database evaluates conditions using operators such as equality, range comparisons, membership, and logical AND/OR, depending on its features. This prevents otherwise similar but irrelevant records—say, engineering documents—from crowding out HR policy chunks. Pre filtering and post filtering Two common execution approaches are: Pre filtering: Apply metadata conditions before or during nearest neighbor search. The search focuses on eligible records, but performance depends on the database's indexing and query strategy. Post filtering: Search for vector candidates first, then discard those failing the filter. This can be convenient, but if few candidates qualify, the system may return fewer than the requested Top K or miss useful eligible results. Databases may combine these strategies or implement filtering inside their ANN algorithms. A filter in the API does not necessarily reveal the precise execution order. Why filters matter for RAG Metadata filters can: Improve relevance: Search only the appropriate product, department, language, or document version. Reduce wasted search: Avoid scoring clearly ineligible records when the engine supports efficient filtered search. Support freshness: Restrict to current documents or particular date ranges. Enforce retrieval scope: Keep search constrained to the right organization or tenant when backed by proper authorization. Critical security distinction A user provided filter is not sufficient authorization . For multi tenant RAG, the backend must derive tenant and permission constraints from verified identity and enforce them throughout retrieval. Never trust an arbitrary tenant id supplied by a caller. Database level access controls or equivalent server side enforcement are important safeguards. Practical pitfalls Highly selective filters can reduce the candidate pool and change ANN recall or latency. Missing or inconsistent metadata can accidentally hide relevant chunks. Post filtering after a small Top K can leave too few results; over retrieving may help but costs more. Frequent document updates require metadata and indexed content to stay synchronized. For production evaluation, measure Recall@K and latency on queries with realistic filters—not just unrestricted vector searches. Interview takeaway Metadata filters combine semantic similarity with structured conditions to control which vector records can be retrieved. They may be applied before, during, or after ANN search, and their placement affects recall and latency. In enterprise RAG, filters improve relevance and retrieval scoping, but authorization must be enforced independently and securely.