Retrieval-Augmented Generation

Retrieval-augmented generation connects a generative model to a searchable knowledge source. Instead of relying only on model weights, the system fetches relevant documents and places them into the prompt or context.

When should a multimodal model be used after image retrieval?

Use a multimodal (vision language) model after image retrieval when answering the question requires inspecting visual information that text extraction, metadata, or captions do not capture reliably. Retrieving the correct image and understanding its contents are separate tasks. Beginner friendly example A technician asks: “Which switch in this wiring diagram connects to the backup battery?” A retriever finds the correct wiring diagram. Its stored caption only says “Power distribution diagram,” so a text only LLM cannot identify the relevant switch. A vision capable model can examine the retrieved diagram , read its labels and connections, and answer with the switch identifier— provided the image is legible and the visual evidence supports it . By contrast, if the question is “Which page contains the wiring diagram?” , image metadata and page references may be enough; an additional vision model call is unnecessary. When is visual interpretation needed? Spatial or structural details: Which object is above another, where a button appears, or how components connect. Charts and diagrams: Reading a plotted trend, interpreting a flowchart, or following connections that captions omit. Screenshots or photographs: Recognizing visible interface states, equipment conditions, or product features. Text embedded in images: Reading labels where trustworthy OCR output is missing, although small text can remain difficult. Questions combining modalities: Connecting instructions from nearby text with an image's visual details. Practical decision flow 1. Retrieve an image and retain its source document, page, bounding box, access permissions, and caption or OCR. 2. Check the question: Does it ask about the visual contents, or merely for the image's title, location, or textual description? 3. Try reliable text evidence first: If surrounding text, OCR, or an expert caption fully answers the question, a text model may suffice. 4. Invoke a vision language model if necessary: Send the relevant image at suitable resolution together with the user's question and supporting text. Avoid passing unrelated images. 5. Verify the response: Check extracted labels, numbers, spatial relations, and any safety critical details against the original figure or a human reviewer. 6. Cite the evidence: Point to the exact source, page, and figure; abstain when visual detail is unreadable or ambiguous. Small cost example Suppose a RAG application handles 100 questions about manuals : 70 can be answered using retrieved text and captions. 30 require direct inspection of diagrams or screenshots. Routing only those 30 questions to a multimodal model avoids 70 unnecessary vision calls , assuming the router accurately detects which requests need visual interpretation. This is an illustrative workload, not a general benchmark. Common pitfalls Assuming retrieval equals understanding: A nearest neighbor image result might be relevant but still need visual reasoning. Misreading fine details: Low resolution, cropped figures, dense legends, and tiny labels can cause errors. Ignoring surrounding text: A figure may depend on definitions or warnings in its caption and neighboring paragraph. Hallucinating beyond the image: A model may guess invisible components or numeric values. Unnecessary expense: Calling a large multimodal model for simple page title lookups increases latency and cost without improving answers. Interview takeaway: Retrieve the right image first; use a multimodal model only when the answer genuinely depends on interpreting visual evidence. Always preserve citations and validate uncertain visual claims.

What is hierarchical retrieval in RAG?

Hierarchical retrieval in RAG searches information in stages across different levels of detail —for example, identifying relevant documents first, then sections, then individual passages. Instead of comparing the query against every small chunk in a large corpus, the retriever narrows its search progressively. Beginner friendly example Imagine an engineering knowledge base with 1,000 documents , each containing 10 sections , and each section containing 20 paragraph chunks . A question asks: “How is the refresh token expiry configured in Service A?” A hierarchical retrieval pipeline might: 1. Document level: Search titles, summaries, or document embeddings and select 5 of 1,000 documents relevant to Service A. 2. Section level: Search within those documents and retain 3 sections about authentication. 3. Passage level: Search the paragraphs in those sections and return 4 passages explaining refresh token expiry. 4. Evidence generation: Rerank and pass the strongest passages, with document/section references, to the LLM. This illustrative funnel searches progressively smaller sets; selecting the wrong document at stage 1 can still exclude the correct answer. What makes it hierarchical? Multiple retrieval levels: Corpus → document → section → passage, or topic → page → chunk. Coarse to fine selection: Broad candidates are selected before detailed evidence. Explicit relationships: Lower level passages retain references to their parent sections and documents. Level specific rankings: Document relevance and passage relevance may use different indexes, models, or scores. A common implementation stores summary embeddings for documents, section embeddings for headings plus descriptions, and detailed chunk embeddings for final evidence. Other implementations combine keyword filters, structured metadata, and vectors at different stages. Hierarchical retrieval vs hierarchical chunking Hierarchical chunking is an ingestion and representation strategy : split documents into linked levels. Hierarchical retrieval is a query time search strategy : decide which levels and candidates to search, then narrow down to useful evidence. They work well together, but one does not require the other. A system can use document metadata and separate indexes to retrieve hierarchically without building a full chunk tree. Parent document retrieval is one specific related pattern: retrieve small child chunks and expand back to a larger parent for generation. That is not the same as every coarse to fine hierarchical search design. Benefits and trade offs Benefits Reduces candidate search and reranking work in large collections. Helps filter irrelevant documents or sections before expensive passage matching. Preserves document context and supports source aware citations. Limitations Early stage recall risk: An incorrect document or section filter can eliminate the right passage. Index maintenance: Multiple levels need updates when content changes. Extra stages: Additional lookups can increase latency if the indexes or routing are poorly designed. How should you evaluate it? Compare against flat chunk search on the same test questions . Measure final answer accuracy, passage recall , citation correctness, p95 latency, and cost. Also inspect how often a needed passage is lost during the document or section selection stage. Interview takeaway: Hierarchical retrieval is coarse to fine evidence search across document levels . It can improve efficiency and context organization, but its first stage filtering must preserve enough recall. Related concepts: Hierarchical chunking, parent document retrieval, and multi stage retrieval each solve different pieces of this pipeline.

What is multi-stage retrieval?

Multi stage retrieval in RAG is a search pipeline that uses two or more successive retrieval or ranking stages to narrow a broad candidate set into a small set of high quality evidence. Each stage can use a different algorithm and cost budget. Beginner friendly example Suppose a support knowledge base contains 100,000 document chunks . A user asks: “How can I rotate an expired API key without downtime?” A production pipeline might: 1. Fast candidate search: Use BM25 and/or vector search to retrieve 100 potentially useful chunks. 2. Reranking: Use a more accurate but slower cross encoder to reorder those 100 and retain the top 10 . 3. Evidence validation: Remove duplicates, obsolete instructions, and results lacking relevant permissions; pass the best 5 supported passages to the language model. 4. Generation: Answer using those passages and cite their originating documents. The numbers are illustrative, not universal settings. Why use multiple stages? A fast initial retriever can cheaply inspect a large collection, but semantic similarity or keyword overlap alone may rank a weak passage above the right answer. A slower reranker can examine query–passage interactions more carefully once the candidate set is small. Common stages Candidate generation: Keyword search, dense vectors, hybrid search, or structured filtering. Candidate merging: Deduplicate overlapping results and combine source rankings if there are multiple retrievers. Reranking: Cross encoders or other relevance models evaluate candidate evidence more thoroughly. Context selection: Apply source freshness, permissions, diversity, token limits, and sometimes claim support checks. Generation: Pass the final grounded context to the LLM. The generator is downstream of retrieval, not a retrieval stage itself. Multi stage vs related concepts Single stage retrieval: One search retrieves the final candidates directly. Hierarchical retrieval: Searches document or topic levels (document → section → passage); that is one possible multi stage design. Parent document retrieval: Retrieves small child chunks, then fetches their larger parent sections to give the generator more context. Hybrid retrieval: Combines multiple search techniques, possibly in the same stage . Hybrid search does not automatically mean multi stage retrieval. Practical trade offs Benefits Improves precision at the top of the ranked list when reranking works well. Reduces the number of items exposed to expensive models. Makes it easier to enforce evidence and context budgets. Risks Lost recall: A relevant passage excluded by the first stage cannot be recovered by reranking. Latency: Extra ranking and validation calls take time. Cost: Complex rerankers and additional APIs consume resources. How to evaluate it Measure candidate recall@K before reranking, nDCG or MRR after reranking, grounded answer quality, total p95 latency, and per query cost. Compare with a simpler baseline and inspect failures at every stage. Interview takeaway: Multi stage retrieval uses fast broad search → slower accurate ranking → evidence selection to balance coverage, precision, latency, and cost.

What is adaptive retrieval in RAG?

Adaptive retrieval in RAG changes the retrieval strategy at runtime according to a question's complexity, available evidence, and system constraints. Rather than always searching the same way, the system can choose whether to retrieve, which sources to search, and whether another retrieval round is needed . Beginner friendly example Consider three questions to a company support assistant: Simple: “Where is the password reset guide?” → Search the help center index once and return the cited guide. Complex: “Which services are affected when the payments database fails?” → Retrieve dependency information, identify connected services, then retrieve their operational runbooks. Incomplete evidence: “Does this refund exception apply to my order?” → Retrieve the policy; if eligibility still depends on missing order data, query the authorized order system or ask for clarification instead of guessing. Unlike a fixed pipeline, the assistant makes a bounded decision after inspecting the question or intermediate evidence. Typical workflow 1. Assess the query: Estimate complexity, freshness needs, domain, and whether trusted external evidence is mandatory. 2. Choose a retrieval path: Select a fast single retrieval, targeted sources, or multi step retrieval. 3. Inspect results: Check whether the evidence is relevant, current, authorized, and sufficient to answer. 4. Adapt if necessary: Rewrite an ambiguous search, change retrievers, or gather missing evidence in another round. 5. Stop safely: Answer when evidence is sufficient; otherwise abstain or request missing information. Limit retrieval rounds, latency, and cost. 6. Generate with citations: Keep the final response grounded in the retrieved material. Numerical example Suppose 100 requests are routed as follows: 60 straightforward questions: one retrieval each → 60 calls . 30 moderately difficult questions: two retrieval rounds each → 60 calls . 10 complex questions: three retrieval rounds each → 30 calls . Total: 150 retrieval calls . A fixed three round system would make 300 calls for the same 100 requests. This is an illustrative routing example , not a measured benchmark; misclassification may hurt answer quality. How is adaptive retrieval different from Dynamic Top K? Dynamic Top K adjusts how many passages are returned or retained. Adaptive retrieval decides what retrieval actions are necessary : one search, multiple steps, alternate sources, another query, or stopping. Dynamic Top K can be part of an adaptive pipeline, but changing K alone is not its full definition. Research perspective The Adaptive RAG research line uses question complexity to choose among simpler and multi step retrieval strategies. Other dynamic RAG approaches decide when to retrieve additional evidence during generation or reasoning. These are examples, not one universal implementation. Benefits and limitations Benefits: Reduced unnecessary searches on easy queries, more evidence gathering on hard queries, and flexible source selection. Limitations: Routers can underestimate complexity; confidence scores can be poorly calibrated; extra retrieval rounds increase tail latency. In regulated or source grounded applications, do not skip mandatory retrieval solely because the model appears confident. Interview takeaway: Adaptive retrieval turns search into a decision process : assess the question → choose a retrieval strategy → inspect evidence → search again or stop. Evaluate it using answer quality, evidence sufficiency, retrieval calls, and p95 latency. References: https://arxiv.org/abs/2403.14403 ; https://arxiv.org/abs/2506.06704

How would you design a RAG system for millions of documents?

Designing RAG for millions of documents is primarily a problem of scalable ingestion, efficient retrieval, data freshness, security, and observable answer quality—not simply choosing a bigger LLM. Beginner friendly architecture Step by step design 1. Clarify workload and goals: Estimate document sizes, updates per day, query rate, tenants, languages, relevance targets, latency SLOs, and permitted infrastructure costs. 2. Build a durable ingestion pipeline: Use queues and batch workers to extract text, normalize formats, split into chunks, and compute embeddings. Track source IDs, document versions, and processing status; make retries idempotent. 3. Choose chunking for the corpus: Preserve headings and source references. Use smaller passages for precise search while maintaining parent document links when surrounding context matters. 4. Plan vector scale, not just document count: The key capacity metric is the number of chunks × embedding dimensions × storage and index overhead. Support updates, deletions, and re embedding after model changes. 5. Partition the index: Shard by balanced IDs or other suitable keys; apply tenant, region, and access control filtering. Avoid a partition design that creates severe query hot spots or silently excludes relevant evidence. 6. Use two stage retrieval: Combine lexical BM25 with vector candidates, merge/deduplicate, and rerank a bounded candidate pool. Tune candidate counts and filters against a labeled evaluation set. 7. Enforce security before generation: Apply document level entitlements and tenant isolation during retrieval—not merely after content reaches the model. 8. Control latency and cost: Cache stable results when access rights and freshness permit; batch embedding jobs; use parallel shard searches and bounded timeouts; keep prompt evidence concise. 9. Keep content fresh: Incrementally ingest changes, propagate deletes, monitor indexing lag, and carry document version or last updated timestamps into citations. 10. Operate with observability: Measure ingestion failure rate, index freshness, retrieval recall, citation support, answer groundedness, latency percentiles, cost, and per tenant failures. Numerical sizing example Suppose the corpus contains 2 million documents , averaging 10 searchable chunks each : 20 million vectors total. At 768 dimensions with 32 bit floats, raw vector storage is 768 × 4 = 3,072 bytes per vector. Raw vector data is approximately 61.44 GB (decimal units). This excludes index structures, metadata, keyword indexes, replicas, temporary build space, and backups; actual provisioned storage can be substantially larger. Compressed embeddings reduce vector footprint but require recall/latency benchmarking. Critical trade offs HNSW can provide low latency approximate search but may require substantial memory at large scale. IVF/PQ or disk backed indexes can reduce memory needs, with potential recall and tuning trade offs. More shards add parallelism but also fan out, coordination, and operational overhead. Large Top K can improve candidate coverage yet increases reranking and context costs. Choose an index using representative benchmark queries, realistic metadata filters, target recall, p95 latency, and hardware constraints; no single ANN method is optimal for every workload. Production failure modes Watch for stale or deleted documents still being cited, slow ingestion backlogs, cross tenant data exposure, shard imbalance, duplicated chunks, and false confidence from an answer that sounds fluent but lacks supporting passages. Interview takeaway: For million document RAG, design around vector count, ingestion throughput, index layout, filtered retrieval, reranking, security, freshness, and quality/latency SLOs . Validate each layer independently before scaling capacity.

How does document count translate into total vector count?

Document count does not equal vector count in RAG. Most systems split a document into multiple searchable chunks and create one embedding vector per chunk. Your index size therefore depends on how many chunks each document produces , not just how many documents exist. Core formula For a simple one vector per chunk design: Total vectors = number of documents × average chunks per document For documents of different lengths, use the more accurate formula: Total vectors = sum of chunks generated by each document If you store additional vectors per chunk (for example, separate title and body embeddings), multiply by the applicable embeddings per chunk factor. Multimodal pages, document summaries, or parent level vectors add further entries. Beginner friendly numerical example A company has 10 million documents , averaging 8 chunks per document : Documents: 10,000,000 Average chunks per document: 8 Vectors per chunk: 1 Total indexed vectors: 80,000,000 (80 million) If the average rises to 12 chunks per document , the same corpus produces 120 million vectors . That is a 50% increase in vector count even though the number of source documents is unchanged. Why does chunk count vary? 1. Document length: A two page note needs fewer chunks than a 200 page technical manual. 2. Chunk size: Smaller chunks generally create more vectors. 3. Overlap: Repeated overlap between adjacent chunks increases chunk count and storage. 4. Parsing rules: Tables, headings, sections, extracted slides, and OCR may change how many units are indexed. 5. Indexing strategy: Document summaries, child chunks, image embeddings, and alternative representations can introduce extra vectors. For a single document containing T tokens, a fixed window of C tokens and overlap O tokens yields an approximate count: chunks = max(1, ceil((T O) / (C O))) This assumes T 0, 0 ≤ O < C, and straightforward sliding window chunking. Real parsers often create fewer or more chunks because boundaries are semantic rather than fixed. Worked overlap example A 2,000 token document with 500 token chunks and 100 token overlap has a stride of 400 tokens . ceil((2000 100) / 400) = ceil(4.75) = 5 chunks With zero overlap, it needs 4 chunks . Across 1 million similar documents , this raises the index from 4 million to 5 million vectors . Estimate raw vector bytes For dense embeddings stored as float32: Raw vector bytes = vector count × dimensions × 4 For 80 million vectors × 768 dimensions × 4 bytes , raw embedding payload alone is 245.76 GB in decimal units. This is not total production index storage : ANN graph or quantization structures, IDs, metadata, keyword indexes, replication, snapshots, and capacity headroom must be budgeted separately. Practical checklist Sample representative documents before estimating the average chunks per document. Use actual parser output rather than assuming every document is the same length. Track vectors per tenant, document category, and embedding model. Account for re indexing, deleted document cleanup, and temporary coexistence of old and new embeddings. Distinguish stored vectors from source documents and from runtime retrieved passages. Interview takeaway: For large scale RAG, first estimate total chunks , then vectors per chunk , then dimensions and storage format . Ten million documents can easily mean many tens or hundreds of millions of searchable vectors.

How do you estimate storage requirements for a large RAG index?

Estimating storage for a large RAG index means calculating the raw embedding payload and adding the space required for ANN indexes, searchable text, metadata, replicas, backups, and maintenance. Raw vectors alone are not a production storage estimate. Step 1: Estimate the vector payload For dense embeddings stored without compression: Raw vector bytes = vector count × dimensions × bytes per dimension Common numeric formats include float32 ( 4 bytes/value ), float16 ( 2 bytes/value ), and int8 ( 1 byte/value , usually with quantization metadata). Actual storage depends on the engine's representation. Beginner friendly numerical example Assume: 10 million documents 8 chunks per document , one vector each → 80 million vectors 768 dimensions per embedding float32 storage: 4 bytes per dimension Then: 80,000,000 × 768 × 4 = 245,760,000,000 bytes That is 245.76 GB (decimal) or approximately 228.88 GiB of raw vector values only . If the store uses float16 instead, the raw payload is 122.88 GB . This comparison excludes any extra quantization/index structures and must be validated for retrieval quality. Step 2: Add index and document overhead Budget separately for: 1. ANN index structures: HNSW neighbor links and graph structures; IVF centroid/lists; PQ codebooks and codes, depending on the chosen algorithm. 2. IDs and metadata: Document/chunk identifiers, tenant and ACL fields, timestamps, source URLs, filters, and payload indexing. 3. Original evidence: Chunk text, titles, extracted figure descriptions, and document references if stored in the retrieval engine. 4. Additional indexes: BM25 or other lexical indexes for hybrid retrieval. 5. Replicas and recovery: Copies for high availability, snapshots, and backups. 6. Operational headroom: Segment compaction, re embedding migrations, index builds, deletions, and temporary old/new versions. Step 3: Build a capacity scenario For illustration only, suppose the 245.76 GB raw vector payload has the following additional primary index costs: ANN structures, IDs, metadata, and text/index overhead: 150 GB . Primary indexed corpus: 395.76 GB . Two full serving replicas: 791.52 GB total serving storage. Add 25% operational headroom : 989.4 GB , before separately stored backups and object storage source files. These overhead values are assumptions , not benchmark derived constants. The overhead could be very different for an HNSW index, a quantized IVF PQ index, or a managed vector database. Step 4: Validate against the actual engine Index a representative sample (for example, 1 million vectors ), measure disk consumption after ingestion and compaction, and extrapolate cautiously. Test metadata filtering, text payloads, replication, and index builds. Verify whether compressed vectors replace or supplement float32 originals. Storage vs memory Disk storage covers persistent index files, payloads, replicas, and backups. RAM covers resident index structures, caching, search buffers, and sometimes full in memory vectors. An index can consume far more disk than RAM—or require a memory resident working set—depending on the search engine. Estimate them independently. Common mistakes Multiplying document count by embedding dimensions without first estimating chunks. Counting raw vectors but not ANN indexes, metadata, or source text. Forgetting replication, backups, temporary rebuilds, and growth. Applying a universal overhead multiplier without measuring the chosen engine. Mixing GB (10⁹ bytes) with GiB (2³⁰ bytes). Interview takeaway: Total storage ≈ vector payload + index/metadata/text overhead, multiplied by serving replica count, plus headroom and separately budgeted backups. Use a representative pilot to replace assumptions with actual measured usage.

How do you estimate memory requirements for vector search?

Memory sizing for vector search estimates how much RAM a retrieval service needs to keep its searchable working set, index structures, and query time buffers available at the required latency. It differs from total disk storage, which also includes persistent files, replicas, and backups. Step 1: Estimate resident vector bytes For an index that stores uncompressed vectors in RAM: Vector RAM = vector count × embedding dimensions × bytes per value Example: 20 million vectors , each 768 dimensional , stored as float32: One vector: 768 × 4 = 3,072 bytes All vectors: 20,000,000 × 3,072 = 61.44 GB (approximately 57.22 GiB ) This is only the raw vector payload—not the memory needed by the complete retrieval service. Some engines keep vectors on disk or memory map them, so do not automatically assume all vector bytes must remain resident in RAM . Step 2: Account for index specific memory HNSW: Neighbor lists, graph levels, node metadata, allocator overhead, and potentially full precision vectors. Exact overhead depends on parameters and implementation. IVF: Centroids, posting lists, IDs, and whichever vector representations are resident. IVF PQ: Compact PQ codes and codebooks can greatly reduce resident vector bytes, but accuracy and search behavior must be measured. Disk backed indexes: May use smaller RAM caches but depend more heavily on storage I/O and page cache behavior. Step 3: Add operational memory Include: 1. Metadata/filter indexes and cached payloads required at search time. 2. Per query working buffers , candidate heaps, temporary distances, and reranking allocations. 3. Concurrent queries : A buffer of 10 MB per active query becomes roughly 1 GB for 100 simultaneous active queries . 4. Index builds, updates, and compaction : Temporary memory may substantially exceed steady state usage. 5. Runtime and safety headroom : Memory for the service process, OS, fragmentation, and traffic spikes. Beginner friendly capacity example Assume an in memory service with: Raw vectors: 61.44 GB Graph/index structures and filter data: 18 GB (illustrative assumption) Concurrent search buffers and application overhead: 4 GB (illustrative assumption) The estimated steady state working set is 83.44 GB . Adding 25% sizing headroom gives about 104.3 GB RAM for that serving instance. This does not mean 25% is a universal production target or that a specific engine will fit this estimate. For a sharded cluster , distribute the index working set across nodes, but budget each node's replicas, runtime overhead, uneven shard sizes, and query concurrency separately. A replica generally consumes its own serving resources. How do you validate the estimate? 1. Index a realistic sample using the actual engine, embedding format, index parameters, and metadata filters . 2. Measure resident set size (RSS), page cache usage, and peak memory during indexing and search. 3. Load test at target concurrency and p95/p99 latency, including difficult filtered queries. 4. Scale the measured working set to the planned shard sizes and include growth and failover capacity. 5. Watch for swapping, out of memory restarts, latency spikes, and recall regressions after compression. Common mistake: Equating raw disk bytes with required RAM. An HNSW implementation that keeps most structures in memory may need much more RAM than a compressed, disk backed configuration—but the exact difference is engine specific. Interview takeaway: RAM ≈ resident vectors + index structures + metadata/filter caches + concurrent query buffers + runtime headroom. Estimate per shard and replica, then verify with real memory measurements and latency tests.

When should IVF-PQ be preferred over HNSW at large scale?

Prefer IVF PQ over an uncompressed HNSW index when a very large vector collection must fit a strict memory budget and you can tolerate some approximate search recall loss. Choose based on measured relevance, latency, RAM, and operating cost—not document count alone. Beginner friendly example A company indexes 50 million passages using 768 dimensional float32 embeddings . Raw vectors alone require 50,000,000 × 768 × 4 = 153.6 GB before graph links and other HNSW overhead. In an illustrative IVF PQ configuration with 96 eight bit subquantizer codes per vector , PQ codes require 50,000,000 × 96 = 4.8 GB . That 4.8 GB excludes IDs, IVF centroids and lists, codebooks, metadata, optional original vectors, and other system costs. It is not a complete index sizing estimate. IVF PQ can therefore make an otherwise expensive vector collection much smaller, but compression introduces approximation error. When IVF PQ is a strong candidate 1. RAM is the binding constraint. Full precision vectors plus HNSW's graph connections exceed affordable serving memory. 2. The collection is very large. Tens or hundreds of millions of vectors magnify per vector savings. 3. Moderate recall loss is acceptable. A later reranker or exact rescoring of a shortlist may recover useful answer quality, though missed candidates cannot always be recovered. 4. Batch training is feasible. You can train representative IVF centroids and PQ codebooks and periodically revisit them as data changes. 5. Your retrieval engine supports the required filtering, updates, and deployment pattern. Index features vary significantly by implementation. When HNSW is usually preferable High recall and low interactive latency are more important than reducing RAM. The full working set fits comfortably within available memory. You want to avoid IVF/PQ training and complicated compression tuning. Your workload and chosen engine benefit from graph navigation; verify update/delete behavior because it differs between implementations. How to decide in production 1. Fix a test workload: Use realistic embeddings, query distribution, tenant filters, and a labeled relevance set. 2. Build both indexes: Benchmark uncompressed HNSW and appropriately tuned IVF PQ with the same vectors. 3. Tune fairly: For HNSW, vary M and efSearch; for IVF PQ, vary cluster count, nprobe, PQ code size, and optional reranking. 4. Measure end to end: Compare recall@K and answer grounding quality, p95 retrieval latency, memory, build time, update costs, and total infrastructure spend. 5. Use a hybrid design if needed: Compressed candidate retrieval followed by full precision rescoring can be worthwhile if the original vectors are retained somewhere accessible. Important limitations IVF PQ does not automatically beat HNSW in speed or accuracy ; dataset geometry, parameters, filters, and hardware determine results. PQ compression can distort neighbors, especially close candidates whose differences matter. IVF PQ adds training and operational complexity. Some systems combine HNSW with compression; the choice is not always a strict either/or. Interview takeaway: Use IVF PQ when memory efficiency at massive scale matters enough to justify approximation and training. Use HNSW when the index fits memory and strong recall/latency is the priority. Benchmark both. Reference: Faiss, Guidelines to choose an index : https://github.com/facebookresearch/faiss/wiki/Guidelines to choose an index

Why can HNSW become expensive for tens of millions of vectors?

HNSW can become expensive at tens of millions of vectors because it commonly keeps high dimensional embeddings and a multilayer neighbor graph readily accessible for fast search. Large index builds, replication, updates, and memory requirements can make hardware and operations costly. Beginner friendly numerical example Suppose a RAG system indexes 50 million text chunks using 768 dimensional float32 vectors . Each vector stores 768 × 4 = 3,072 bytes. Raw vectors alone occupy 50,000,000 × 3,072 = 153.6 GB (decimal). Suppose each node has an illustrative average of 20 stored neighbor IDs across graph levels, at 4 bytes per ID : 50,000,000 × 20 × 4 = 4 GB for neighbor IDs alone. Together that is 157.6 GB before graph level bookkeeping, offsets, metadata, runtime allocations, filtering structures, and allocator overhead. This is a simplified calculation, not a universal HNSW memory formula . Actual neighbor counts, pointer widths, compression, and index layouts vary by engine. Where does the expense come from? 1. Resident embedding memory: If an implementation requires the vectors in RAM, capacity increases roughly with vector count × dimensions × bytes per component. Disk backed variants have different trade offs. 2. Graph connections: HNSW stores neighbor lists and information about graph levels. Larger construction parameter M typically increases index footprint and construction work. 3. Index construction: Building graph connections involves many distance comparisons and temporary working memory. Rebuilding a large index can be time consuming. 4. Replication: Serving two full copies can nearly double the index related serving capacity, although some overhead is shared or engine specific. 5. Updates and deletions: Frequent changes may trigger maintenance, tombstones, or rebuilding; behavior depends on the implementation. 6. Query time quality requirements: Higher search effort (for example, efSearch) often increases computation and latency to achieve better recall. 7. Operational headroom: Servers need memory beyond the nominal index size to absorb concurrent queries, failover, growth, and peak indexing tasks. Illustrative deployment sizing If the measured resident index working set is 190 GB per copy and an application needs two serving replicas , the cluster must support approximately 380 GB of resident index data , plus per node application memory and safety headroom . This is not the same as specifying a single 380 GB machine: sharding and placement determine actual node sizing. How can costs be controlled? Benchmark HNSW parameter settings , especially M, construction effort, and search effort, against recall@K and p95 latency. Consider float16 or vector compression where the search engine supports it; measure recall loss before rollout. Compare IVF PQ and disk backed ANN indexes when RAM dominates cost. Use sharding to fit indexes across nodes, while accounting for routing, shard fan out, and replica overhead. Keep only necessary versions of indexed content; handle deletions and re embedding carefully. Use real production like filtered queries for capacity and cost testing. Important nuance HNSW is not inherently too expensive at 10 million or 50 million vectors. Feasibility depends on vector dimensions, storage format, memory architecture, query rate, accuracy targets, and hardware pricing. Many implementations also combine HNSW with compression. Interview takeaway: At large scale, HNSW's cost comes from vector payload + graph structure + index building + replication + operational headroom . Choose it when its recall and latency justify the measured total cost, rather than assuming it is always best or always too expensive.