How do you estimate memory requirements for vector search?

Memory sizing for vector search estimates how much RAM a retrieval service needs to keep its searchable working set, index structures, and query-time buffers available at the required latency. It differs from total disk storage, which also includes persistent files, replicas, and backups.

Step 1: Estimate resident vector bytes

For an index that stores uncompressed vectors in RAM:

Vector RAM = vector count × embedding dimensions × bytes per value

Example: 20 million vectors, each 768-dimensional, stored as float32:

• One vector: 768 × 4 = 3,072 bytes
• All vectors: 20,000,000 × 3,072 = 61.44 GB (approximately 57.22 GiB)

This is only the raw vector payload—not the memory needed by the complete retrieval service. Some engines keep vectors on disk or memory-map them, so do not automatically assume all vector bytes must remain resident in RAM.

Step 2: Account for index-specific memory

• HNSW: Neighbor lists, graph levels, node metadata, allocator overhead, and potentially full-precision vectors. Exact overhead depends on parameters and implementation.
• IVF: Centroids, posting lists, IDs, and whichever vector representations are resident.
• IVF-PQ: Compact PQ codes and codebooks can greatly reduce resident vector bytes, but accuracy and search behavior must be measured.
• Disk-backed indexes: May use smaller RAM caches but depend more heavily on storage I/O and page-cache behavior.

Step 3: Add operational memory

Include:

1. Metadata/filter indexes and cached payloads required at search time.
2. Per-query working buffers, candidate heaps, temporary distances, and reranking allocations.
3. Concurrent queries: A buffer of 10 MB per active query becomes roughly 1 GB for 100 simultaneous active queries.
4. Index builds, updates, and compaction: Temporary memory may substantially exceed steady-state usage.
5. Runtime and safety headroom: Memory for the service process, OS, fragmentation, and traffic spikes.

Beginner-friendly capacity example

Assume an in-memory service with:

• Raw vectors: 61.44 GB
• Graph/index structures and filter data: 18 GB *(illustrative assumption)*
• Concurrent search buffers and application overhead: 4 GB *(illustrative assumption)*

The estimated steady-state working set is 83.44 GB. Adding 25% sizing headroom gives about 104.3 GB RAM for that serving instance. This does *not* mean 25% is a universal production target or that a specific engine will fit this estimate.

For a sharded cluster, distribute the index working set across nodes, but budget each node's replicas, runtime overhead, uneven shard sizes, and query concurrency separately. A replica generally consumes its own serving resources.

How do you validate the estimate?

1. Index a realistic sample using the actual engine, embedding format, index parameters, and metadata filters.
2. Measure resident set size (RSS), page-cache usage, and peak memory during indexing and search.
3. Load-test at target concurrency and p95/p99 latency, including difficult filtered queries.
4. Scale the measured working set to the planned shard sizes and include growth and failover capacity.
5. Watch for swapping, out-of-memory restarts, latency spikes, and recall regressions after compression.

Common mistake: Equating raw disk bytes with required RAM. An HNSW implementation that keeps most structures in memory may need much more RAM than a compressed, disk-backed configuration—but the exact difference is engine-specific.

Interview takeaway: RAM ≈ resident vectors + index structures + metadata/filter caches + concurrent-query buffers + runtime headroom. Estimate per shard and replica, then verify with real memory measurements and latency tests.