A vector database node reaches 95% memory usage during peak traffic. What would you investigate before adding more nodes?

Production incident (illustrative): A multi-tenant RAG platform runs vector search on a 32 GiB node. During peak traffic, the dashboard reports 95% memory utilization and an on-call engineer proposes doubling the node count. At the same time, some queries slow down, ingestion is active, and a replica restarts once. But “95% memory” alone cannot distinguish useful file-system cache from non-reclaimable memory, nor prove that adding nodes will solve the incident.

Goal: Identify the actual memory consumer, prove whether there is harmful memory pressure, relieve the bottleneck safely, and scale only if measured steady-state demand and required headroom justify it.

1. Contain the production incident

• Verify whether the node is in immediate danger: OOMKilled events, container memory-limit breaches, failed allocations, increasing major page faults, sustained swap in/out, memory pressure stalls, elevated p95/p99, request errors and long GC pauses.
• If OOMs or errors are active, temporarily cap ingestion/optimizer concurrency and expensive concurrent searches, and route load to healthy, sufficiently provisioned replicas where available. Avoid removing security filters or disabling durability to save memory.
• Preserve dashboards, process/container metrics, deployment/config changes, collection/shard inventory and a representative failing search trace before restarting anything.
• Separate an alert on utilization from an incident of memory exhaustion; high cache usage can be desirable and reclaimable.

2. Diagnose what “95% used” measures

• Read host MemTotal, MemAvailable, file/page cache, anonymous memory, slab and swap; compare with container/cgroup memory.current, memory.max, memory.stat (notably anon and file), memory.events OOM counters, and process RSS/PSS.
• Interpret values carefully: RSS includes memory-mapped file pages, shared mappings can be counted in multiple processes, and host “used” memory may largely be reclaimable page cache. Container limits and truly available headroom matter more than a single dashboard percentage.
• Check time series of anonymous/resident memory, cache hit/miss ratio, memory pressure stalls and major faults. Is memory stable after warm-up, growing without bound, or peaking only during index optimization?
• Attribute consumption by collection, shard, replica and component: raw dense/sparse vectors, HNSW graph, quantized copies, payload/filter indexes, ID mappings, mmap/page cache, loaded/growing segments, query buffers, connection pools, caches and ingestion/compaction scratch space.
• Inspect vectors per node, dimensions and dtype; replication/shard layout; large filters and oversized result payloads; pending optimizers/compactions; background batch concurrency; and active-query concurrency.
• Don't assume every vendor stores these structures identically: Qdrant supports per-structure memory tiers, Weaviate documents HNSW/index and vector-cache requirements, and Milvus has query and growing/loaded segment behavior.

3. Trace cause rather than guessing

• Reclaimable cache: high reported host memory yet healthy MemAvailable, little reclaim/PSI pressure, no OOM and stable p99. The fix may simply be correcting alert thresholds and tracking meaningful signals, not scaling.
• Pinned vector/index/payload memory: anonymous memory and OOM risk rise with collection size; estimate vectors, graph links and payload indexes. Review per-component memory tier and configured index dimensions before scaling.
• Temporary optimizer peak: memory increases in sync with compaction, segment merging, HNSW builds or bulk upserts. Compare active optimizers, work queues and peak memory against normal serving baseline.
• Request-concurrency spike: query buffers and result allocations scale with simultaneous queries, large top_k, nested filters, oversize payload fetches or co-located workloads. Compare allocations per request and in-flight concurrency.
• Skew or uneven placement: only one node/tenant/shard spikes while peers have space. Check shard sizes, local hot collections and replica routing before buying more capacity.
• Leak or cache misconfiguration: monotonically growing resident/heap memory despite stable dataset and throughput, never reclaimed after load subsides. Profile allocations and inspect deployments for regressions.
• I/O bottleneck masquerading as memory: moving vectors/index to disk relieves pinned RAM, but page-cache misses and slow SSD/network storage can worsen latency. Investigate disk read IOPS/await and tail latency together.

4. Practical numerical example

• The node has 32 GiB of RAM and reports 30.4 GiB used = 95%. Breakdown from an illustrative host/cgroup investigation: 13 GiB non-reclaimable anonymous/pinned process memory, 12 GiB mostly reclaimable page cache, 5.4 GiB other committed memory, and 1.6 GiB nominally unused. These categories total 32 GiB, while the first three total 30.4 GiB used; actual MemAvailable may be higher than nominally unused RAM because cache is reclaimable. Real kernel accounting can overlap: verify cgroup stats and MemAvailable rather than relying on this simplified inventory.
• The vector collection holds 2,000,000 vectors of 768 dimensions in float32. Raw storage estimate = 2,000,000 × 768 × 4 bytes = 6,144,000,000 bytes ≈ 5.72 GiB, *before* HNSW links, payload indexes, ID tracking, quantized replicas, allocator overhead and temporary builders.
• During bulk ingestion, optimizer/build workers add an observed 5 GiB temporary allocation: illustrative non-reclaimable demand rises from 13 to 18 GiB. A parallel batch of large-filter searches adds 4 GiB, producing 22 GiB non-reclaimable demand. On a node also running OS services and page cache, this triggers reclaim and latency spikes.
• Controlled test: cap ingestion builders from 4 to 1, limit concurrent expensive searches from 40 to 20, and keep the same validation workload. Peak non-reclaimable memory falls from 22 to 16 GiB and vector-search p99 from 850 to 290 ms, while the index remains complete. Those results support transient write/query concurrency as the proximate cause rather than permanent vector-footprint growth.
• If demand instead settles at 28 GiB non-reclaimable with an operational 25% headroom policy, required RAM is 28 / 0.75 ≈ 37.3 GiB, so a 32-GiB node is undersized even before replicas and build peaks. That is measured evidence to resize or rebalance. Headroom percentages are business-specific, not universal.
• Compression illustration: a raw float32 vector uses 3,072 bytes at 768 dimensions; an int8 representation uses 768 bytes (75% less for that representation), but actual node savings will be much smaller if graphs, raw vectors, payloads, caches or indexing buffers remain. Benchmark recall and rescoring before enabling quantization.

5. Apply fixes in risk-aware order

• First: observation and safety. Fix misleading utilization dashboards; monitor OOM counters, MemAvailable, cgroup usage/limit, PSI, faults, swap, disk latency and search tail latency. Reduce unnecessary work only with clear freshness/latency limits.
• Second: reversible workload controls. Rate-limit ingestion, cap concurrent optimizer tasks, smooth bursty traffic, reduce excessive query concurrency, prevent unbounded result sets and tune request caching. Verify no SLA regression and no failed writes.
• Third: correct skew and lifecycle waste. Rebalance existing shards/replicas, remove proven orphaned/superseded data through governed deletion, and review over-indexed payload fields or duplicate vectors. Never delete active or legally retained data merely to free RAM.
• Fourth: tune representation and storage. Move appropriate structures to cached/cold/on-disk tiers when supported; choose fast SSD/NVMe; quantize or use compression where validated. Index/payload changes may require rebuilds or optimizer passes, extra temporary capacity, and controlled rollout. Recheck Recall@k, filter correctness and p99, not memory alone.
• Fifth: scale with evidence. Add memory, rebalance, or add nodes when active non-reclaimable working set plus build headroom exceeds safe capacity, when load is horizontally partitionable, and when projected growth justifies it. Extra nodes won't automatically fix a leak, poor placement, excessive per-query buffers or a bad filter.

6. Trade-offs and edge cases

• Tighter concurrency: protects memory/p99 but may increase queueing and source-to-search freshness lag; define separate ingestion and search service budgets.
• On-disk vectors or HNSW: reduces pinned RAM but raises disk dependencies, page faults and potentially p99; mmap pages still appear as process memory. Memory-tier options depend on vendor/version.
• Quantization: reduces stored representation size but may lose nearest-neighbor recall, especially for borderline vectors. Original-vector rescoring can recover quality at added I/O/CPU cost.
• Lower graph connectivity / smaller cache: reduces memory but can hurt Recall@k or slow search; evaluate real query/filter workloads before rebuild.
• Scaling out: may help when sharding distributes memory and QPS, but replicas can duplicate memory footprint and certain hot shards remain problematic. Verify placement and replication semantics.
• Aggressive garbage collection or restarts: can temporarily reduce apparent usage but may cause latency spikes, cold-cache effects or rebalancing; do not use as the root-cause fix.
• Kubernetes nuances: cgroup memory counters, limits, oom_kill, Kubernetes working-set approximations and host MemAvailable have different semantics. Use several corroborating signals rather than treating any one metric as exact available RAM.

7. Prevent recurrence

• Record a component-level memory budget: vector data, HNSW graph, payload/filter indexes, replicas, per-query buffers, optimizers, cache and deliberate headroom. Reforecast with vector counts, dimensions and data growth.
• Dashboard non-reclaimable vs reclaimable memory, OOM events, swap/PSI, major faults, page-cache hit rate, disk pressure, concurrent queries, background optimizations and p50/p99 per node and tenant.
• Alert on sustained pressure and projected exhaustion, not only 95% utilization. Correlate memory warnings with user-facing latency/errors and ingest lag.
• Run representative peak traffic + bulk indexing load tests, cold/warm cache tests and growth projections. Include node restarts, failover, compaction and large filtered queries.
• Review configuration/rollout history and maintain a playbook for memory attribution, reversible mitigations, quality checks and capacity changes.

Interview takeaway: 95% memory usage is a symptom, not a scale-out decision. Determine whether memory is reclaimable cache, active vector/index footprint, temporary optimizer/query buffers, a leak, or shard imbalance. Fix the demonstrated cause and preserve search recall and freshness; scale only when the steady-state plus peak headroom still exceeds safe capacity.

References

• Qdrant: Database optimization FAQ — https://qdrant.tech/documentation/faq/database-optimization/
• Qdrant: Memory tiers — https://qdrant.tech/documentation/ops-configuration/memory-tiers/
• Qdrant: Optimize performance — https://qdrant.tech/documentation/ops-optimization/optimize/
• Weaviate: Resource planning — https://docs.weaviate.io/weaviate/concepts/resources
• Milvus: Data processing and loaded segments — https://milvus.io/docs/data_processing.md