Your vector index grows rapidly even though the number of active documents remains constant. How would you investigate storage amplification and deleted-vector accumulation?

Production incident (illustrative): A RAG knowledge-base collection maintains 2 million active document chunks, yet its vector database expands from 18 GiB to 45 GiB during a month of nightly document replacements. Ingestion reports success, search generally works, and the active chunk count is nearly unchanged. The operations team considers adding storage, but a hot node approaches its disk watermark and p99 spikes during background compaction. Root cause hypothesis: old vector versions have been logically deleted but not physically reclaimed, and optimizer work falls behind repeated upserts. Prove that hypothesis before running destructive cleanup.

1. Understand what is growing

• Logical live data: active, authorized point IDs and their document versions. A stable active count does not imply stable physical bytes.
• Deleted/tombstoned data: deleted points and overwritten vector versions can remain in segment files until safe background cleanup/merge. A query correctly excludes them even while storage still holds bytes.
• Write amplification: new segment files, WAL, optimizer copy-on-write, merge inputs/outputs and replica updates temporarily or persistently consume extra bytes.
• Independent storage contributors: raw vectors, HNSW graph, payload/filter indexes, text payloads, quantized copies, sparse vectors, WAL, snapshots, old collections, shard replicas and backups. Never blame tombstones from total disk growth alone.
• Replication and cloud billing: total cluster bytes may include copies of each shard, snapshots, backup/object-store versions or temporary transfer replicas; compare equivalent scope (per-primary shard vs all replica nodes).
• Storage amplification ratio: define actual_physical_index_bytes / estimated_ideal_live_index_bytes for the *same node/shard set and feature configuration*. Include the scope in dashboards; a ratio over 1 does not automatically mean waste.

2. Contain the incident before touching storage

• Confirm immediate disk risk using free bytes/inodes, growth rate, read/write I/O latency, disk watermarks, optimizer backlog, allocation failures and read/search error rates.
• Preserve latest source and index manifests, snapshot metadata, recent deployment/config changes, active-version ledger, retention/legal-hold policies and query-latency baselines.
• If safe, cap redundant upserts, rate-limit ingestion or optimization concurrency, isolate a busy tenant/collection and reserve temporary disk for compaction. Do not run forced merges on a nearly full disk without estimating peak working space.
• Never permanently delete current records, backups required for recovery, or protected audit evidence just to relieve pressure. Verify tombstone serving blocks and current ACL filters independently.
• If there is insufficient space even for controlled cleanup, expand disk temporarily or migrate to a properly sized healthy environment; this is capacity protection, not a root-cause remedy.

3. Trace the real byte growth

• Compare day-by-day live point count, total physical records if exposed, deleted-point ratio, segments, indexed vectors, storage bytes by segment, replica factor and snapshot/WAL bytes. Inspect per-shard differences instead of relying on collection-wide averages.
• Break down disk by directory or provider-reported categories: vectors, hnsw/index, payload, WAL, optimizer temporary, snapshots, replicas, old aliases/collections. Avoid scanning large files destructively during an incident.
• Reconstruct input churn: inserts, updates, deletes and repeated upserts of the same external point ID. Qdrant documents that an upsert of an unchanged point can still mark the prior representation deleted and insert a new one; avoid sending unnecessary replacement writes.
• Compare optimizer/vacuum candidate threshold, segment count, active merge/vacuum threads, last successful cleanup, queue age, CPU/I/O saturation, available scratch space and repeated optimizer failures.
• Inspect record ID policy and application lineage. Are nightly jobs creating *new IDs* for unchanged chunks or keeping obsolete document versions in a supposedly active collection? If both, there is also logical data duplication not solved by tombstone vacuum.
• Compare vector dimensions, dtype, quantization mode, payload-index additions, sparse/hybrid indexes, changed chunk sizes and replication changes; any of these can enlarge the *valid* footprint without adding documents.
• Check source deletes against serving visibility as a separate security requirement. Physical disk cleanup is usually asynchronous and must not be confused with timely authorization/deletion enforcement.

4. Root-cause debugging decision tree

• Deleted ratio rising, vacuum backlog increasing: suspect reclamation lag from churn or constrained optimizer resources. Examine per-segment eligibility and failed optimization.
• Deleted ratio low, live bytes high: suspect changed vector dimension, duplicate new IDs, payload growth, new named/sparse vectors, extra indexes, replication or old serving generations.
• Disk spikes during optimizer and then shrinks: temporary segment rewrite/copy-on-write is plausible; quantify peak headroom and tune concurrent compactions.
• WAL/snapshot bytes grow while segment bytes are stable: inspect snapshot lifecycle, retention, WAL replay/checkpoint status and abandoned backups; do not delete files outside supported management APIs.
• Only one shard/node grows: inspect tenant skew, shard placement, replica transfers, write distribution and a stuck local optimizer.
• Queries return deleted records: an independent visibility/deletion bug exists; remediate serving filters immediately, not only physical reclamation.
• Search slows after aggressive vacuum: optimizer I/O contention can raise p99. Optimize for both stable disk consumption and query SLOs.

5. Beginner-friendly numerical example

• Collection has 2,000,000 active points. Each stores a 768-dimensional float32 vector: 768 × 4 = 3,072 bytes raw, so live raw vectors use 2,000,000 × 3,072 = 6,144,000,000 bytes ≈ 5.72 GiB. This excludes graph edges, payload, metadata, replicas and snapshots.
• Assume 1,000,000 upserts every night overwrite existing IDs and the older representations are marked deleted until vacuum. Over 10 days, that is 10,000,000 historical write events; do not assume all produce ten million still-retained tombstones, because maintenance may reclaim some along the way.
• At incident time, suppose per-shard inventory aggregated across the collection shows 2 million live and 3 million still physically represented but logically deleted point versions. The tombstone share is 3/(2+3) = 60% of represented versions. The 3 million stale raw vectors alone would occupy 3,000,000 × 3,072 / 2^30 ≈ 8.58 GiB before graph/payload/segment overhead, assuming each retains a raw representation.
• Baseline total index storage was 18 GiB; measured storage is 45 GiB, so physical growth = 27 GiB, 150% increase, and 45/18 = 2.5× baseline bytes. This 2.5× is a baseline growth multiple, not a rigorously normalized amplification ratio unless the original 18 GiB is independently proven to be the ideal current live footprint.
• Suppose measured breakdown explains the 27 GiB increase as 8.6 GiB retained stale vector bytes + 6.4 GiB associated obsolete indexes/payload + 7.0 GiB temporary merge/copy-on-write files + 5.0 GiB additional snapshot/WAL files = 27.0 GiB. These are illustrative attributed components on an equivalent disk scope, and should not be double-counted in real filesystem statistics.
• After disabling identical no-op upserts, letting supported vacuum/merge processes catch up, and cleaning verified expired snapshots under retention policy, suppose storage stabilizes at 21 GiB, a 24 GiB (53.3%) drop from 45 GiB. Search p99 must be monitored while cleanup runs; bytes reclaimed is not success if search or ingestion SLAs fail.

6. Engineering solution

• Stop avoidable churn: use stable tenant/document/version/chunk identifiers, hashes or source revision checks; skip unchanged data instead of blindly upserting. Reconcile live manifests to prevent accumulating active but obsolete versions.
• Schedule safe reclamation: tune the vendor's deleted-ratio threshold and vacuum/compaction workers against disk headroom, shard size and read/write workload. Monitor that each cleanup completes and physical bytes decline.
• Budget scratch capacity: segment rewrites can keep old data readable while new files are built. Estimate temporary disk amplification from representative compactions; do not assume vacuum immediately frees space.
• Protect search during maintenance: bound optimizer concurrency, pace work outside peak hours where appropriate, prioritize safety-critical deletes, and benchmark p50/p95/p99, CPU and disk I/O before/after. Qdrant guidance describes the storage-versus-latency trade-off of different vacuum thresholds.
• Maintain snapshots and WAL responsibly: use vendor-supported snapshot/retention and checkpoint APIs, verify recoverability before expiry, and treat offline backups as distinct from serving-index bytes.
• Validate after cleanup: reconcile live point IDs, active document versions and deletion/tombstone authorization; test Recall@k, citation correctness, replica coverage, per-shard disk footprint and ingest freshness. Consider a shadow rebuild only if maintenance cannot restore a healthy index, with full write replay and security gates.
• Escalate capacity only when justified: if the expected *legitimate live data and indexes*, replica requirements and peak maintenance overhead exceed storage budget after cleanup, resize or rebalance with evidence.

7. Vendor-specific behavior and operational trade-offs

• Qdrant: records can be soft-deleted until the Vacuum Optimizer rebuilds segments; optimization uses segment-level work and can keep old segments readable while new ones are written. Tune deleted_threshold, vacuum_min_vector_number and optimizer limits for the installed version. Qdrant docs note that even unchanged upserts can cause delete-and-reinsert behavior. Avoid generic advice to edit index files manually.
• Weaviate: HNSW tombstone cleanup runs periodically; cleanupIntervalSeconds and supported TOMBSTONE_DELETION_* parameters affect cleanup batch size/concurrency. For very large indexes, cleanup itself may cause resource pressure. Check deployment-version support and actual tombstone backlog.
• Elasticsearch/Lucene: updates/deletes create soft-deleted segments that are reclaimed during merges. Elastic advises force merging only read-only indexes; forcing a live index into very large segments can *worsen* retained deletes and future disk usage.
• Lower vacuum threshold: typically reclaims deleted bytes sooner but increases background work and search contention. A higher threshold can preserve p99 during peak writes at the cost of extra disk usage.
• Index compression or quantization: may lower legitimate footprint but can hurt Recall@k or incur rescoring costs. It does not fix waste from duplicate IDs, snapshots or blocked vacuum.
• Scale out: distributes some load but may duplicate shards/replicas and does not automatically reclaim obsolete versions; validate placement, storage and consistency.
• Tombstone retention versus compliance: some deletion records are essential governance evidence, but embedding/text copies must meet the system's deletion policy. Retain minimal audit metadata separately from stale searchable content.

8. Prevent recurrence

• Dashboard physical bytes by component and shard, live vs deleted versions, deleted fraction, optimizer last-success time/backlog, disk growth velocity, snapshot/WAL retention, unique chunk-ID churn and replica overhead.
• Alert on accelerating bytes per live vector, sustained vacuum backlog, high deleted ratio, rapidly increasing disk-watermark risk and compaction failures—not merely count of active documents.
• Add ingestion idempotency, content-hash checks and no-op upsert suppression; run periodic source-to-index manifest reconciliation and stale-version cleanup.
• Load-test millions of realistic upserts and deletes with simultaneous reads, measuring search p99, disk peak/headroom, optimizer completion and final steady-state bytes. Include optimizer crashes and restore drills.
• Establish disk capacity budgets for active corpus + expected tombstones + WAL/snapshots + compaction scratch + replication + safety margin, with policies for old index generations and backup retention.
• Review deletion audit trails and serving eligibility independently so cleanup delays never result in retrieval of revoked documents.

Interview takeaway: A stable document count with exploding index bytes often reflects storage amplification from soft deletes, repeated upserts, segment rewrites or retained backups. Attribute bytes first, distinguish logical deletion from physical reclamation, suppress pointless writes, and tune safe cleanup under query-SLO and free-disk constraints.

References

• Qdrant, Optimizer and Vacuum Optimizer: https://qdrant.tech/documentation/ops-optimization/optimizer/
• Qdrant, Storage and WAL details: https://qdrant.tech/documentation/manage-data/storage/
• Qdrant, Soft-deletion and unchanged upserts FAQ: https://qdrant.tech/documentation/faq/
• Qdrant, Predictable search latency and vacuum trade-offs: https://qdrant.tech/documentation/operate-and-maintain/predictable-search-latency/
• Weaviate, HNSW tombstone cleanup parameters: https://docs.weaviate.io/weaviate/config-refs/indexing/vector-index
• Elastic, Disk usage and merge guidance: https://www.elastic.co/docs/deploy-manage/production-guidance/optimize-performance/disk-usage