A large index rebuild takes six hours. How would you keep search available during the rebuild?

Production incident (illustrative): A 12-million-vector RAG service needs to rebuild its HNSW index after an index-format change. A straightforward rebuild takes six hours. The first attempt rebuilds the live collection in place: memory and disk I/O spike, retrieval p99 rises from 180 ms to 2,000 ms, and some queries time out. Stopping search for six hours is unacceptable. The problem is the deployment strategy, not necessarily a faulty ANN algorithm.

1. Contain and identify the bottleneck

• Keep the last verified index serving queries through a stable knowledge_prod alias or application routing pointer. If an in-place rebuild is already damaging service, stop or throttle it when the backend permits safely; restore traffic to a healthy replica or pre-rebuild collection if available.
• Capture the rebuild stage and worker logs, read/write IOPS, disk capacity, RAM/page faults, optimizer CPU use, query queues, shard health, p50/p95/p99, errors, recall, ingest lag and current ACL/deletion state.
• Identify whether failure is caused by read/write contention, the unavailability of the index being rebuilt, optimizer pauses, poor resource isolation or premature cutover before all writes are visible.
• Avoid assuming replicas solve everything: if all replicas share the same collection update or disk bottleneck, they can degrade together. Confirm isolation and vendor semantics.

2. Use blue/green rebuilding with isolation

• Blue: rag_index_v1, currently serving all reads through knowledge_prod.
• Green: rag_index_v2, a separately named collection/index with matching tenant, metadata filter, authorization and citation schemas. Create it with enough independent CPU, RAM, I/O and storage to avoid starving blue.
• Freeze a consistent source snapshot at durable change-log offset L0. Include immutable chunk IDs, document/version manifest, source hashes, embedding model version and an ACL/deletion ledger. Build green from the authoritative snapshot without changing blue.
• Continue writes to blue and record *every change since L0* in an ordered durable change log (CDC/outbox): inserts, upserts, updates, deletes, permission revocations and document supersessions. If carefully implemented, optional dual-writing can reduce catch-up lag, but is not a substitute for an independently replayable log and idempotent reconciliation.
• Build green in the background, then replay log entries from L0 in per-document/version sequence. Use deterministic IDs and idempotent operations. Fence against late older writes, and apply deletions/ACL revocations to both indices or at a shared serving gate immediately to prevent re-exposure.
• When green is near real time, briefly fence/synchronize the write stream: capture cutover offset L1, ensure green is caught up and query-visible through L1, then switch the alias or routing pointer. Continue event delivery to the new primary; do not drop writes during the switch.
• In Qdrant, an atomic multi-action alias update can redirect a stable alias to the new collection. Weaviate's collection alias update is also atomic. Confirm the specific provider/version behavior, especially for in-flight requests, replicas and writes to an alias.
• Keep blue retained for a bounded rollback window. A rollback is safe only if deletes, revocations and writes after cutover have been propagated to blue or are enforced independently; an outdated blue index is not a safe rollback target.

3. Validate before cutover

• Completeness: reconcile snapshot manifests with green's live IDs, counts and document versions; use hashes and sample individual records—not counts alone. Confirm that active records are present and deleted records are absent/blocked.
• Consistency: verify CDC cursor and end-to-end read visibility through L1, query-time freshness, current ACL/tenant filters and citation targets. Protect against async replica lag and stale caches.
• Search quality: run golden queries on both versions, compare Recall@k against an appropriate exact/labelled reference, nDCG, answer citations, filter correctness and p95/p99. If the rebuild also changes embedding model or metric, evaluate each against its own valid reference and compare task relevance on labelled queries; don't conflate vector spaces.
• Operational health: ensure no failed shards, no growing ingestion backlog, sufficient resource headroom and p99 below agreed SLO under concurrent writes.
• Release: canary a small cohort to green using an application-level router if the backend alias only supports a whole-collection switch. After passing tests, perform one guarded alias cutover. Monitor errors, quality and source-to-search freshness; invalidate caches keyed to the old index version.

4. Numerical example: six-hour rebuild without downtime

• Blue has 12,000,000 active vectors. At 10:00, capture snapshot and start green build at log offset L0 = 50,000,000. Blue continues serving queries.
• Average change rate is 100 events per minute for 360 minutes: 36,000 inserts, updates, deletions and ACL/version operations accumulate during the six-hour build.
• Green finishes at 16:00 with the snapshot's 12,000,000 records. Assume a dedicated replay worker applies 600 events/minute while fresh changes continue at 100/minute. Net backlog reduction = 600 − 100 = 500 events/minute. First-order catch-up time is 36,000/500 = 72 minutes, ending around 17:12. This assumes steady rates, no bursts and no replay failures.
• At 17:12, freeze a short cutover checkpoint L1, wait for green replay and search visibility, then switch knowledge_prod: v1 → v2. Suppose a coordinated fence lasts 15 seconds: query traffic remains on blue during the fence, so read downtime is 0 seconds; write acceptance may queue briefly, and this must be tracked separately from read availability.
• Sample validation: 12,000,000/12,000,000 snapshot records accounted for by the manifest, every event through L1 reconciled, 1,000/1,000 security-filter canary queries pass, and representative Recall@10 remains ≥0.94 (illustrative acceptance criterion). p99 stays ≤200 ms during the rebuild because the green resource budget doesn't starve blue.
• The naïve in-place rebuild would risk 360 minutes of degraded or interrupted service. The isolated rebuild serves blue continuously for 432 minutes of build plus catch-up, then changes a pointer; this improves availability, not rebuild duration. If green cannot pass gates, continue blue service rather than forcing a cutover.

5. Engineering choices and trade-offs

• Separate collection / blue-green: simple rollback and no read interruption, but temporary extra vectors, storage, network and compute. Peak disk may exceed 2× for segments, WAL, compaction and snapshots; capacity-plan with headroom.
• Replica-by-replica rolling rebuild: useful where the backend supports independent replica rebuild and quorum-compatible reads/writes, but can reduce redundancy or cause version divergence. Prove consistency rather than assuming it.
• Dual writes: reduce catch-up lag but introduce partial success, ordering races and drift; durable CDC plus reconciliation is usually easier to audit.
• Short write fence: simplifies an atomic cutover but increases write latency/backlog. For strict continuous writes, use an ordered router/outbox and generation fencing so accepted writes are never lost or applied to only the retired collection.
• Hot swap alias: fast control-plane operation, but not an atomic transaction over independent blob stores, caches, CDC and policy ledgers. Verify those separately.
• Keeping blue for rollback: costs space and creates risk from stale ACLs/deletes; use a shared deny-serving tombstone gate plus replay of newer events.
• Resource isolation: speeds/stabilizes production queries but can increase infrastructure cost. Without isolation the rebuilt green index can consume all disk I/O and degrade blue even without a serving cutover.
• Freshness vs stability: delaying cutover for full reconciliation protects correctness, but extends how long the old search configuration is served. If blue no longer meets safety/quality gates, use an approved degraded retrieval mode or controlled fail-closed response.

6. Prevent repeat incidents

• Make versioned collections, stable aliases, immutable snapshots, durable CDC replay and guarded promotion part of the standard production pipeline.
• Maintain read p99/error and retrieval quality SLOs alongside write acceptance and source-to-index freshness SLOs.
• Rehearse failures: builder crash at 95%, replay duplicates/out-of-order deletes, change-log retention expiration, alias-update failure, ACL revocation immediately before cutover, stale replica, failed rollback and full disk.
• Monitor build progress, CPU/I/O impact on blue, event backlog, per-shard record/visibility counts, CDC offset lag, delete propagation, filter correctness and green canary quality.
• Use a sign-off checklist: manifest reconciled, event log replayed through a committed checkpoint, no security regressions, recall/latency validated, rollback index safe, and operator/audit evidence captured.

Interview takeaway: Do not rebuild the only serving index in place. Keep blue serving; build green from a snapshot, replay every intervening mutation, verify query-visible completeness and permissions, then atomically switch the read alias. A zero-downtime *pointer switch* is not enough unless writes, deletes, ACLs and caches are also consistent.

References

• Qdrant, Collection aliases (atomic switch): https://qdrant.tech/documentation/manage-data/collections/
• Weaviate, Collection aliases: https://docs.weaviate.io/weaviate/manage-collections/collection-aliases
• Weaviate, Zero-downtime collection migration: https://docs.weaviate.io/weaviate/tutorials/collection-aliases
• Elastic, Alias operations and reindex without downtime: https://www.elastic.co/docs/manage-data/data-store/aliases