A production vector database becomes unavailable. How would you restore search using replicas, backups, or degraded retrieval?

Production incident (illustrative): At 14:00, a multi-tenant RAG support assistant starts returning errors because its vector-database cluster becomes unavailable after a storage failure. The cluster holds 8 million active embeddings, spread across four shards. The system has replicas, nightly backups, and a keyword-search index, but the on-call team has never tested which recovery path preserves the latest document deletions and ACL changes. Repeated retry storms overwhelm the surviving components.

Goal: Restore safe, useful search, not merely a database health-check response. Decide among replica failover, backup restoration, and approved degraded retrieval using measured availability, data currency, security and business recovery objectives.

1. Declare and contain the incident

• Confirm the boundary: vector database only, network/DNS, client credentials, coordinator, a subset of shards, or entire region. Check connection failures, healthy peers, shard allocation, coordinator logs, storage and quorum/control-plane status.
• Freeze risky schema changes, index rebuilds and bulk ingestion; preserve write-ahead/change logs and recovery evidence. Reduce retry amplification with circuit breakers, bounded exponential backoff, jitter and admission control.
• Establish incident commander, timestamp, impacted tenants/collections and latest acknowledged source changes. Capture the last known-good search checkpoint, versioned ACL/tombstone ledger and recovery-state snapshots.
• Fail closed for sensitive content unless its current tenant permissions and deletion status can be enforced. Never surface a stale document just because its vector exists on an available backup.
• Keep user messaging specific: “Search is temporarily limited; some results may be unavailable.” Do not claim fresh or complete retrieval during fallback.

2. Diagnose which recovery path is viable

• Healthy replica exists: inspect per-shard replica state, readiness, allocation, replication lag, read consistency, disk health and recent ACL/deletion propagation. Redirect reads only to a replica that satisfies the application’s security and freshness contract. Confirm all required shards are covered; having two replicas of one shard does not repair a missing different shard.
• Replica lag or corruption: compare source/record versions and durable operation offsets with the last acknowledged writes. A healthy TCP endpoint is not enough if the query-visible index is stale.
• No safe replica: identify latest verified and compatible backup/snapshot plus necessary cluster schema, shard configuration, aliases and key material. Check restore prerequisites, available disk, snapshot integrity, backend version and whether recovery includes all required shards/collections.
• Backups older than deletion/ACL changes: replay immutable, ordered CDC/outbox or source change events including tombstones and ACL modifications before permitting the restored index to serve. If replay is impossible, use a central current authorization/deletion gate and quarantine questionable records; do not treat the old backup as fully safe.
• Database not immediately recoverable: activate an approved degraded retrieval mode: keyword/BM25 search over a current authorized index, direct authoritative document or API lookup, or restricted cached answers only when tenant, version, ACL and deletion eligibility can be revalidated. Otherwise abstain.

3. Engineer replica failover first when safe

• Validate coverage of every serving shard, replica health, replication offset/lag, search readiness and index consistency; compare deterministic canary IDs under tenant/ACL filters.
• Use a supported load-balancer/DB routing mechanism to exclude unhealthy nodes; preserve stable collection/alias names and consistent client endpoints. Avoid split-brain writes and force-promoting stale replicas without reviewing quorum guarantees.
• Resume reads under bounded concurrency, then reintroduce writes according to the vendor’s tested failover process. Reconcile any writes accepted during partial outage using idempotent event IDs and version fences.
• Measure success with per-shard availability, search error rate, p95/p99, retrieval quality and absence of revoked records—not health checks alone.
• Vendor notes: replication and availability guarantees differ by product and version. Elasticsearch may allocate regular replica shards after primary failure; Qdrant distributes collection shards and replicas. Verify whether quorum, shard coverage and read consistency meet your actual requirements.

4. Restore from a verified backup if replica recovery fails

• Bring up an isolated target collection/cluster; never blindly overwrite a partially healthy primary. Restore all required shards, payloads, schema/config, index settings and metadata, then reconstruct serving aliases or routing.
• Qdrant’s collection snapshots do not include aliases; in distributed installations snapshots are node-specific, so verify complete shard coverage. Check restore priority and version compatibility before applying snapshots to a live cluster.
• Weaviate supports live backups with staged restore states; verify restore completion and post-restore replication/visibility. Elasticsearch snapshots may also include relevant cluster state and feature configuration; inventory these separately from application-level RAG metadata.
• Replay writes, deletes, versions and ACL changes after the snapshot checkpoint in order, with idempotency and generation fencing. Confirm manifests, active document counts, search-visible IDs and revoked/deleted documents.
• Canary dense/hybrid queries, citations, cross-tenant isolation and freshness. Atomically switch the serving alias/application pointer only after documented gates pass. Keep source version and snapshot/checkpoint IDs in the audit trail.

5. Degraded retrieval without hallucinating recovery

• Keep search UI/API up with a feature flag for approved fallback. Use existing authorized keyword search or live document-source APIs; label that semantic ranking is temporarily unavailable.
• Apply the same tenant, ACL, effective-version and deletion filters as the vector path. Do not use old result caches unless security and version eligibility can be checked at serve time.
• Separate failure classes: if keyword results are incomplete but safe, answer with citations and an explicit limited-search notice. If the user requests a high-stakes current policy and freshness cannot be established, decline and route to human support.
• Track partial-coverage and empty-result rates, not just HTTP success. Run fallback-specific relevance tests and log which retrieval mode produced each answer.
• Deactivate fallback only after canaries confirm the recovered vector index meets agreed freshness, recall and SLOs.

6. Numerical incident and RPO/RTO example

• At 14:00, the vector DB fails. At 14:03, traffic is safely switched to a pre-tested BM25 fallback: degraded-service recovery time 3 minutes, but *full semantic-search RTO has not been met*.
• Healthy replica inspection fails because one of four shard groups has no sufficiently current safe copy. Latest verified full snapshot is 12:00, with 8,000,000 active vectors at that checkpoint.
• A durable CDC stream records 120 updates per minute between 12:00 and 14:00: 120 × 120 = 14,400 events to inspect/replay, including 300 deletes/ACL revocations. Some updates replace existing vectors, so 14,400 events is not a claim that 14,400 extra vectors should exist.
• At 14:10, isolated snapshot restoration starts. Suppose restoring and validating base shards takes 30 minutes, ending 14:40. Replay then runs at 600 events/minute while new changes arrive at 120/minute: net drain = 480/minute. Initial 14,400 / 480 = 30 minutes to catch up in this simplified model, ending around 15:10, assuming the stream contains all changes and there are no replays/errors.
• At 15:15, after index/ACL/deletion validation and a guarded cutover, full search resumes. Full-service RTO = 75 minutes from 14:00, while degraded retrieval was available after 3 minutes. If the business full-service RTO target is 60 minutes, the incident misses it by 15 minutes.
• The backup alone represents a 120-minute recovery-point gap at the time of failure. If the CDC ledger is complete and replayed through a committed cutover checkpoint, effective data loss can be zero for changes durably captured in that ledger; if not, the actual RPO is determined by the last fully recovered change, not backup timestamp alone.
• Success gates: all four shards healthy, source-to-index manifest reconciled, all 300 sensitive deletes/revocations verified, 1,000/1,000 representative ACL canaries pass, no known revoked document appears, and retrieval-quality/latency metrics meet agreed thresholds. Figures are illustrative.

7. Trade-offs and failure modes

• Replica failover: fastest when a complete healthy copy exists; uses extra hardware and can serve stale data if replication/read consistency is misconfigured. A replica is not an independent backup against accidental deletion or logical corruption.
• Backup restore: can recover from lost nodes/regions or logical corruption, but incurs RTO, storage/egress costs, compatibility checks and replay burden. A backup must be routinely tested, not merely created.
• Degraded keyword/direct retrieval: preserves some functionality with fewer vector dependencies, but usually reduces semantic recall and may change latency. Explicitly disclose its coverage and avoid invented answers.
• Serving stale cache: potentially fast but risky after source deletions, permission changes or pricing/policy updates; use only if current authorization and eligibility can be validated.
• Tighter RPO: requires more frequent snapshots or durable change streams; increases storage and replication cost. Tighter RTO: requires warm capacity, rehearsed failover and ready fallback dependencies.
• Partial shard coverage: results may look plausible but omit documents; fail closed or clearly label restricted scope rather than silently reporting complete search.
• Multi-region replication: helps regional outages but demands careful split-brain, consistency, tenancy and data-residency design.

8. Prevent recurrence

• Define separate full search RTO, degraded-mode RTO, RPO, freshness and retrieval quality targets with business owners. Know which application functions truly require semantic search.
• Ensure shard/replica placement across failure domains, durable offsite backups, restore-key access and a replayable version/deletion/ACL event log.
• Regularly test full restore into an isolated environment, shard/alias reconstruction, compatibility upgrades, and safe replay of mutations since snapshot.
• Run chaos drills: single replica down, entire shard missing, zone outage, snapshot corruption, expired log retention, replay duplicates, revoked ACL immediately before failover and keyword fallback outages.
• Monitor per-shard readiness, replica lag, backup age and success, restore duration, CDC backlog, canary deletion visibility, error budgets and fallbacks; alert before the disaster consumes the recovery window.
• Maintain an operator runbook with decision authority, tested commands, contacts, cutover/rollback gates, user communication and post-incident audit evidence.

Interview takeaway: Restore search through a decision tree: use current healthy replicas when safe; otherwise restore all shards from a verified backup and replay post-snapshot changes; while recovering, serve explicitly degraded authorized keyword/direct retrieval or abstain. Measure RTO, RPO, recall and security separately.

Further reading

• Qdrant: Snapshots and restore — https://qdrant.tech/documentation/snapshots/
• Weaviate: Backups and restore — https://docs.weaviate.io/deploy/configuration/backups
• Elasticsearch: Snapshot and restore — https://www.elastic.co/docs/deploy-manage/tools/snapshot-and-restore
• Elasticsearch: Searchable snapshots — https://www.elastic.co/docs/deploy-manage/tools/snapshot-and-restore/searchable-snapshots