Your nightly sync accidentally removes 20% of production embeddings. How would you recover without a full rebuild?

Production incident (illustrative): An enterprise RAG system starts the morning with 1,000,000 active chunk embeddings. A nightly sync incorrectly interprets a temporarily empty source-connector page as a mass deletion and removes 200,000 vectors (20%). Search still responds, but users cannot find evidence from many documents. Re-embedding the entire corpus would be costly, slow, and unnecessary.

Goal: Contain the bad deletion, identify the exact missing document versions/chunks, recover only those vectors from a trustworthy source, and verify real search quality before declaring the incident resolved.

1. Contain the incident immediately

• Pause the faulty sync and destructive jobs. Disable automated delete propagation and suspend retries or subsequent batches that might delete more data. Preserve original logs, job IDs, index snapshots, and source manifests.
• Preserve the remaining good index. Take a provider-supported snapshot or backup of the current state where feasible. Do not overwrite the live collection with an older snapshot without understanding what legitimate writes happened after that snapshot.
• Keep retrieval safe. Mark affected document versions RECOVERING; avoid returning misleading “no evidence exists” for queries known to depend on missing content. If available, route affected requests to an authorized lexical index or validated replica, with clear degraded-service handling.
• Record the change boundary: Nightly sync run ID, source cursor, deletion request IDs, first and last affected timestamps, index generation, tenant IDs, and the deletion-selection predicate.

This is a partial data-loss incident. The first task is to stop the delete machinery, not to launch a full reindex.

2. Diagnose: did embeddings disappear, or did search stop seeing them?

1. Compare counts by tenant, namespace, index generation, source connector, document version, and shard. A 20% fall in collection point count suggests physical/logical deletion; unchanged counts with broken search may mean wrong aliases, filters, index refresh, or permissions.
2. Trace the nightly job's source manifest and delete-selection logic. Look for a failed pagination request, misread empty result, expired cursor, null source IDs, incorrect tenant scope, or “not seen this run ⇒ delete” code.
3. Inspect vector-database delete acknowledgments and operation logs to distinguish delete requests that were issued from deletes actually applied.
4. Compare the expected chunk-ID manifest with the actual active chunk IDs. Do not decide recovery solely from total record counts: counts can match despite the wrong chunks being present.
5. Determine the last good snapshot or replica and its timestamp. Identify newly created/updated documents and legitimate deletions since that point; simply rolling everything back could resurrect deleted or permission-revoked material.
6. Check for independent embedding artifacts or a durable embedding cache. If matching vectors exist, restore them directly without invoking the embedding model.
7. Audit access-control changes. A restored vector must use the current effective ACL, source version, and deletion tombstones, not stale permissions from an old backup.

3. Build a targeted recovery manifest

Maintain an authoritative inventory per expected document version:

• tenant_id, canonical_document_id, source_version, chunk_id, content_hash
• embedding_model_id, vector dimension/normalization, chunker_version, and index_generation
• Current source status and deletion tombstone, owner, authorization scope, and expected searchable state.

Compute:

missing_chunk_ids = eligible_expected_chunk_ids − actual_active_chunk_ids

This set difference must be scoped to the same tenant, document versions, model, and index generation. Exclude intentionally deleted, superseded, or inaccessible versions. Also inspect unexpected extra IDs to catch orphan records.

A documented manifest is superior to recomputing the entire corpus or relying on file counts.

4. Recover only affected embeddings

1. Preferred: verified existing replica/embedding cache. Read matching vector payloads and metadata from an unaffected, authorized replica or durable artifact store. Confirm hashes, model version, and dimensions.
2. Snapshot staging: Restore the last good provider snapshot into an isolated temporary collection or cluster, when supported. Extract only the missing eligible vectors into a staging index. Avoid restoring an old snapshot directly over the live index: doing so may lose newer writes or revive valid deletions.
3. Re-embed only as a fallback: For missing vectors absent from all trustworthy backups, retrieve the current source content, reuse the original chunker/model configuration, and embed only affected chunks. If the document has changed, process its current approved version instead of resurrecting stale chunks.
4. Idempotent restoration: Upsert using deterministic chunk_id and versioned metadata. Use bounded batches and per-tenant concurrency limits. Track progress in a durable recovery ledger.
5. Version and ACL fencing: Before making restored chunks searchable, re-check the source version, current ACLs, and deletion tombstones. Older recovery jobs must not override newer document states.
6. Validate and activate: Check expected versus restored chunk IDs, payload dimensions, searchable visibility, correct tenant isolation, citations, and representative queries before switching any route or marking document versions healthy.
7. Resume the connector cautiously: Fix the bad delete predicate, replay missed legitimate updates, canary the corrected sync, and enable deletions only after reconciliation passes.

Backend-specific caveat: A snapshot commonly restores a collection or index, not arbitrarily selected individual vectors into an existing live collection. For example, Qdrant documents collection snapshots and recovery priorities; Elasticsearch supports restoring selected indices and restoring into renamed indices. Use a staged restore + selective upserts pattern when your backend cannot do item-level snapshot restores. Snapshot compatibility, aliases, and replicas must be checked for that provider. Source: Qdrant snapshots https://qdrant.tech/documentation/snapshots/ ; Elasticsearch snapshot and restore https://www.elastic.co/docs/deploy-manage/tools/snapshot-and-restore

5. Beginner-friendly numerical example

Assume:

• Before incident: 1,000,000 valid chunk vectors
• Accidentally deleted: 200,000
• Surviving: 800,000
• Of the missing vectors, 150,000 are available in a compatible embedding artifact backup and 50,000 must be regenerated.
• Embedding cost for this example: $0.0001 per chunk (illustrative; real providers often price by tokens).

Then:

• Lost fraction: 200,000 / 1,000,000 × 100 = 20%.
• Direct vector restore: 150,000 vectors, with no new embedding inference.
• Re-embedding: 50,000 chunks at $0.0001 = $5.
• Full rebuild alternative: 1,000,000 × $0.0001 = $100.
• Illustrative embedding-charge saving: $95, excluding backup recovery, storage, vector writes, labor, and validation.

Suppose each worker safely restores/upserts 1,000 vectors per minute, and 5 workers can operate without saturating the index: 200,000 / (5 × 1,000) = 40 minutes of theoretical write work. Real recovery time also includes snapshot access, embedding generation, index visibility, and validation, so 40 minutes is not an SLA.

The target is 1,000,000 correct, currently eligible vectors, not merely returning to a count of one million; some source documents might have changed after the incident.

6. Root-cause fix: make deletions conservative and reversible

• Don't use missing-from-one-scan as proof of deletion. Require authoritative source tombstones or a completed, validated full manifest, and distinguish access errors/empty pages from confirmed removal.
• Apply a deletion safety gate: If a sync proposes removing an anomalously large fraction (e.g., >5% in one run, a configurable illustrative threshold), hold the operation for review instead of automatically executing.
• Two-phase deletion: First mark candidates PENDING_DELETE/tombstoned, reconcile with source and entitlement checks, then perform physical cleanup after validation and a suitable grace period. Security revocations still need immediate query-time denial, even if physical cleanup is delayed.
• Use versioned, auditable delete events: Include source cursor, tenant, document IDs, reason, and batch size; enforce idempotency and conditional source-version checks.
• Protect backups and the recovery manifest from the same faulty connector or credentials that issue deletes.
• Test failure modes: Missing pagination page, connector returning zero due to authorization loss, partial sync, clock skew, stale cursors, and accidental tenant-wide deletes.

7. Verification and prevention

• Reconcile exact eligible chunk IDs, document versions, and metadata against the manifest; investigate unexpected extras.
• Run exact-ID, keyword, and semantic-search canaries on representative missing and surviving documents, plus negative tests ensuring deleted/restricted documents stay invisible.
• Monitor deletion rate, anomalous batch sizes, source-to-index drift, recovery coverage, index alias changes, stale-source versions, snapshot age, and time since last successful tested restore.
• Record affected customer queries, restore decisions, recovery start/end, responsible job IDs, and postmortem actions. Document the actual recovery-time and recovery-point objectives.
• Re-run the corrected sync in shadow/dry-run mode before allowing another destructive batch.

Trade-offs: A recoverable staged snapshot costs temporary storage and I/O but avoids harming current writes. A delayed delete policy reduces accidental data loss but must be paired with immediate tombstone-based access denial. Targeted restoration is more operationally complex than a full rebuild, yet preserves newer valid data and dramatically reduces unnecessary embedding work.

Interview takeaway: Freeze destructive sync, diff expected and active chunk IDs, restore only eligible missing vectors from a safe replica/cache/staged snapshot, re-embed only what cannot be recovered, and verify current versions plus ACLs before reactivation. A fixed count is not proof of correct recovery.

Further reading: Qdrant Snapshots; Qdrant Migration and Recovery; Elasticsearch Restore from Snapshot.