A document is partially indexed before a deployment failure. How would you prevent users from seeing incomplete content?

Production incident (illustrative): A healthcare support assistant serves a 40-page policy document. A deployment starts indexing a new revision expected to produce 120 chunks, but the worker crashes after writing only 75. Users now retrieve sections from the new revision without the exclusions and eligibility rules that appear in the remaining 45 chunks. The assistant confidently gives incomplete advice. The indexing job succeeded for individual writes, but the document version was not fully published.

1. Contain the incident

• Immediately disable retrieval of the incomplete document version using a server-enforced visibility filter or active-version manifest. Never rely on the LLM prompt to ignore it.
• Continue serving the last fully verified document version if its contents and access permissions remain valid. Otherwise return a clear "source temporarily unavailable" response instead of a partial answer.
• Invalidate retrieval, reranker, answer, and citation caches that could contain the incomplete version. Preserve the failed run for investigation.

2. Trace the root cause

• Correlate tenant_id, stable document_id, document_version, ingestion_run_id, deployment identifier and timestamps across ingestion logs and index writes.
• Compare the parser's expected chunk manifest (IDs, page ranges, content hashes, embedding model version) against acknowledged vector writes, actual query-visible vectors, document-store objects, and ACL metadata.
• Check batch failures, retries, queue redelivery, indexing lag and whether partial writes were searchable immediately.
• Root cause: the pipeline treated successful chunk upserts as equivalent to an atomically published document. It exposed the new version before all required data was complete and query-visible.

3. Engineer a publish barrier

• Keep every new revision in STAGING with immutable versioned chunk IDs such as tenant/doc/version/chunk. Never overwrite active chunks in place.
• Parse, chunk, embed, and write the complete candidate version. Record a durable expected-chunk manifest and idempotency key. Retry only missing or failed chunks.
• Validate all required components: expected IDs and counts, content hashes, text/vector pairing, page coverage, embedding dimensions, metadata, citation targets, tenant scoping, and latest source ACLs. Validate actual read/search visibility, not just successful API responses.
• Only after validation, switch active_version from the old version to the new version using one transactional compare-and-swap in a strongly consistent metadata store. Every retrieval request must resolve that pointer and enforce a server-side version filter; bind the version for the whole retrieval-and-citation operation.
• Mark the candidate ACTIVE only after the pointer commit. Retain the previous complete version for rollback and garbage-collect old chunks after a safe retention period.
• For whole-index migrations, blue/green collections with an atomic alias switch can be used where the vendor guarantees it. Weaviate supports atomic alias updates; document-level publication still needs a version-aware serving layer unless each document has its own separately switchable collection.
• If a vector backend cannot reliably enforce version filters or read-after-write visibility, route new versions into isolated collections/namespaces and expose them only through a verified serving gateway. A SQL transaction cannot atomically commit arbitrary external vector writes.

4. Numerical example

• Version 7 contains 100 valid chunks and remains active. Version 8 expects 120 chunks.
• The failed deploy writes 75/120 = 62.5%. 45 chunks (37.5%) are missing, so the gate rejects publication; users continue retrieving the 100/100 complete chunks of version 7.
• The recovery worker replays the manifest and upserts only the missing 45 chunks, then verifies 120/120, matching hashes, ACLs and query visibility.
• A single metadata pointer update changes active_version: 7 → 8. New requests now see version 8; requests pinned to version 7 finish consistently.
• If version 8 fails a subsequent canary, switch the pointer back to version 7. No full re-embedding is needed.

5. Trade-offs and edge cases

• Staging costs extra storage and writes, but prevents a broken version from leaking into search. Old-version retention uses more space.
• Version filters avoid full-index duplication, but every query path (dense, sparse/hybrid, reranking and citations) must enforce them; a missed path can leak staging chunks.
• Aliases enable simple whole-collection cutovers, but may be expensive for single-document changes and cannot magically coordinate an external document store.
• Counts alone are insufficient: duplicate chunk IDs, missing pages, incorrect text, mismatched vectors or stale permissions can pass a count check. Use a manifest plus integrity and authorization checks.
• If the old version was revoked or contains sensitive content, fail closed rather than serve it. A deletion or permission revocation takes precedence over availability.
• Concurrent updates require generation fencing/compare-and-swap so a late retry cannot republish an older revision. Account for vector-index propagation lag and in-flight requests.

6. Prevent recurrence

• Treat ingestion as a state machine: DISCOVERED → STAGING → VALIDATING → READY → ACTIVE, with FAILED and ROLLED_BACK paths.
• Make write/replay steps idempotent and publication conditional on a verified manifest and source-version fence.
• Add deployment fault-injection tests: crash after 1%, 50%, and 99% of chunks; delayed indexing; duplicated events; ACL changes; concurrent versions.
• Alert on stuck staging jobs, expected-versus-visible chunk mismatch, staging chunks returned by retrieval, failed version swaps and partial citation coverage.
• Perform post-publish canary queries, check citations and permissions, and retain an audit trail of who or what promoted each version.

Interview takeaway: Make ingestion incremental but publication atomic from the user's perspective: write privately, validate completeness and security, then atomically change the serving pointer. Never let a successful chunk upsert imply that an entire document revision is ready.

References

• Weaviate: Collection aliases — https://docs.weaviate.io/weaviate/manage-collections/collection-aliases
• Weaviate: Zero-downtime collection migration — https://docs.weaviate.io/weaviate/tutorials/collection-aliases
• MachinoAI: Updating or deleting indexed RAG documents — https://www.machinoai.in/concepts/rag/how-do-you-update-or-delete-indexed-documents-in-a-rag-vector-database