A user uploads a document but cannot retrieve it immediately. How would you explain and manage read-after-write consistency?

Production incident (illustrative): A user uploads a new HR policy PDF at 10:00. The API responds 201 Created, but the user immediately asks the RAG assistant about it and gets “No relevant documents found.” At 10:01 the same question works. The upload succeeded; the searchable index had not caught up. This is a read-after-write consistency and user-expectation problem.

1. Explain the consistency boundary

A RAG ingestion system performs several distinct operations:

• Upload accepted: Original bytes are durably stored; this says nothing about the document's search visibility.
• Processing: Parsing, OCR, chunking, embeddings, and indexing are asynchronous and can take seconds or minutes.
• Write acknowledged: The vector or search backend accepts the write; depending on the engine, a refresh, replica propagation, or asynchronous indexing may still be pending.
• Searchable: A query on the *actual active retrieval path*, in the correct tenant/collection and under the user's permissions, can find the intended document version.

Read-after-write consistency means a client that completed a write can subsequently observe that write under the documented read interface. Ordinary asynchronous RAG ingestion often offers eventual search visibility, not immediate read-after-write search consistency. Do not equate upload succeeded with search-ready.

Elasticsearch documents that index writes are near-real-time for search: refresh=wait_for waits for a refresh to expose a write, while forcing refresh=true can hurt indexing throughput. These options govern Elasticsearch's index refresh; they do not make the upstream OCR/embedding pipeline synchronous. Weaviate documents tunable consistency and asynchronous indexing/replication considerations, so guarantees must be checked for the actual backend.

2. Diagnose where the delay occurs

1. Correlate upload_id, tenant_id, document_id, source version, ingestion_run_id, and index generation.
2. Record timestamps for upload stored, queue accepted, worker started, parsed, embedded, vector write acknowledged, index refreshed/visible, and first successful authorized retrieval.
3. Compare exact document-ID lookup with search using the correct tenant, active index alias, access filters, and source version. A document may exist in storage but not the searchable index.
4. Test whether the problem is indexing lag (queued work), refresh/replica lag (written but not visible), wrong index/namespace, permissions, or ranking (present but not in top-k).
5. Check p50/p95/p99 upload-to-search latency rather than relying on an average. Investigate outliers caused by scanned PDFs, rate limits, or worker backlog.
6. Retry visibility probes with a bounded deadline and backoff. Avoid assuming that one unsuccessful search means permanent data loss.

3. Beginner-friendly numerical example

Suppose one upload completes these stages:

• 10:00:00 — file saved; API returns 202 Accepted with document_id=doc-42.
• 10:00:04 — worker begins parsing.
• 10:00:12 — parsing and chunking complete.
• 10:00:20 — embeddings complete.
• 10:00:23 — vector upserts acknowledged.
• 10:00:27 — expected document/version verified through the active authorized search path.

Then:

• Upload-to-searchable latency: 27 seconds.
• Time after vector write acknowledgment but before visibility: 4 seconds.
• If the product promises 95% of eligible documents searchable within 60 seconds, this upload passes. One passing upload does *not* prove the overall 95% SLO; calculate the percentage across a representative time window and document types.

These numbers are illustrative. “Visible in exact-ID search” and “ranked highly for every natural-language query” are separate properties.

4. Engineering solution: make readiness explicit

1. Expose document states, such as UPLOADED → QUEUED → PROCESSING → INDEX_WRITTEN → SEARCHABLE (or FAILED/QUARANTINED). Record transitions durably and include document version.
2. Return the right API response: 202 Accepted plus a document ID and a status endpoint for asynchronous uploads. 201 Created may accurately describe resource creation, but the UI must not imply search readiness.
3. Provide status polling or events: Offer GET /documents/{id}/status, webhooks, or server-sent events so the client knows when search is ready. Use exponential polling backoff rather than querying continuously.
4. Gate `SEARCHABLE` on verification: Check the manifest's expected chunk count, correct generation and tenant, permitted read scope, and provider-appropriate search visibility. An upsert acknowledgment is insufficient.
5. Offer session-specific read-your-writes only when justified: For critical UX, temporarily route the uploader's questions to a securely authorized temporary document store or recently ingested chunks, clearly marked as processing. Ensure the same ACL checks, expiration, citation provenance, and bounded resource use; never bypass permissions or claim the document is in the global index.
6. Use backend-appropriate visibility controls: For Elasticsearch/OpenSearch, consider batched refresh=wait_for when a synchronous search-after-index contract is needed; avoid forcing a refresh after every bulk write. For vector databases, follow documented consistency controls or verify reads through the active query path.
7. Handle permanent failures: If deadline expires, mark DELAYED or FAILED, retain the original upload, explain the state to the user, and use idempotent retries/checkpoint recovery. Do not silently return a final “not found” for a known in-progress upload.

5. Trade-offs and prevention

• Synchronous wait: Simpler immediate UX but longer API response times, timeouts, and lower throughput for large documents.
• Asynchronous readiness: Scales better and avoids blocking uploads, but needs a clear progress/status experience.
• Forced refresh or stronger consistency: Reduces visibility delay in supported engines but can increase I/O and latency. It does not fix slow parsing or missing embeddings.
• Temporary session overlay: Can give the uploading user earlier answers, but introduces extra authorization, freshness, and consistency complexity.

Monitor upload-to-searchable latency by file type/tenant, queue wait time, embedding latency, index-refresh lag, stuck PROCESSING documents, the percentage meeting the freshness SLO, and successful authorized read-after-write canary checks. Test replicas, index aliases, permission changes, and delayed OCR under production-like load.

Interview takeaway: A successful upload is not the same as a successful searchable write. Make the consistency contract explicit, trace every stage, expose readiness states, measure upload-to-search latency, and use search-visibility confirmation or a carefully authorized read-your-writes path when the UX requires it.

Research references:

• Elasticsearch refresh parameter (official): https://www.elastic.co/docs/reference/elasticsearch/rest-apis/refresh-parameter
• Elasticsearch index refresh API (official): https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-refresh
• Weaviate consistency and replication (official): https://docs.weaviate.io/weaviate/concepts/replication-architecture/consistency