Your vector database reports successful writes, but new embeddings are not searchable. How would you investigate index refresh and consistency?

Production incident (illustrative): A SaaS knowledge assistant writes 40,000 new product-support embeddings to its production vector database during a peak-hour migration. Upsert responses show success, and an exact-ID lookup finds the new points, but filtered similarity searches return old articles or no results for recently added documents. Search p99 remains acceptable, so uptime monitoring misses the incident. Support agents make decisions using stale evidence.

The distinction: write accepted ≠ write durably applied ≠ vector indexed ≠ query-visible on the replica actually serving a request ≠ top-k retrieval guaranteed. The meaning of an acknowledged write varies by engine, client options, async-index configuration, replication consistency and search mode. First locate the lost visibility boundary; do not immediately re-embed everything.

1. Contain and reproduce the failure

• Tag affected documents with tenant_id, immutable point_id, document_version, embedding_model_version, collection, shard, replica, write_operation_id, write_ack_at, and first_search_visible_at. Collect a small set of known new IDs.
• If freshness is critical, temporarily route those cases to an approved exact/authoritative lookup or abstain with a transparent indexing pending status. Do not relax tenant/ACL filters or silently return outdated data.
• Issue four probes against the actual production collection/alias: (1) direct lookup by point ID, (2) unfiltered exact/flat vector search where supported, (3) unfiltered ANN search using the identical vector and distance metric, and (4) production filtered ANN search with the same tenant, ACL, effective-version and index settings.
• Compare results on every replica or consistency level the client can target, with a fixed query vector, top-k and query timestamp. A nearest-neighbor search may omit a present point because it scores below top-k; force a controlled self-query/candidate set or exact baseline before calling this a visibility failure.
• Record queue/optimizer depth, pending writes, indexed_vectors_count or equivalent, deferred/unindexed points, shard/replica readiness, update status, refresh operations, error logs and query-cache versions.

2. Root-cause decision tree

• ID lookup fails: check whether the write was only accepted or queued, rejected on an individual batch item, sent to the wrong collection/namespace/shard, overwritten by an older event or deleted by a concurrent sync. Check durable WAL/operation status and idempotency keys.
• ID lookup succeeds, exact search fails: investigate search-snapshot/refresh boundary, index visibility, incorrect distance/normalization and filter eligibility. An exact API may still use a specific snapshot or replica; compare consistent snapshots.
• Exact search succeeds, ANN fails: inspect indexing backlog, segment optimization, ANN search parameters, selective filters, index health and whether the new point is actually included in searchable ANN structures.
• Unfiltered works, production filtered fails: verify tenant field types and values, ACL propagation, active_version, effective dates and metadata/payload indexes. Fix metadata or filter construction rather than increasing ef_search blindly. Never remove mandatory security filters in production.
• One replica works, another does not: investigate asynchronous replication, routing, lagging peers, stale reads and read-consistency controls; an acknowledged write by one replica does not necessarily establish visibility on all replicas.
• Only a cached search result fails: inspect query-response cache TTL, routing alias, embedding cache and invalidation keys, including tenant/ACL/version identifiers.

3. Understand the provider-specific behavior

• Elasticsearch: a successful index operation can precede the next search refresh. The write API's refresh=wait_for waits for a refresh to expose that write; refresh=true forces refresh and can degrade throughput. Check the real index.refresh_interval, refresh counters, alias, active shards and failed bulk items. Avoid assuming a single forced refresh solves asynchronous upstream ingestion.
• Weaviate: optional asynchronous vector indexing decouples object-store persistence from ANN index updates. With async HNSW/HFresh indexing, newly imported objects may exist before they appear in vector search; inspect node indexing status and persistent queue/backlog, including whether updates/deletions are still queued.
• Qdrant: inspect wait/ordering settings and operation status. Under normal operation wait=true waits for application, but indexed_only=true can omit larger unindexed segments. The experimental prevent_unoptimized mode (introduced in v1.17.1) deliberately defers some freshly written points from searches until optimization completes, even though writes have been durably accepted. In that mode, use wait=false as documented and monitor deferred_points; wait=true can block for a long time. Confirm version-specific behavior before changing settings.
• All engines: distinguish index refresh (search snapshot publication), ANN graph/segment building (search structure readiness), replication visibility (peer synchronization), and application document publication (authorized version selection). They are different gates.

4. Numerical example: measure the precise gap

• At 10:00:00, the system issues 40,000 upserts to a 4-replica collection; all batch requests return acknowledged success. That alone does not prove per-point search visibility.
• At 10:00:15, exact-ID probes find all 40,000 expected points, while production ANN searches using a controlled known-ID test return 36,000. An indexed/search-visible coverage estimate is 36,000 / 40,000 = 90%; 4,000 = 10% are not query-visible under those test conditions.
• Inspecting the affected shard finds 3,200 deferred/unindexed points under a read mode that excludes them, plus 800 points on a replica whose indexing/refresh state lags. This is an illustrative, disjoint attribution, not a universal vendor counter.
• A controlled route to the current healthy replica raises test visibility to 39,200 / 40,000 = 98% while the indexer catches up. After queue/optimizer drainage and read-path verification, all 40,000 / 40,000 = 100% self-query checks pass. Real production top-k recall must still be evaluated separately; 100% self-query discoverability does not mean 100% semantic retrieval quality.
• For a sampled point, write_ack_at = 10:00:00 and first_search_visible_at = 10:02:30; its ack-to-search visibility lag is 150 seconds. If the internal SLO is ≤60 seconds, that point breaches it by 90 seconds. To compute SLO compliance over the 40,000 points, count how many became visible within 60 seconds, including timeouts/censored observations rather than excluding missing points.
• Illustrative diagnosis: the indexer falls behind under bulk writes while application queries set indexed_only; read routing sometimes selects a delayed replica. Mitigation is not a global re-embedding run: restore indexer throughput, correct the read mode/consistency contract and verify query visibility.

5. Engineering solution

• Make write acknowledgments explicit: represent stages as ACCEPTED → APPLIED → INDEXING → SEARCH_VISIBLE → VERIFIED with durable operation IDs and versioned point manifests; do not expose a document as search-ready based solely on HTTP 200.
• Create a visibility barrier: for critical workflows, verify the intended vector/version through the *same authorized production read path*, with bounded retries/backoff and sufficient exact or self-query evidence. Only then promote a document's searchable version. Avoid holding all client writes synchronously for slow background indexing.
• Configure engine-native refresh appropriately: choose targeted refresh=wait_for in Elasticsearch when immediate search visibility is required, not refresh=true on every bulk request. For async Weaviate/Qdrant modes, observe indexer/deferred queues and deliver an explicit eventual-visibility contract instead of pretending the write is synchronous.
• Reconcile replicas: monitor peer lag and stale routing, use supported read consistency levels for important read-after-write paths, and route away from genuinely unhealthy peers after checking current ACL/deletion visibility.
• Repair instead of rebuilding: replay only missing/incorrect IDs using immutable chunk keys and idempotent writes; repair wrong metadata/ACLs, index settings or aliases; run a rebuild only if verified index corruption or incompatible configuration justifies it.
• Protect correctness: maintain a source-side deletion/tombstone gate and authoritative ACL checks during any stale-replica or cache fallback. An older replica is never a safe substitute when it can expose revoked content.

6. Trade-offs and failure modes

• Stronger read consistency / refresh waiting: better read-after-write guarantees, but can increase write latency, CPU, I/O and replica coordination; set an appropriate SLA rather than applying it universally.
• Async indexing: higher ingestion throughput with delayed search visibility; monitor queue age and add backpressure before freshness SLOs are breached.
• `indexed_only` / `prevent_unoptimized`: can stabilize latency at the cost of temporary search invisibility; the exact semantics are vendor- and version-specific, and Qdrant's modes should not be blindly combined.
• Self-query probes: great for visibility smoke tests, but they don't measure user-question relevance or prove all production filters work. Evaluate separate Recall@k and filtered retrieval quality.
• Aggressive retries: may amplify load, cause duplicate events and worsen queues. Use bounded retries, idempotency, jitter and a dead-letter workflow for persistent failures.
• Automatic replica failover: improves availability but may expose stale results; respect version fences, deletion/ACL safety and consistency requirements.
• Full forced refresh or rebuild: may reduce temporary invisibility but risks expensive I/O, write stalls, cold caches and latency regression.

7. Prevent recurrence

• Track write_ack → applied → index_ready → search_visible timestamps by collection, tenant and shard, with p50/p95/p99 visibility lag, queue depth, deferred points, refresh counts, failed bulk items and per-replica divergence.
• Add end-to-end canary writes: create an authorized synthetic vector with a unique ID, verify direct lookup and production filtered ANN self-query within a bounded SLO, then clean it up and confirm deletion visibility.
• Run fault-injection tests for high-volume writes, disabled optimization, delayed refresh, replica lag, wrong alias, missing ACL payload, out-of-order updates and cache staleness. Verify no revoked document becomes visible in a fallback.
• Alert on divergence between acknowledged active IDs and search-visible IDs, stale indexed_only behavior, growing async queues and SLO breaches. Instrument client defaults during version upgrades.
• Document the database's exact acknowledgment, refresh, indexing and replica-consistency semantics in a tested production runbook; revisit after backend upgrades.

Interview takeaway: When writes succeed but new embeddings cannot be found, prove where visibility stops using ID lookup, exact search, ANN search and production-filtered search, then compare replicas and indexer state. Fix the specific refresh, async-index, filter, routing or replica issue; don't claim that a successful upsert guarantees discoverability.

Sources

• Elasticsearch: Index API refresh parameter — https://www.elastic.co/docs/reference/elasticsearch/rest-apis/refresh-parameter
• Weaviate: Vector indexing and asynchronous indexing — https://docs.weaviate.io/weaviate/concepts/vector-index
• Qdrant: Optimizer, deferred points and write-wait semantics — https://qdrant.tech/documentation/ops-optimization/optimizer/
• Qdrant: Low-latency search and indexed-only visibility — https://qdrant.tech/documentation/search/low-latency-search/