Retrieval-Augmented Generation
Retrieval-augmented generation connects a generative model to a searchable knowledge source. Instead of relying only on model weights, the system fetches relevant documents and places them into the prompt or context.
Your vector database reports successful writes, but new embeddings are not searchable. How would you investigate index refresh and consistency?
Production incident (illustrative): A SaaS knowledge assistant writes 40,000 new product support embeddings to its production vector database during a peak hour migration. Upsert responses show success, and an exact ID lookup finds the new points, but filtered similarity searches return old articles or no results for recently added documents. Search p99 remains acceptable, so uptime monitoring misses the incident. Support agents make decisions using stale evidence. The distinction: write accepted ≠ write durably applied ≠ vector indexed ≠ query visible on the replica actually serving a request ≠ top k retrieval guaranteed. The meaning of an acknowledged write varies by engine, client options, async index configuration, replication consistency and search mode. First locate the lost visibility boundary; do not immediately re embed everything. 1. Contain and reproduce the failure Tag affected documents with tenant id, immutable point id, document version, embedding model version, collection, shard, replica, write operation id, write ack at, and first search visible at. Collect a small set of known new IDs. If freshness is critical, temporarily route those cases to an approved exact/authoritative lookup or abstain with a transparent indexing pending status. Do not relax tenant/ACL filters or silently return outdated data. Issue four probes against the actual production collection/alias : (1) direct lookup by point ID, (2) unfiltered exact/flat vector search where supported, (3) unfiltered ANN search using the identical vector and distance metric, and (4) production filtered ANN search with the same tenant, ACL, effective version and index settings. Compare results on every replica or consistency level the client can target, with a fixed query vector, top k and query timestamp. A nearest neighbor search may omit a present point because it scores below top k; force a controlled self query/candidate set or exact baseline before calling this a visibility failure. Record queue/optimizer depth, pending writes, indexed vectors count or equivalent, deferred/unindexed points, shard/replica readiness, update status, refresh operations, error logs and query cache versions. 2. Root cause decision tree ID lookup fails: check whether the write was only accepted or queued, rejected on an individual batch item, sent to the wrong collection/namespace/shard, overwritten by an older event or deleted by a concurrent sync. Check durable WAL/operation status and idempotency keys. ID lookup succeeds, exact search fails: investigate search snapshot/refresh boundary, index visibility, incorrect distance/normalization and filter eligibility. An exact API may still use a specific snapshot or replica; compare consistent snapshots. Exact search succeeds, ANN fails: inspect indexing backlog, segment optimization, ANN search parameters, selective filters, index health and whether the new point is actually included in searchable ANN structures. Unfiltered works, production filtered fails: verify tenant field types and values, ACL propagation, active version, effective dates and metadata/payload indexes. Fix metadata or filter construction rather than increasing ef search blindly. Never remove mandatory security filters in production. One replica works, another does not: investigate asynchronous replication, routing, lagging peers, stale reads and read consistency controls; an acknowledged write by one replica does not necessarily establish visibility on all replicas. Only a cached search result fails: inspect query response cache TTL, routing alias, embedding cache and invalidation keys, including tenant/ACL/version identifiers. 3. Understand the provider specific behavior Elasticsearch: a successful index operation can precede the next search refresh. The write API's refresh=wait for waits for a refresh to expose that write; refresh=true forces refresh and can degrade throughput. Check the real index.refresh interval, refresh counters, alias, active shards and failed bulk items. Avoid assuming a single forced refresh solves asynchronous upstream ingestion. Weaviate: optional asynchronous vector indexing decouples object store persistence from ANN index updates. With async HNSW/HFresh indexing, newly imported objects may exist before they appear in vector search; inspect node indexing status and persistent queue/backlog, including whether updates/deletions are still queued. Qdrant: inspect wait/ordering settings and operation status. Under normal operation wait=true waits for application, but indexed only=true can omit larger unindexed segments. The experimental prevent unoptimized mode (introduced in v1.17.1) deliberately defers some freshly written points from searches until optimization completes, even though writes have been durably accepted. In that mode, use wait=false as documented and monitor deferred points; wait=true can block for a long time. Confirm version specific behavior before changing settings. All engines: distinguish index refresh (search snapshot publication), ANN graph/segment building (search structure readiness), replication visibility (peer synchronization), and application document publication (authorized version selection). They are different gates. 4. Numerical example: measure the precise gap At 10:00:00 , the system issues 40,000 upserts to a 4 replica collection; all batch requests return acknowledged success. That alone does not prove per point search visibility. At 10:00:15 , exact ID probes find all 40,000 expected points, while production ANN searches using a controlled known ID test return 36,000 . An indexed/search visible coverage estimate is 36,000 / 40,000 = 90% ; 4,000 = 10% are not query visible under those test conditions. Inspecting the affected shard finds 3,200 deferred/unindexed points under a read mode that excludes them, plus 800 points on a replica whose indexing/refresh state lags. This is an illustrative, disjoint attribution , not a universal vendor counter. A controlled route to the current healthy replica raises test visibility to 39,200 / 40,000 = 98% while the indexer catches up. After queue/optimizer drainage and read path verification, all 40,000 / 40,000 = 100% self query checks pass. Real production top k recall must still be evaluated separately; 100% self query discoverability does not mean 100% semantic retrieval quality. For a sampled point, write ack at = 10:00:00 and first search visible at = 10:02:30; its ack to search visibility lag is 150 seconds . If the internal SLO is ≤60 seconds , that point breaches it by 90 seconds . To compute SLO compliance over the 40,000 points, count how many became visible within 60 seconds, including timeouts/censored observations rather than excluding missing points. Illustrative diagnosis: the indexer falls behind under bulk writes while application queries set indexed only; read routing sometimes selects a delayed replica. Mitigation is not a global re embedding run: restore indexer throughput, correct the read mode/consistency contract and verify query visibility. 5. Engineering solution Make write acknowledgments explicit: represent stages as ACCEPTED → APPLIED → INDEXING → SEARCH VISIBLE → VERIFIED with durable operation IDs and versioned point manifests; do not expose a document as search ready based solely on HTTP 200. Create a visibility barrier: for critical workflows, verify the intended vector/version through the same authorized production read path , with bounded retries/backoff and sufficient exact or self query evidence. Only then promote a document's searchable version. Avoid holding all client writes synchronously for slow background indexing. Configure engine native refresh appropriately: choose targeted refresh=wait for in Elasticsearch when immediate search visibility is required, not refresh=true on every bulk request. For async Weaviate/Qdrant modes, observe indexer/deferred queues and deliver an explicit eventual visibility contract instead of pretending the write is synchronous. Reconcile replicas: monitor peer lag and stale routing, use supported read consistency levels for important read after write paths, and route away from genuinely unhealthy peers after checking current ACL/deletion visibility. Repair instead of rebuilding: replay only missing/incorrect IDs using immutable chunk keys and idempotent writes; repair wrong metadata/ACLs, index settings or aliases; run a rebuild only if verified index corruption or incompatible configuration justifies it. Protect correctness: maintain a source side deletion/tombstone gate and authoritative ACL checks during any stale replica or cache fallback. An older replica is never a safe substitute when it can expose revoked content. 6. Trade offs and failure modes Stronger read consistency / refresh waiting: better read after write guarantees, but can increase write latency, CPU, I/O and replica coordination; set an appropriate SLA rather than applying it universally. Async indexing: higher ingestion throughput with delayed search visibility; monitor queue age and add backpressure before freshness SLOs are breached. indexed only / prevent unoptimized: can stabilize latency at the cost of temporary search invisibility; the exact semantics are vendor and version specific, and Qdrant's modes should not be blindly combined. Self query probes: great for visibility smoke tests, but they don't measure user question relevance or prove all production filters work. Evaluate separate Recall@k and filtered retrieval quality. Aggressive retries: may amplify load, cause duplicate events and worsen queues. Use bounded retries, idempotency, jitter and a dead letter workflow for persistent failures. Automatic replica failover: improves availability but may expose stale results; respect version fences, deletion/ACL safety and consistency requirements. Full forced refresh or rebuild: may reduce temporary invisibility but risks expensive I/O, write stalls, cold caches and latency regression. 7. Prevent recurrence Track write ack → applied → index ready → search visible timestamps by collection, tenant and shard, with p50/p95/p99 visibility lag , queue depth, deferred points, refresh counts, failed bulk items and per replica divergence. Add end to end canary writes : create an authorized synthetic vector with a unique ID, verify direct lookup and production filtered ANN self query within a bounded SLO, then clean it up and confirm deletion visibility. Run fault injection tests for high volume writes, disabled optimization, delayed refresh, replica lag, wrong alias, missing ACL payload, out of order updates and cache staleness. Verify no revoked document becomes visible in a fallback. Alert on divergence between acknowledged active IDs and search visible IDs, stale indexed only behavior, growing async queues and SLO breaches. Instrument client defaults during version upgrades. Document the database's exact acknowledgment, refresh, indexing and replica consistency semantics in a tested production runbook; revisit after backend upgrades. Interview takeaway: When writes succeed but new embeddings cannot be found, prove where visibility stops using ID lookup, exact search, ANN search and production filtered search, then compare replicas and indexer state. Fix the specific refresh, async index, filter, routing or replica issue; don't claim that a successful upsert guarantees discoverability. Sources Elasticsearch: Index API refresh parameter — https://www.elastic.co/docs/reference/elasticsearch/rest apis/refresh parameter Weaviate: Vector indexing and asynchronous indexing — https://docs.weaviate.io/weaviate/concepts/vector index Qdrant: Optimizer, deferred points and write wait semantics — https://qdrant.tech/documentation/ops optimization/optimizer/ Qdrant: Low latency search and indexed only visibility — https://qdrant.tech/documentation/search/low latency search/
A production vector database becomes unavailable. How would you restore search using replicas, backups, or degraded retrieval?
Production incident (illustrative): At 14:00, a multi tenant RAG support assistant starts returning errors because its vector database cluster becomes unavailable after a storage failure. The cluster holds 8 million active embeddings , spread across four shards. The system has replicas, nightly backups, and a keyword search index, but the on call team has never tested which recovery path preserves the latest document deletions and ACL changes. Repeated retry storms overwhelm the surviving components. Goal: Restore safe, useful search , not merely a database health check response. Decide among replica failover , backup restoration , and approved degraded retrieval using measured availability, data currency, security and business recovery objectives. 1. Declare and contain the incident Confirm the boundary: vector database only, network/DNS, client credentials, coordinator, a subset of shards, or entire region. Check connection failures, healthy peers, shard allocation, coordinator logs, storage and quorum/control plane status. Freeze risky schema changes, index rebuilds and bulk ingestion; preserve write ahead/change logs and recovery evidence. Reduce retry amplification with circuit breakers, bounded exponential backoff, jitter and admission control. Establish incident commander, timestamp, impacted tenants/collections and latest acknowledged source changes. Capture the last known good search checkpoint , versioned ACL/tombstone ledger and recovery state snapshots. Fail closed for sensitive content unless its current tenant permissions and deletion status can be enforced. Never surface a stale document just because its vector exists on an available backup. Keep user messaging specific: “Search is temporarily limited; some results may be unavailable.” Do not claim fresh or complete retrieval during fallback. 2. Diagnose which recovery path is viable Healthy replica exists: inspect per shard replica state, readiness, allocation, replication lag, read consistency, disk health and recent ACL/deletion propagation. Redirect reads only to a replica that satisfies the application’s security and freshness contract. Confirm all required shards are covered; having two replicas of one shard does not repair a missing different shard. Replica lag or corruption: compare source/record versions and durable operation offsets with the last acknowledged writes. A healthy TCP endpoint is not enough if the query visible index is stale. No safe replica: identify latest verified and compatible backup/snapshot plus necessary cluster schema, shard configuration, aliases and key material. Check restore prerequisites, available disk, snapshot integrity, backend version and whether recovery includes all required shards/collections. Backups older than deletion/ACL changes: replay immutable, ordered CDC/outbox or source change events including tombstones and ACL modifications before permitting the restored index to serve. If replay is impossible, use a central current authorization/deletion gate and quarantine questionable records; do not treat the old backup as fully safe. Database not immediately recoverable: activate an approved degraded retrieval mode: keyword/BM25 search over a current authorized index, direct authoritative document or API lookup, or restricted cached answers only when tenant, version, ACL and deletion eligibility can be revalidated. Otherwise abstain. 3. Engineer replica failover first when safe Validate coverage of every serving shard , replica health, replication offset/lag, search readiness and index consistency; compare deterministic canary IDs under tenant/ACL filters. Use a supported load balancer/DB routing mechanism to exclude unhealthy nodes; preserve stable collection/alias names and consistent client endpoints. Avoid split brain writes and force promoting stale replicas without reviewing quorum guarantees. Resume reads under bounded concurrency, then reintroduce writes according to the vendor’s tested failover process. Reconcile any writes accepted during partial outage using idempotent event IDs and version fences. Measure success with per shard availability, search error rate, p95/p99, retrieval quality and absence of revoked records—not health checks alone. Vendor notes: replication and availability guarantees differ by product and version. Elasticsearch may allocate regular replica shards after primary failure; Qdrant distributes collection shards and replicas. Verify whether quorum, shard coverage and read consistency meet your actual requirements. 4. Restore from a verified backup if replica recovery fails Bring up an isolated target collection/cluster; never blindly overwrite a partially healthy primary. Restore all required shards, payloads, schema/config, index settings and metadata , then reconstruct serving aliases or routing. Qdrant’s collection snapshots do not include aliases; in distributed installations snapshots are node specific, so verify complete shard coverage. Check restore priority and version compatibility before applying snapshots to a live cluster. Weaviate supports live backups with staged restore states; verify restore completion and post restore replication/visibility. Elasticsearch snapshots may also include relevant cluster state and feature configuration; inventory these separately from application level RAG metadata. Replay writes, deletes, versions and ACL changes after the snapshot checkpoint in order, with idempotency and generation fencing. Confirm manifests, active document counts, search visible IDs and revoked/deleted documents. Canary dense/hybrid queries, citations, cross tenant isolation and freshness. Atomically switch the serving alias/application pointer only after documented gates pass. Keep source version and snapshot/checkpoint IDs in the audit trail. 5. Degraded retrieval without hallucinating recovery Keep search UI/API up with a feature flag for approved fallback. Use existing authorized keyword search or live document source APIs; label that semantic ranking is temporarily unavailable. Apply the same tenant, ACL, effective version and deletion filters as the vector path. Do not use old result caches unless security and version eligibility can be checked at serve time. Separate failure classes: if keyword results are incomplete but safe, answer with citations and an explicit limited search notice. If the user requests a high stakes current policy and freshness cannot be established, decline and route to human support. Track partial coverage and empty result rates, not just HTTP success. Run fallback specific relevance tests and log which retrieval mode produced each answer. Deactivate fallback only after canaries confirm the recovered vector index meets agreed freshness, recall and SLOs. 6. Numerical incident and RPO/RTO example At 14:00 , the vector DB fails. At 14:03 , traffic is safely switched to a pre tested BM25 fallback: degraded service recovery time 3 minutes , but full semantic search RTO has not been met . Healthy replica inspection fails because one of four shard groups has no sufficiently current safe copy. Latest verified full snapshot is 12:00 , with 8,000,000 active vectors at that checkpoint. A durable CDC stream records 120 updates per minute between 12:00 and 14:00 : 120 × 120 = 14,400 events to inspect/replay, including 300 deletes/ACL revocations . Some updates replace existing vectors, so 14,400 events is not a claim that 14,400 extra vectors should exist. At 14:10 , isolated snapshot restoration starts. Suppose restoring and validating base shards takes 30 minutes , ending 14:40 . Replay then runs at 600 events/minute while new changes arrive at 120/minute : net drain = 480/minute . Initial 14,400 / 480 = 30 minutes to catch up in this simplified model, ending around 15:10 , assuming the stream contains all changes and there are no replays/errors. At 15:15 , after index/ACL/deletion validation and a guarded cutover, full search resumes. Full service RTO = 75 minutes from 14:00, while degraded retrieval was available after 3 minutes. If the business full service RTO target is 60 minutes , the incident misses it by 15 minutes . The backup alone represents a 120 minute recovery point gap at the time of failure. If the CDC ledger is complete and replayed through a committed cutover checkpoint, effective data loss can be zero for changes durably captured in that ledger ; if not, the actual RPO is determined by the last fully recovered change, not backup timestamp alone. Success gates: all four shards healthy, source to index manifest reconciled, all 300 sensitive deletes/revocations verified, 1,000/1,000 representative ACL canaries pass, no known revoked document appears, and retrieval quality/latency metrics meet agreed thresholds. Figures are illustrative. 7. Trade offs and failure modes Replica failover: fastest when a complete healthy copy exists; uses extra hardware and can serve stale data if replication/read consistency is misconfigured. A replica is not an independent backup against accidental deletion or logical corruption. Backup restore: can recover from lost nodes/regions or logical corruption, but incurs RTO, storage/egress costs, compatibility checks and replay burden. A backup must be routinely tested, not merely created. Degraded keyword/direct retrieval: preserves some functionality with fewer vector dependencies, but usually reduces semantic recall and may change latency. Explicitly disclose its coverage and avoid invented answers. Serving stale cache: potentially fast but risky after source deletions, permission changes or pricing/policy updates; use only if current authorization and eligibility can be validated. Tighter RPO: requires more frequent snapshots or durable change streams; increases storage and replication cost. Tighter RTO: requires warm capacity, rehearsed failover and ready fallback dependencies. Partial shard coverage: results may look plausible but omit documents; fail closed or clearly label restricted scope rather than silently reporting complete search. Multi region replication: helps regional outages but demands careful split brain, consistency, tenancy and data residency design. 8. Prevent recurrence Define separate full search RTO , degraded mode RTO , RPO , freshness and retrieval quality targets with business owners. Know which application functions truly require semantic search. Ensure shard/replica placement across failure domains, durable offsite backups, restore key access and a replayable version/deletion/ACL event log. Regularly test full restore into an isolated environment, shard/alias reconstruction, compatibility upgrades, and safe replay of mutations since snapshot. Run chaos drills: single replica down, entire shard missing, zone outage, snapshot corruption, expired log retention, replay duplicates, revoked ACL immediately before failover and keyword fallback outages. Monitor per shard readiness, replica lag, backup age and success, restore duration, CDC backlog, canary deletion visibility, error budgets and fallbacks; alert before the disaster consumes the recovery window. Maintain an operator runbook with decision authority, tested commands, contacts, cutover/rollback gates, user communication and post incident audit evidence. Interview takeaway: Restore search through a decision tree : use current healthy replicas when safe; otherwise restore all shards from a verified backup and replay post snapshot changes; while recovering, serve explicitly degraded authorized keyword/direct retrieval or abstain. Measure RTO, RPO, recall and security separately. Further reading Qdrant: Snapshots and restore — https://qdrant.tech/documentation/snapshots/ Weaviate: Backups and restore — https://docs.weaviate.io/deploy/configuration/backups Elasticsearch: Snapshot and restore — https://www.elastic.co/docs/deploy manage/tools/snapshot and restore Elasticsearch: Searchable snapshots — https://www.elastic.co/docs/deploy manage/tools/snapshot and restore/searchable snapshots
After deploying a new embedding model, retrieval quality drops despite successful reindexing. How would you troubleshoot vector compatibility?
Production incident (illustrative): A RAG support assistant upgrades to a new embedding model and reports that all 2,000,000 document chunks were successfully reindexed . Nevertheless, Recall@10 falls from 92% to 61% and the assistant retrieves irrelevant troubleshooting procedures. The upsert dashboard is green. Reindex completion proves that vectors were written; it does not prove that queries and documents use a compatible vector space or that the new retrieval pipeline is correct. Core rule: A vector's dimension is a schema constraint , not a guarantee of semantic compatibility. Two unrelated 768 dimensional embedding models can produce numbers of the same length in entirely different coordinate systems. Query embeddings must be produced using the matching model family/version and intended query role for the indexed document embeddings. 1. Contain the incident and establish a comparable baseline Stop broad rollout; route a controlled cohort back to the last verified old model index + old query encoder + old preprocessing , not merely to the old collection while continuing to send new model query vectors. Capture the embedding provider, exact model/revision, dimensions, distance metric, normalization, tokenization, query/document input roles, text templates, truncation, quantization, chunking pipeline version, index/alias, tenant and ACL filters for both ingestion and online queries . Collect failing query traces: the user question, transformed query text, query encoder identifier, vector dimension and norm, target index/vector name, retrieved IDs, scores, filters, rerank scores and citations. Redact private content in telemetry. Re run identical labelled queries through old and new complete pipelines on a consistent eligible corpus, with fixed top k, filters and snapshot. Separate end to end relevance regression from ANN approximation, missing vectors or document version differences. Compare retrieved source versions and ACLs; a migration must not reactivate deleted, expired or unauthorized records even if the old index appears to return better results. 2. Root cause debugging: validate the vector contract Model mismatch: query service still calls model A while documents use model B; check endpoint aliases, environment variables, background worker image, cache keys and fallback encoders. Matching dimensions can hide this bug because the database accepts the query. Role mismatch: some embedding providers use asymmetric input modes or query/document prefixes. For example, Cohere documents input type="search query" for queries and input type="search document" for indexed documents. Swapping these or using the document role for both can reduce ranking quality even with one model ID. Normalization/metric mismatch: compare cosine, inner product and Euclidean configurations and the embedding model's intended behavior. Cosine and dot product can rank differently for non unit vectors; inspect observed vector norms, zero vectors, NaNs, clipping and transforms rather than assuming every provider produces unit length output. Preprocessing mismatch: old index may embed titles + body + metadata while new index embeds only body; query side prefixes, language selection, whitespace processing, OCR, chunk boundaries, truncation at the provider limit and document templates may have changed. Wrong vector target/index: verify collection alias, named vector target, namespace, tenant, embedding output dimension setting and index version. Weaviate supports independently configured named vector spaces; queries must select the intended target vector. Partial or stale content: reconcile all active IDs/hashes/model version metadata with new index and check per shard/index readiness. Successful write counters can mask failed individual items, overwritten IDs or unexpectedly reused cached old model vectors. Ranking and filters: compare dense only retrieval before hybrid fusion, reranking, metadata eligibility and LLM generation. Changing fusion weights or a reranker during migration can produce apparent "embedding" regression. ANN quality: compare exact brute force nearest neighbors versus approximate HNSW results for the same new model vectors and filters; if exact quality is good but ANN diverges, investigate ANN build/search settings rather than embeddings. 3. Reproduce with a small compatibility test matrix A query → A documents (old known good pipeline) is the baseline. B query → B documents (new intended pipeline) is the correct candidate. A query → B documents and B query → A documents are negative controls , not meaningful cross model evaluations unless the models were explicitly trained to share a compatible space. Hold the corpus, filters and labels constant, and separately test the provider's correct search query versus search document input modes, preprocessing templates and distance metrics. Never compare raw similarity scores across independent vector spaces as though they share one calibrated scale. Use a golden vector fingerprint : for fixed non sensitive strings, record model ID/revision, input role, dimensions, norm range and reproducible vector checksum/tolerance. Detect unintended provider/model changes before production rollout. 4. Numerical incident and engineering diagnosis The old index contains 2,000,000 vectors of 768 dimensions , embedded with model A. The newly deployed index contains 2,000,000 vectors of 768 dimensions , embedded with model B. Equal dimensions mean the client/database raise no dimension error , even though the spaces are incompatible. A labelled set of 1,000 questions has a known relevant document. Old production Recall@10 finds that document in 920/1,000 = 92% cases. New production finds it in 610/1,000 = 61% : a 31 percentage point drop , or 33.7% relative decline . A trace review finds the online API is still producing model A query vectors but searching the model B document index . This is the primary incident root cause: wrong query/index pairing, not missing document writes. After routing queries through model B's correct query encoder and search query input role , the same evaluation improves to 900/1,000 = 90% . The remaining 2 percentage point gap from the old baseline still warrants labelled analysis; matching models does not guarantee a model upgrade improves your domain. An illustrative second check shows 100 documents missing their title prefix after re embedding. Restoring the agreed document template and re embedding only affected versioned chunks raises the measured set to 925/1,000 = 92.5% . This is a hypothetical result, not a guarantee. Confirm no leak/overlap between training, tuning and evaluation sets. Sanity check for metric differences: for vectors q=(1,0), d1=(2,0), d2=(0.8,0.6), dot products are 2 and 0.8; cosine similarities are 1.0 and 0.8. Changing from dot product to cosine can change cross document ranking when norms differ. This example illustrates why the original metric and normalization contract matter. 5. Engineering solution and safe rollout Define an immutable embedding contract per index: model provider, model id, revision, dimension, distance metric, normalization, input type, preprocessing version, chunk template version, quantization, vector space id, and source manifest. Check compatibility at ingestion and query boundaries. Enforce query vector space id == target index vector space id before search; reject or route incompatible queries rather than hoping that equal dimensions are adequate. Use version tagged embedding caches and separate model specific named vectors or collections. Pin matching query/document model versions; use documented asymmetric query and document modes. Recreate vectors if the embedding configuration or input template changed; do not reuse old vectors solely because the dimensions match. Build green separately from the last verified blue index, validate active IDs and ACLs, and run dense only + filtered exact vs ANN tests and labelled Recall@k/nDCG/answer citation evaluation. Compare performance and cost as well as relevance. Run shadow traffic and canaries, then switch both query encoder routing and index alias as one coordinated release/configuration version. A vector db alias switch alone does not atomically update another service's model endpoint; implement a generation fenced routing manifest. If gates fail, roll back the whole retrieval bundle (encoder, preprocessing, vector space target, metric and index). Retain current deletion/ACL gates and catch up mutations on blue to avoid exposing revoked content. Reconcile any writes accepted during canary. 6. Trade offs and failure modes Parallel indexes/named vectors allow safe comparisons and rollback but temporarily duplicate vector storage and index memory. Some vendors restrict removal of named vectors; plan long term cleanup. Fresh model may improve benchmark results but be worse for the domain, language, document length or rare technical terms. Use representative human labelled questions rather than generic similarity demos. Normalization and metric changes can affect ranking or index efficiency; rebuilding the database with a different metric is not necessarily required when both metrics are mathematically equivalent for consistently unit normalized vectors, but verify the engine semantics. Caching old query embeddings is cheap but dangerous when model IDs change; include model revision, input role and preprocessing parameters in cache keys. Hybrid/BM25 fallback can preserve exact code and product name retrieval while the dense migration is evaluated, but don't use it to conceal a major dense search regression. Migration CPU, cost and latency rise with double embedding and dual querying; measure p95/p99 and provider token costs while maintaining freshness and version/ACL safety. Score thresholds learned under model A may not transfer to model B; re calibrate retrieval cutoffs and hybrid fusion with labelled data rather than copying score numbers. 7. Prevent recurrence Maintain a versioned embedding compatibility matrix and reject cross space query routing, including accidental model fallback, provider alias changes and cached vectors from older versions. Add deploy time fingerprint tests for fixed sentences, dimensions, input roles, vector norms, distance metric config and query/document embedding version parity. Require semantic golden query benchmarks and regression thresholds for Recall@k, nDCG, MRR, citation support, filter correctness and latency before promotion. Audit ingested active point counts and vector space version per shard; trace online requests to their encoder and target vector space. Alert on wrong space attempts and missing model version metadata. Run migration drills with stale caches, mixed model dual writes, delayed events, rolling API deployments and simultaneous document ACL revocations. Test whole bundle rollback without reactivating unauthorized content. Review model release notes and re test after provider updates, quantization changes or preprocessing refactors; do not assume stable model names imply identical embeddings forever. Interview takeaway: A successful reindex says the writes landed, not that the query and documents share a compatible embedding space . First verify encoder/version/role, metric, preprocessing, index routing and exact vs ANN results; then repair the mismatched pipeline and deploy the query encoder + vector index as a unit . References Weaviate, Switching vectorizers: https://docs.weaviate.io/weaviate/tutorials/vectorizer migration Weaviate, Named vectors: https://docs.weaviate.io/weaviate/manage objects/create Cohere, Embeddings and input types: https://docs.cohere.com/docs/embeddings Weaviate, Search target vectors: https://docs.weaviate.io/weaviate/concepts/search
A vector database upgrade changes approximate-search results for the same queries. How would you validate and safely roll out the upgrade?
Production incident (illustrative): A customer support RAG system upgrades its vector database to obtain performance and security fixes. After the upgrade, the same query embeddings against an apparently unchanged 5 million vector collection return different top 10 neighbors. API error rates are normal, but a regression benchmark shows Recall@10 falling from 96% to 89% , and certain policy citations disappear. Engineers suspect corrupted embeddings and propose undoing the upgrade immediately. Yet approximate nearest neighbor (ANN) results can legitimately vary with graph layout, index parameters, search implementation, tie breaking and replica/index state. The task is to decide whether the change harms retrieval correctness , identify why, and roll out safely. Principle: A vector database upgrade is a retrieval behavior change , not merely a server version change. Identical approximate top k IDs and floating point scores across releases are usually not the right acceptance criterion. Compare both releases against a reproducible exact search baseline and representative task labels, while enforcing security and availability invariants. 1. Contain and reproduce Pause further rollout; preserve a tested, currently serving release path. If an upgraded node already returns unsafe or materially wrong results, remove it from production traffic using a supported routing or replica mechanism after confirming coverage, freshness and ACL safety. Capture old/new server versions, patches, SDK versions, container images, index build generation, shard/replica placement, active collection alias, data snapshot hashes, embedding model/version, vector dimensions and distance metric. Record unchanged request vectors and exact query parameters: top k, HNSW ef search, payload filters, tenant/ACL predicates, quantization, rescoring, hybrid fusion and read consistency. Re run a fixed labelled query set and compare old/new results on the same eligible corpus snapshot , tenant permissions and query embeddings. Separate changed ANN neighbor order from actual missing relevant evidence, incorrect filters, delayed indexing or inconsistent replica reads. Do not re embed documents as the first response. If the query/document embedding space did not change, investigate the vector database and index/search path first. 2. Root cause debugging: locate the behavioral difference Exact versus ANN: compute brute force/top k reference neighbors with the same vector distance, filters and source snapshot. If both old and new approximate results differ but stay near the exact reference, variation may be expected. If new ANN recall degrades, inspect index/search configuration and implementation. Index construction: check whether the upgrade rebuilt HNSW/IVF graphs, changed build defaults (m, construction effort, optimizer thresholds), compacted segments, rewrote payload indexes, changed quantization/rescoring or adopted new index storage. Different graph insertion orders can produce different approximate neighbors without changing vector data. Search configuration: inspect runtime defaults for candidate budget, ef search, exact search threshold, quantization, score thresholds, timeout/cancellation, filter planner and hybrid search. Pin parameters explicitly rather than relying on release specific defaults. Corpus/index mismatch: reconcile live point IDs, hashes, vector dimensions, document versions, tombstones and ACL payloads per shard. Verify that no source changes or background updates occurred in only one environment. A new replica may lag in writes or index readiness. Distance/numerical effects: check cosine versus dot product versus Euclidean and normalization, zero/NaN vectors, SIMD/CPU/GPU implementation changes and ties. Minor score differences need not be wrong; changed top k relevance and filtering correctness matter. Filter and access semantics: compare exact and ANN results under the identical tenant, ACL and effective version filters. A planner change that drops, misapplies or widens mandatory filters is a security blocker , even if latency improves. Client/router effects: review SDK requests and compatibility, named vector targets, collection aliases, retries and replica consistency. Verify the supposedly identical query is not being routed through different code paths. Release notes: read every intermediate minor/patch migration note for deprecations, behavior changes, incompatible storage formats and client/server constraints. In Qdrant, supported upgrade paths require stepping through intermediate minor versions; don't assume an arbitrary version skip is safe. 3. Build a trustworthy validation harness Freeze a labelled production like query set across frequent, rare, highly selective, multilingual, long tail and filter heavy queries. Include exact identifiers, citations, sensitive deleted/ACL revoked documents and multiple tenant scopes. Store query id, immutable query vector, allowed corpus snapshot, expected relevant document/chunk IDs, relevant metadata predicates and index/query configurations. Run on two isolated but equivalently provisioned environments. Compute exact nearest neighbors using a flat scan on the same vectors/filters, then ANN Recall@k = overlap with exact top k divided by k; also measure labelled Recall@k, nDCG@k, MRR, citation support and answer correctness . Exact neighbor recall is not the same as human relevance. Measure p50/p95/p99, throughput, CPU, memory, disk I/O, ingestion lag, query timeout and resource cost under both read only and simultaneous read/write loads. Test shard/replica consistency, missing or duplicate points, unexpected score thresholds, filtered retrieval, high churn, quantized index behavior and cold/warm caches. Fix traffic mix and concurrency so the comparison is meaningful. Choose documented release gates before running the test: for example ANN Recall@10 ≥ 0.95 , labelled Recall@10 ≥ baseline minus 1 percentage point, zero unauthorized result violations, p99 ≤ 250 ms and no material indexing/freshness regression. These thresholds are illustrative requirements, not universal defaults. 4. Numerical incident: separate harmless differences from regression Benchmark 1,000 test queries , each with 10 exact eligible nearest neighbors, giving 10,000 reference neighbor occurrences . Old version returns 9,600/10,000 = 96% ANN Recall@10 . New version returns 8,900/10,000 = 89% : a 7 percentage point drop and 7.29% relative decline from the old ANN recall. Although 20% of queries have at least one changed top 10 result, that alone is not proof of damage. The exact reference comparison, however, confirms a meaningful quality drop. Separately, labelled Recall@10 falls from 94% to 88% , strengthening the regression diagnosis. A controlled comparison shows the old version used an effective HNSW search candidate budget of 128 , while the new deployment unintentionally uses 64 after a configuration/default change. Pinning the budget to 128 on the new release raises ANN Recall@10 to 95.5% ( 9,550/10,000 ) but raises p99 from 170 ms to 215 ms . If the release gate is ANN recall ≥95%, p99 ≤250 ms , this controlled configuration passes those two illustrative gates. The team must still test labelled relevance, all tenant filters, error rates, ingestion freshness and security before approval. With 5,000 search requests per minute and a 1% canary , only 50 requests/minute reach the candidate release initially; 4,950/minute remain on the stable release. Do not infer significance from a tiny canary: use paired offline tests first and sufficient live sample sizes. No production metrics in this story are real. These are worked examples to show why the correct action is to investigate a specific search parameter rather than blindly rebuild embeddings or demand bit for bit identical rankings. 5. Safe rollout and rollback engineering Preflight: inventory server/SDK versions and compatibility, read all intervening release notes, record immutable manifests, take verified backups , test restore in an isolated environment, and document a vendor supported upgrade path. Reserve CPU/RAM/disk headroom and check replication/shard coverage. Shadow testing: restore or reproduce the active corpus in a green environment on the target version, run old/new paired retrieval and complete security/quality tests. Keep active deletions and ACL events synchronized. Canary: route a small, auditable percentage of searches to green (or an upgrade compatible isolated replica); expand gradually only when recall, labels, filters, p99, errors and freshness pass thresholds. Rolling upgrade: use this only if the vendor guarantees mixed version compatibility and healthy replica coverage. For Qdrant, a replication factor of at least 2 is part of its documented zero downtime rolling update guidance; one replica deployments need a downtime plan. Weaviate recommends complete backups and progressing one minor release at a time using current patch versions. Provider and version specifics override a generic recipe. Coordinated configuration: pin explicit search parameters, distance metric, vector name, filter rules and SDK settings. Log a deployment generation so query traces reveal exactly which version and configuration produced each result. Cutover: after a full canary and data/ACL reconciliation, switch the read router or a supported atomic collection alias to green. Maintain version aware caches and ensure in flight requests are not mixed with incompatible indexes. Rollback: plan for traffic rollback to a retained old version cluster or collection , not an unsupported in place binary downgrade. Qdrant warns that upgraded on disk formats may make downgrades irreversible. Reconcile every post checkpoint insert/update/delete and current ACL change to avoid resurrecting deleted records when falling back. Hard stop conditions: any cross tenant/ACL violation, lost active vectors, missing critical citations, material relevance regression, unacceptable p99/errors or unbounded replication/ingestion lag blocks promotion. 6. Trade offs and tricky cases Matching exact top k: useful for controlled debugging but too strict as a production acceptance rule for ANN algorithms; approximate indices may differ legitimately. Higher search effort: can restore recall but increases CPU, memory and p99. Tune under realistic concurrent traffic, not a small laboratory sample. Rolling upgrades: reduce extra capacity but require confirmed mixed version compatibility and enough healthy replicas; temporary version mixtures can complicate investigation. Blue/green: offers clean A/B and traffic rollback at the cost of duplicate storage, snapshot/replay work and temporary infrastructure. Backup only rollback: may lose post snapshot writes/deletes and cause unsafe ACL drift; a durable change log and serving level tombstone gate are important. Small canaries: limit blast radius but can miss rare query classes and tenant specific regressions; pair them with offline quality gates. Score changes: scores need not match exactly across implementations; thresholds and hybrid fusion weights may require careful recalibration with labelled evidence. Mixed precision and quantization: output order may change with rescoring or precision, yet production relevance can be unchanged or worse; validate, do not assume either direction. 7. Prevent recurrence Maintain a reproducible upgrade validation suite with immutable vectors, source manifest snapshots, ACL/filter canaries and labelled questions; run against each candidate server/SDK/configuration release. Alert on ANN Recall@k versus exact baseline , labelled relevance, critical citation coverage, tenant/ACL violations, p99/errors, replica health and source to search freshness. Segment by tenant and query type. Pin search parameters instead of relying on undocumented defaults; record a deployment/embedding/index generation on every retrieval trace. Regularly rehearse snapshot restore, CDC catch up, traffic rollback and cutover under simultaneous writes, deletes, ACL revocations and shard failures. Track version specific migration requirements, release note changes, index re optimization and SDK compatibility in a preflight checklist. Run post upgrade audits before retiring the previous verified environment. Interview takeaway: Changing ANN results after an upgrade is evidence to investigate, not automatically an error . Compare both versions to exact neighbor and labelled relevance baselines on the same secure corpus, isolate configuration/index differences, then use shadow, canary and vendor compatible rolling or blue/green deployment—with a tested traffic rollback path. References Qdrant, Upgrading Qdrant: https://qdrant.tech/documentation/upgrades/ Qdrant, Compatibility and downgrades: https://qdrant.tech/documentation/faq/qdrant fundamentals/ Weaviate, Migration and upgrades: https://docs.weaviate.io/deploy/migration Weaviate, Deployment best practices: https://docs.weaviate.io/weaviate/best practices
Your vector index grows rapidly even though the number of active documents remains constant. How would you investigate storage amplification and deleted-vector accumulation?
Production incident (illustrative): A RAG knowledge base collection maintains 2 million active document chunks , yet its vector database expands from 18 GiB to 45 GiB during a month of nightly document replacements. Ingestion reports success, search generally works, and the active chunk count is nearly unchanged. The operations team considers adding storage, but a hot node approaches its disk watermark and p99 spikes during background compaction. Root cause hypothesis: old vector versions have been logically deleted but not physically reclaimed, and optimizer work falls behind repeated upserts. Prove that hypothesis before running destructive cleanup. 1. Understand what is growing Logical live data: active, authorized point IDs and their document versions. A stable active count does not imply stable physical bytes. Deleted/tombstoned data: deleted points and overwritten vector versions can remain in segment files until safe background cleanup/merge. A query correctly excludes them even while storage still holds bytes. Write amplification: new segment files, WAL, optimizer copy on write, merge inputs/outputs and replica updates temporarily or persistently consume extra bytes. Independent storage contributors: raw vectors, HNSW graph, payload/filter indexes, text payloads, quantized copies, sparse vectors, WAL, snapshots, old collections, shard replicas and backups. Never blame tombstones from total disk growth alone. Replication and cloud billing: total cluster bytes may include copies of each shard, snapshots, backup/object store versions or temporary transfer replicas; compare equivalent scope (per primary shard vs all replica nodes). Storage amplification ratio: define actual physical index bytes / estimated ideal live index bytes for the same node/shard set and feature configuration . Include the scope in dashboards; a ratio over 1 does not automatically mean waste. 2. Contain the incident before touching storage Confirm immediate disk risk using free bytes/inodes, growth rate, read/write I/O latency, disk watermarks, optimizer backlog, allocation failures and read/search error rates. Preserve latest source and index manifests, snapshot metadata, recent deployment/config changes, active version ledger, retention/legal hold policies and query latency baselines. If safe, cap redundant upserts, rate limit ingestion or optimization concurrency, isolate a busy tenant/collection and reserve temporary disk for compaction. Do not run forced merges on a nearly full disk without estimating peak working space. Never permanently delete current records, backups required for recovery, or protected audit evidence just to relieve pressure. Verify tombstone serving blocks and current ACL filters independently. If there is insufficient space even for controlled cleanup, expand disk temporarily or migrate to a properly sized healthy environment; this is capacity protection, not a root cause remedy. 3. Trace the real byte growth Compare day by day live point count, total physical records if exposed, deleted point ratio, segments, indexed vectors, storage bytes by segment, replica factor and snapshot/WAL bytes . Inspect per shard differences instead of relying on collection wide averages. Break down disk by directory or provider reported categories: vectors, hnsw/index, payload, WAL, optimizer temporary, snapshots, replicas, old aliases/collections. Avoid scanning large files destructively during an incident. Reconstruct input churn: inserts, updates, deletes and repeated upserts of the same external point ID . Qdrant documents that an upsert of an unchanged point can still mark the prior representation deleted and insert a new one; avoid sending unnecessary replacement writes. Compare optimizer/vacuum candidate threshold , segment count, active merge/vacuum threads, last successful cleanup, queue age, CPU/I/O saturation, available scratch space and repeated optimizer failures. Inspect record ID policy and application lineage. Are nightly jobs creating new IDs for unchanged chunks or keeping obsolete document versions in a supposedly active collection? If both, there is also logical data duplication not solved by tombstone vacuum. Compare vector dimensions, dtype, quantization mode, payload index additions, sparse/hybrid indexes, changed chunk sizes and replication changes; any of these can enlarge the valid footprint without adding documents. Check source deletes against serving visibility as a separate security requirement. Physical disk cleanup is usually asynchronous and must not be confused with timely authorization/deletion enforcement. 4. Root cause debugging decision tree Deleted ratio rising, vacuum backlog increasing: suspect reclamation lag from churn or constrained optimizer resources. Examine per segment eligibility and failed optimization. Deleted ratio low, live bytes high: suspect changed vector dimension, duplicate new IDs, payload growth, new named/sparse vectors, extra indexes, replication or old serving generations. Disk spikes during optimizer and then shrinks: temporary segment rewrite/copy on write is plausible; quantify peak headroom and tune concurrent compactions. WAL/snapshot bytes grow while segment bytes are stable: inspect snapshot lifecycle, retention, WAL replay/checkpoint status and abandoned backups; do not delete files outside supported management APIs. Only one shard/node grows: inspect tenant skew, shard placement, replica transfers, write distribution and a stuck local optimizer. Queries return deleted records: an independent visibility/deletion bug exists; remediate serving filters immediately, not only physical reclamation. Search slows after aggressive vacuum: optimizer I/O contention can raise p99. Optimize for both stable disk consumption and query SLOs. 5. Beginner friendly numerical example Collection has 2,000,000 active points . Each stores a 768 dimensional float32 vector : 768 × 4 = 3,072 bytes raw, so live raw vectors use 2,000,000 × 3,072 = 6,144,000,000 bytes ≈ 5.72 GiB. This excludes graph edges, payload, metadata, replicas and snapshots. Assume 1,000,000 upserts every night overwrite existing IDs and the older representations are marked deleted until vacuum. Over 10 days , that is 10,000,000 historical write events ; do not assume all produce ten million still retained tombstones, because maintenance may reclaim some along the way. At incident time, suppose per shard inventory aggregated across the collection shows 2 million live and 3 million still physically represented but logically deleted point versions. The tombstone share is 3/(2+3) = 60% of represented versions. The 3 million stale raw vectors alone would occupy 3,000,000 × 3,072 / 2^30 ≈ 8.58 GiB before graph/payload/segment overhead, assuming each retains a raw representation. Baseline total index storage was 18 GiB ; measured storage is 45 GiB , so physical growth = 27 GiB, 150% increase , and 45/18 = 2.5× baseline bytes. This 2.5× is a baseline growth multiple, not a rigorously normalized amplification ratio unless the original 18 GiB is independently proven to be the ideal current live footprint. Suppose measured breakdown explains the 27 GiB increase as 8.6 GiB retained stale vector bytes + 6.4 GiB associated obsolete indexes/payload + 7.0 GiB temporary merge/copy on write files + 5.0 GiB additional snapshot/WAL files = 27.0 GiB . These are illustrative attributed components on an equivalent disk scope, and should not be double counted in real filesystem statistics. After disabling identical no op upserts, letting supported vacuum/merge processes catch up, and cleaning verified expired snapshots under retention policy, suppose storage stabilizes at 21 GiB , a 24 GiB (53.3%) drop from 45 GiB . Search p99 must be monitored while cleanup runs; bytes reclaimed is not success if search or ingestion SLAs fail. 6. Engineering solution Stop avoidable churn: use stable tenant/document/version/chunk identifiers, hashes or source revision checks; skip unchanged data instead of blindly upserting. Reconcile live manifests to prevent accumulating active but obsolete versions. Schedule safe reclamation: tune the vendor's deleted ratio threshold and vacuum/compaction workers against disk headroom, shard size and read/write workload. Monitor that each cleanup completes and physical bytes decline. Budget scratch capacity: segment rewrites can keep old data readable while new files are built. Estimate temporary disk amplification from representative compactions; do not assume vacuum immediately frees space. Protect search during maintenance: bound optimizer concurrency, pace work outside peak hours where appropriate, prioritize safety critical deletes, and benchmark p50/p95/p99, CPU and disk I/O before/after. Qdrant guidance describes the storage versus latency trade off of different vacuum thresholds. Maintain snapshots and WAL responsibly: use vendor supported snapshot/retention and checkpoint APIs, verify recoverability before expiry, and treat offline backups as distinct from serving index bytes. Validate after cleanup: reconcile live point IDs, active document versions and deletion/tombstone authorization; test Recall@k, citation correctness, replica coverage, per shard disk footprint and ingest freshness. Consider a shadow rebuild only if maintenance cannot restore a healthy index, with full write replay and security gates. Escalate capacity only when justified: if the expected legitimate live data and indexes , replica requirements and peak maintenance overhead exceed storage budget after cleanup, resize or rebalance with evidence. 7. Vendor specific behavior and operational trade offs Qdrant: records can be soft deleted until the Vacuum Optimizer rebuilds segments; optimization uses segment level work and can keep old segments readable while new ones are written. Tune deleted threshold, vacuum min vector number and optimizer limits for the installed version. Qdrant docs note that even unchanged upserts can cause delete and reinsert behavior. Avoid generic advice to edit index files manually. Weaviate: HNSW tombstone cleanup runs periodically; cleanupIntervalSeconds and supported TOMBSTONE DELETION parameters affect cleanup batch size/concurrency. For very large indexes, cleanup itself may cause resource pressure. Check deployment version support and actual tombstone backlog. Elasticsearch/Lucene: updates/deletes create soft deleted segments that are reclaimed during merges. Elastic advises force merging only read only indexes ; forcing a live index into very large segments can worsen retained deletes and future disk usage. Lower vacuum threshold: typically reclaims deleted bytes sooner but increases background work and search contention. A higher threshold can preserve p99 during peak writes at the cost of extra disk usage. Index compression or quantization: may lower legitimate footprint but can hurt Recall@k or incur rescoring costs. It does not fix waste from duplicate IDs, snapshots or blocked vacuum. Scale out: distributes some load but may duplicate shards/replicas and does not automatically reclaim obsolete versions; validate placement, storage and consistency. Tombstone retention versus compliance: some deletion records are essential governance evidence, but embedding/text copies must meet the system's deletion policy. Retain minimal audit metadata separately from stale searchable content. 8. Prevent recurrence Dashboard physical bytes by component and shard , live vs deleted versions, deleted fraction, optimizer last success time/backlog, disk growth velocity, snapshot/WAL retention, unique chunk ID churn and replica overhead. Alert on accelerating bytes per live vector , sustained vacuum backlog, high deleted ratio, rapidly increasing disk watermark risk and compaction failures—not merely count of active documents. Add ingestion idempotency, content hash checks and no op upsert suppression; run periodic source to index manifest reconciliation and stale version cleanup. Load test millions of realistic upserts and deletes with simultaneous reads, measuring search p99, disk peak/headroom, optimizer completion and final steady state bytes. Include optimizer crashes and restore drills. Establish disk capacity budgets for active corpus + expected tombstones + WAL/snapshots + compaction scratch + replication + safety margin , with policies for old index generations and backup retention. Review deletion audit trails and serving eligibility independently so cleanup delays never result in retrieval of revoked documents. Interview takeaway: A stable document count with exploding index bytes often reflects storage amplification from soft deletes, repeated upserts, segment rewrites or retained backups . Attribute bytes first, distinguish logical deletion from physical reclamation, suppress pointless writes, and tune safe cleanup under query SLO and free disk constraints. References Qdrant, Optimizer and Vacuum Optimizer: https://qdrant.tech/documentation/ops optimization/optimizer/ Qdrant, Storage and WAL details: https://qdrant.tech/documentation/manage data/storage/ Qdrant, Soft deletion and unchanged upserts FAQ: https://qdrant.tech/documentation/faq/ Qdrant, Predictable search latency and vacuum trade offs: https://qdrant.tech/documentation/operate and maintain/predictable search latency/ Weaviate, HNSW tombstone cleanup parameters: https://docs.weaviate.io/weaviate/config refs/indexing/vector index Elastic, Disk usage and merge guidance: https://www.elastic.co/docs/deploy manage/production guidance/optimize performance/disk usage
A user searches for an exact invoice number, but semantic search returns unrelated invoices. How would you debug the failure?
Production incident (illustrative): A finance user asks a RAG assistant, “Show invoice INV 2026 004281 .” The correct invoice is present in the document system, but dense vector search returns several invoices about the same supplier and payment terms. The assistant cites invoice INV 2026 004218 instead—one transposed pair of digits creates a material accounting error. The system has low query latency and healthy vector nodes; the failure is retrieval intent and identifier fidelity , not infrastructure availability. Key principle: Invoice numbers, purchase order IDs, serial numbers and account references behave as exact identifiers , not semantic concepts. An embedding model learns similarity of surrounding language; it does not promise exact character equality. Treat a confidently recognized ID as a structured lookup key, with mandatory authorization and version checks, before using semantic retrieval for explanatory questions. 1. Contain the incident and reproduce it Prevent the assistant from answering an exact invoice request using a different invoice merely because its semantic similarity is high. If the exact authorized record cannot be found, return “I could not find that invoice” or request clarification—never substitute a close identifier. Capture the raw user request, parsed identifier, tenant/organization, active period, exact ID string, retrieved candidates and scores, vector/lexical query, reranker outputs and the final citation. Redact invoice details in logs. Check the authoritative finance system by tenant id + invoice number (plus issuer/business unit when required). Verify whether the requested invoice actually exists, is current, and is visible to the user. Reproduce the failure at each boundary : source → OCR/extraction → chunk metadata → keyword/exact index → query parser → retrieval candidate merge → reranker → final citation. Preserve raw and normalized representations so debugging does not accidentally hide lost zeros, dashes, spaces, Unicode variants or OCR substitutions. 2. Root cause debugging decision tree ID missing from source or user lacks access: report not found/unauthorized in an appropriately non disclosing way. Do not search other tenants or weaken ACLs to return “something similar.” ID in source but missing from index: inspect OCR errors, CSV/PDF ingestion, page headers, metadata parsing, source version, refresh/replica visibility, and indexing failures. Verify the actual stored field and correct collection/alias. ID indexed but keyword/exact lookup fails: inspect mapping and analyzers. On analyzed full text fields, punctuation may be split or removed and case folded; store a dedicated keyword or normalized exact field. Elasticsearch recommends term queries on exact fields, not analyzed text. Exact lookup succeeds but vector query fails: this is expected for many identifiers; measure dense retrieval recall on an identifier specific evaluation set instead of assuming semantic search must match digits. Both retrieve the record but final top k excludes it: inspect BM25/vector score normalization, reciprocal rank fusion, weighting, per branch candidate limits, reranker inputs, threshold pruning, duplicate collapsing and query rewriting that drops the identifier. Reranker or LLM changes the number: compare exact candidate ID at each stage and require post retrieval invariant returned invoice number == requested invoice number for exact lookup requests. Never let generative reasoning replace a structured identity check. Unexpected similar invoice appears: test OCR confusables 0/O, 1/I, 5/S, transposed digits, leading zero stripping, Unicode hyphens, slash normalization, multi company ID collisions and old invoice versions. 3. Correct retrieval design Detect likely exact ID intent with a constrained parser (regex/validated schema) or a hybrid classifier, while supporting natural requests such as “What was the tax on invoice INV 2026 004281?” Preserve the literal token and avoid LLM rewriting of identifier bytes. Canonicalize only known equivalences approved by the source system: e.g., trim surrounding whitespace, normalize known Unicode dash variants and apply documented case rules. Never drop leading zeros or remove arbitrary punctuation if those characters are significant in the issuing system. Index both invoice number raw and an explicit invoice number normalized as exact match fields. Use composite keys such as tenant id + issuer id + normalized invoice number; add document version, active status, ACL principal and source lineage. Route confidently parsed identifiers to an authoritative database or indexed term/keyword query , with tenant and ACL filtering applied inside the lookup . If exactly one permitted active invoice matches, return it with the authoritative citation. For requests with both a valid ID and a conceptual condition (“show INV 2026 004281 and explain why it was delayed”), retrieve the exact invoice first , then use semantic/keyword retrieval within its authorized related context for explanation. The exact invoice ID remains a hard constraint. If no exact match is found, optionally offer clearly labelled suggestions from fuzzy lookup or trigram similarity, but never silently auto select a neighboring invoice. Fuzzy suggestions must preserve tenant/ACL constraints and may expose existence only according to policy. Use hybrid BM25 + vector search for questions without an unambiguous exact ID, or when users combine a broad topic with multiple terms. OpenSearch supports lexical/semantic hybrid search with normalization or rank based fusion; tune on an evaluation set rather than assuming a uniform score scale. Require output validation and source grounding: the invoice number in a citation, fetched record and final answer must match the requested canonical key. If the invoice is absent or ambiguous, abstain safely. 4. Numerical walkthrough: why the error happens A test set has 1,000 invoice number queries with exactly one existing, authorized active invoice each (separate from tests for missing/ambiguous IDs). Dense semantic top 5 returns the exact invoice for only 580 requests: Exact ID Hit@5 = 580/1,000 = 58% . Another 420 return invoices with similar prose, payment terms or customer descriptions. In the illustrative failure, query INV 2026 004281 produces INV 2026 004218 at vector rank 1 with similarity 0.89 , while the exact record appears at vector rank 9 with score 0.78 . A top 5 cut discards the correct record. The higher embedding score is not evidence that the identifier is equal. A dedicated exact lookup using tenant id=ACME, issuer id=SUP 17 and invoice number normalized=INV 2026 004281 retrieves the verified record in 24 ms in this example. Its identity comes from equality and source fields, not a score threshold. After integrating exact routing, suppose 990 of 1,000 test queries succeed. The remaining 10 fail because 6 invoices were stored with OCR corrupted numbers and 4 have missing canonical metadata. Exact ID Hit@1 = 990/1,000 = 99% . Repairing ingestion/metadata and retesting those 10 cases is required; do not proclaim 100% coverage prematurely. Measured improvement in this hypothetical test is 41 percentage points from 58% to 99% on the relevant success metric, noting the original was top 5 and the new is top 1. For apples to apples reporting, separately track baseline Hit@1 , post fix Hit@1 , and end to end correct citation rate rather than comparing different k values as though identical. Test realistic negative queries, such as a nonexistent number or an invoice from another tenant. The required result is zero unauthorized disclosure and no silently substituted invoice —not maximal result count. 5. Engineering implementation and validation Schema: maintain invoice number raw, invoice number normalized, tenant id, issuer id, source record id, document version, active, acl policy and content/source hashes. Store identifiers in both document metadata and a queryable exact lookup index. Search path: run detect identifier → validate/canonicalize → authorized term lookup → verify version and source → optional related document retrieval → citation equality assertion. Keep general dense/hybrid search as a separate route. Elasticsearch/OpenSearch example: map invoice numbers to a keyword exact field (with a deliberately chosen normalizer if needed) and use a term query under tenant filters. A match query on analyzed text is not a safe equality test. PostgreSQL example: use a tenant scoped indexed equality predicate on the canonical text column; if offering suggestions, pg trgm supports similarity/trigram searches but does not replace exact matching . Its trigram extraction may ignore punctuation, another reason to preserve exact raw IDs. Pipeline tests: inject invoice formats with zeros, dashes, slashes, prefixes, Unicode, OCR confusables, multiple issuers, duplicate revisions and deleted records. Check that normalization does not collapse distinct valid IDs. Candidate/rerank tests: assert exact hits survive merge, rerank and context assembly; verify any deduplication is version aware. A reranker must not “improve” relevance by selecting a neighboring invoice. Safe rollout: first shadow log parser classifications, then canary exact lookups, compare Hit@1, wrong invoice rate, authorized not found rate, p95 latency and citation correctness. Roll back routing rules if false identifier detection harms non ID queries. 6. Trade offs and common mistakes Exact only is the safest response for confirmed IDs, but it does not handle vague questions, misspellings or semantic questions without identifiers; use hybrid paths for those intents. Strict regex reduces false positives but misses new vendor formats; configurable per issuer schemas, validation and an unknown format path are safer than one global pattern. Aggressive normalization increases match rate but risks collisions between distinct IDs. Treat normalization as a domain rule, not generic lowercasing and punctuation removal. Fuzzy matching helps typo recovery but is dangerous if silently used for financial identifiers. Offer suggestions with visible differences and require user selection. BM25/hybrid fusion increases broad recall but can still demote an exact invoice when rank fusion is used blindly; exact route or hard filter identity first. Extra structured index costs storage and synchronization effort; it is justified when identifiers are business critical and correctness matters more than semantic ranking alone. Authorization: a “not found” response may need to avoid revealing whether another tenant's invoice exists. Do not debug by bypassing production access controls. Document chunks: invoice IDs buried on page 1 may be absent from chunks of page 3. Preserve parent metadata and structured keys across all versioned chunks, with provenance. 7. Prevent recurrence Create an identifier specific regression suite spanning invoice, order, ticket, customer and error code formats. Track Exact ID Hit@1 , correct citation rate, wrong identifier response rate, false identifier detection and tenant leakage. Continuously reconcile source IDs with exact index metadata and chunk lineage; alert on OCR failures, missing normalized fields, duplicate canonical keys and search visible lag. Add checks in the application that output identifiers equal validated requested identifiers . Block responses with mismatches even when the LLM appears confident. Monitor retrieval path selection and fusion/reranker dropouts, plus query form distributions, latency and authorized not found counts by issuer/format. Require an exact ID search canary before each tokenizer, embedding, hybrid ranking, OCR or schema release, using real world formats and security sensitive negative controls. Interview takeaway: If a user asks for an exact invoice number, first verify the identifier in a structured, authorized exact match index , then use semantic search only for related explanation. Debug from OCR and metadata through query parsing, candidate fusion and final citations; never let high embedding similarity override exact identity. References Elasticsearch, term query and analyzed text warning: https://www.elastic.co/docs/reference/query languages/query dsl/query dsl term query OpenSearch, hybrid search and rank/score fusion: https://docs.opensearch.org/latest/vector search/ai search/hybrid search/ OpenSearch, keyword versus vector search: https://docs.opensearch.org/latest/search plugins/ PostgreSQL, pg trgm and punctuation behavior: https://www.postgresql.org/docs/current/pgtrgm.html
The correct document exists in the index but never appears among the retrieved candidates. What debugging sequence would you follow?
Production incident (illustrative): An enterprise RAG assistant is asked, “What is the reimbursement ceiling for contractors traveling to Singapore?” The authoritative 2026 Travel Policy , paragraph 4.3, explicitly answers the question and is confirmed present in the vector collection. Yet the assistant's first stage top 10 candidates contain only older travel FAQs and unrelated expense policies. The reranker and LLM cannot recover a document they never receive. The team initially increases top k without investigating where the target disappeared. Key principle: “The document exists in the index” is not the same as “its correct version/chunk is eligible, vectorized in the searched space, visible on the serving replica, reachable by the candidate generator, and passed to downstream fusion.” Debug by finding the first stage that excludes the known good chunk , not by changing every parameter simultaneously. 1. Freeze a reproducible missing candidate case Record the raw question, tenant id, authenticated principal, policy effective date, expected document id/version/chunk id, index alias, named vector, query embedding model/revision, search distance metric, all filters, top k, ANN candidate budget, fusion and reranking settings, and precise request trace. Validate the expected answer bearing chunk , not merely the parent document. The correct document may be indexed while paragraph 4.3 is missing, truncated, merged with irrelevant context or represented by an old version. Verify this user is authorized to retrieve that exact chunk under tenant, ACL, version, and effective date rules. If it is genuinely disallowed, absence is correct; never remove security filters to make the benchmark green. Use a fixed point in time snapshot, query vector and query class so every experiment is comparable. Preserve raw candidate IDs and rank scores after each pipeline stage. Diagnostic experiments with filters removed must run only in an isolated privileged test on synthetic or otherwise permitted data, never in production. Separate first stage candidate recall from downstream reranking and answer quality: if the correct chunk never enters the first stage pool, tuning prompts or reranker models cannot solve the root problem. 2. Trace the first failure boundary in order A. Authoritative source and lineage: Does the right policy revision still exist? Trace source record → parsed text → chunk manifest → stable chunk ID → embedding hash/model → vector write receipt. Check OCR, extraction, heading loss, table parsing, chunk boundaries, supersession, tombstones, and partial batches. B. Exact point retrieval: Fetch expected chunk id directly from the production target collection and check payload, vector presence, vector name, model version, tenant, source revision and replica. A successful ID lookup proves stored existence, not ANN search inclusion . Check old aliases, wrong region/shard and stale replicas if lookup disagrees. C. Eligibility predicate: Evaluate every mandatory filter on that exact point: tenant, ACL, active version, date bounds, locale and document type. Compare field types (string versus integer), null/missing metadata, nested filter semantics and logical AND/OR composition. Maintain authorization gates; repair inconsistent metadata rather than weakening access. D. Exact similarity baseline: Within the same eligible corpus and distance metric , run exact/flat kNN where supported or brute force score a controlled filtered subset. Record the missing chunk's exact rank and score. If rank is 213, it is not an ANN failure when top k=10; the embedding/query representation simply does not place it close enough. E. Approximate ANN: If exact rank is 3 but HNSW returns no target in a candidate pool of 50, investigate ef search/hnsw ef, Elasticsearch num candidates, quantization/rescoring, graph health, index readiness, shard health and request timeout. Qdrant supports a diagnostic exact=true; Elasticsearch distinguishes exact and approximate kNN. F. Filtering interaction: Inspect whether the same authorized filter is applied during candidate traversal or after top k selection . Elasticsearch distinguishes filtered approximate kNN from post filtering, which can yield fewer than k. Qdrant's search planner can choose different strategies per segment and offers filterable HNSW / ACORN options where supported. Preserve filter semantics , but consider changing when/how eligibility is enforced within the retrieval engine. G. Multi shard aggregation: If target appears in a local shard's candidate list but not the merged list, examine shard fan out, per shard candidate budgets, global size/top k, partial responses, request deadlines, replicas and routing. H. Hybrid fusion and deduplication: If present in dense or BM25 candidates but absent after reciprocal rank fusion (RRF), weighted blending, metadata deduplication, parent grouping, MMR or result thresholds, log before and after ranks and IDs. Ensure candidate limits are sufficient before fusion; verify document id grouping does not accidentally collapse the answer bearing version. I. Reranker/context assembly: Only if the candidate survives first stage retrieval, investigate reranker truncation, cross encoder ordering and limited context budgets. That is a later failure stage , not evidence that initial retrieval failed. 3. Root cause decision tree Point ID missing or wrong vector name: index/lineage/namespace failure; replay only the missing active version and verify visibility. Point present but filter ineligible: validate business policy; if the record is permitted, repair payload/ACL/version metadata and filter mapping. Keep current permission checks server side. Point eligible, exact rank outside top k: query document semantic mismatch, noisy chunk, unsupported language, poor chunking, wrong embedding model or query transformation. Improve representation, use alternate query formulation or lexical/hybrid recall; increasing k is a measured trade off, not a guarantee. Point exact rank high but absent from ANN: approximate index, candidate budget, index optimization or quantization issue. Tune the engine on a held out benchmark while monitoring latency and Recall@k. Point survives ANN but vanishes after filters or fusion: correct post filtering, candidate pool size, rank fusion, grouping or deduplication logic. Point returned by a shard but absent globally: repair coordinator merge, shard routing, partial results handling, replica freshness or timeout behavior. Point survives all candidates but not the final answer: reranker/context/LLM issue; keep this lesson focused on the first stage visibility boundary. 4. Beginner friendly numerical incident Build a labelled set of 200 production like questions , each with one verified authorized answer bearing chunk in the same snapshot. With first stage top k=10, the baseline retrieval finds the correct chunk for 136/200 = 68% Recall@10 (for one expected chunk per query, hit rate equals this recall definition). For the Singapore reimbursement query, direct ID lookup finds travel 2026 v3 chunk 43. Exact authorized search ranks it 4 of 50 ; the current HNSW call with hnsw ef=16 and top 10 does not return it. This narrows the root cause to ANN candidate generation or an inconsistent serving snapshot—not an absent source document. A controlled same snapshot experiment increases hnsw ef from 16 to 128 (not a universally optimal setting). The chunk now appears at ANN rank 5 . Across 200 queries, illustrative Recall@10 rises from 136/200 = 68% to 174/200 = 87% . Search p99 rises from 90 ms to 165 ms . Do not accept that trade off without checking the SLO. Further inspection discovers a separate issue: 12 relevant chunks are candidates in dense search but dropped because a post filter is evaluated against an incorrectly typed effective year field. After repairing the metadata and moving to a supported pre filtered search path, suppose the final benchmark reaches 186/200 = 93% Recall@10 , with p99 175 ms . Relative gain = (93 − 68)/68 × 100 ≈ 36.8%; absolute gain = 25 percentage points . However, the corrected filter may legitimately exclude some other points; all tests must compare against the authorized relevance set. Hypothetical stage trace for this incident: source present=YES → chunk ID exists=YES → ACL eligible=YES → exact rank=4 → ANN missing → ANN tuned rank=5 → post filter rejected due to schema bug → corrected filtered result rank=5. This is illustrative , not a claim that every missing document incident has these two causes. 5. Engineering solution Establish a candidate provenance trace that records, for each test request: eligibility decisions, exact and ANN rank, per shard IDs, branch candidate sets, fusion decisions, reranker input and final citation. Avoid logging raw confidential text; use protected IDs/hashes and short retention. Add an internal known ID diagnostic tool that fetches and validates the expected point, computes its exact score/rank in a bounded authorized set, and compares to ANN results on the same snapshot. Repair the first failing stage with smallest blast radius: metadata fix, correct collection/named vector route, replica consistency, targeted reindex, tuned ANN candidate budget or corrected fusion. Do not attempt all fixes simultaneously. For Qdrant, compare exact=true versus ANN with matching filter and limit; assess payload indexes, indexed only visibility, HNSW parameters and filtered query planner. For Elasticsearch, compare num candidates and pre filtering to post filtering on the same vector and eligible corpus. For semantic misses where exact rank is low (numerically large), consider better answer bearing chunks, title/header enrichment, query rewriting without losing intent , hybrid lexical retrieval, and a measured wider candidate pool. Run tests on realistic tenant, date and document version filters; guard against exposing revoked sources in any diagnostic/fallback route. Roll out via shadow evaluation and canary traffic with rollback thresholds. Measure retrieval Recall@k, first stage candidate presence, filter eligibility, latency, exact vs ANN gap, and relevant citation rate , not only approximate similarity scores. 6. Trade offs, false fixes, and failure scenarios Increasing top k: can admit more relevant chunks but raises reranker cost/noise and does not recover a point omitted before candidate generation by an index or filter. Increasing HNSW ef or num candidates: often improves ANN recall but consumes CPU, RAM and latency; evaluate production p99 and concurrency. Exact kNN: excellent diagnostic and sometimes viable for small filtered sets, but costly over millions of vectors. Removing metadata filters: may make a missing result appear yet break tenant, legal or source version rules. Isolated authorized debugging may compare eligibility hypotheses; production must never bypass them. Hybrid recall: helps lexical matches and rare terms, but a bad fusion weight or overly small branch depth can still discard the correct candidate. Rebuilding the index: might fix a provable index health defect, but cannot fix wrong embeddings, lost chunks, wrong filters, stale aliases or a truncating reranker. Self query bias: using a document's own embedding to test discoverability verifies a lower level path; it does not prove natural language query relevance. Partial shard responses and deadlines: systems may return superficially plausible top k without full shard coverage. Treat partial results modes and completeness flags as first class evidence. 7. Prevent recurrence Track stage wise candidate survival on golden authorized queries: exists → eligible → exact top k → ANN top k → per branch candidate → fusion output → reranker input → final cited source. Run nightly exact vs ANN comparisons on fixed, versioned query/corpus snapshots and monitor Recall@10 by tenant, filter class, shard, source version, query length and document age. Add deployment checks for alias/model changes, index optimizer states, named vector selection, payload field types, pre/post filter planning, fusion weights, per shard request completeness and candidate cutoffs. Add a debugger runbook with 10 minute triage: capture trace, fetch expected ID, validate filters, exact baseline, ANN comparison, fusion trace, remediate first failure, verify security and latency. Inject failure tests for missing chunk metadata, ACL revisions, stale replicas, narrow candidate budgets, OCR gaps, filter schema mismatch, partial shard errors and truncated reranker contexts. Monitor p95/p99 and cost alongside retrieval gains; require documented acceptance thresholds, canary rollouts and reversible configuration changes. Interview takeaway: Start with a known answer bearing chunk and locate the first boundary where it disappears : source and lineage → point existence → eligibility → exact rank → ANN candidates → per shard merge → filtering/fusion → reranker. The correct fix depends entirely on that boundary. References Qdrant, Search, exact search and indexed only behavior: https://qdrant.tech/documentation/search/ Qdrant, Payload filtering: https://qdrant.tech/documentation/search/filtering/ Elasticsearch, exact and approximate kNN: https://www.elastic.co/docs/solutions/search/vector/knn Elasticsearch, filtered approximate kNN versus post filtering: https://www.elastic.co/docs/solutions/search/vector/knn/filtered knn search Weaviate, vector and named target search: https://docs.weaviate.io/weaviate/concepts/search/vector search
Retrieval works correctly for short questions but fails for long, detailed enterprise queries. How would you isolate the cause?
Production incident (illustrative): A procurement RAG assistant answers “What is our contractor travel reimbursement cap?” correctly. But it fails on: “For a Singapore based contractor traveling to Berlin in Q3 2026, compare the reimbursement cap and required VP approval under the 2026 Global Travel Policy and the regional exception, excluding employees and expired policy revisions.” Search retrieves generic reimbursement FAQs; the dated regional exception and approval clause never reach the answering model. The vector database is healthy and fast. The failure appears only when a query includes several conditions, document references and a comparison. Diagnosis principle: Long queries can fail for different reasons: the embedding input may be truncated ; one dense vector can dilute multiple intents; query rewriting may remove hard constraints; mandatory metadata filters may exclude evidence; relevant facts may be scattered across documents; and a shallow retrieval or fusion pool may discard needed passages. Do not assume every complex query should be decomposed. Isolate the earliest failing stage with a controlled, length and complexity stratified evaluation. 1. Contain the incident and capture a reproducible trace Preserve the raw user question , query after chat history compression, normalized and rewritten queries, actual embedding input, tokenizer token count, embedding model/revision and maximum accepted length, input truncation mode, and the resulting vector dimensions/norm. Avoid storing raw confidential user text in broadly accessible logs. Record the authorized tenant, ACL, geography, document versions, effective dates, and other hard constraints as separate validated fields . Never infer authorization by generating a looser semantic query. Capture all candidate sets at each boundary: keyword/BM25, dense retrieval, per subquery branches, fusion, reranking, final context assembly and cited answer. Include per stage ranking, candidate budget and query latency. Build a paired reproducer: (A) short core question; (B) same question with legitimate qualifiers; (C) same question plus verbose but irrelevant background. Use the same user permissions and corpus snapshot ; compare the first stage where required evidence disappears. If the query asks for a policy decision without sufficient current evidence, return an appropriately qualified abstention or request a missing critical constraint instead of citing a policy for another geography/personnel category. 2. Isolate length versus semantic complexity Length only effect: add harmless context while holding the same target answer fixed; if results fail only once the input exceeds a token threshold, inspect truncation, positional sensitivity, model max length, query compression code and request size limits. Multi intent effect: keep token count roughly constant but vary the number of independent subquestions, comparisons and conditions. If retrieval drops with facet count rather than tokens, the problem may be intent dilution or insufficient evidence coverage. Constraint sensitivity: move an essential identifier, country, year or “except/not” clause from the beginning to the end of the question. If the last clause vanishes, inspect input truncation or preprocessing; if it survives but isn't applied, debug structured parsing/filtering. Document distribution: establish whether one chunk contains the answer or whether the answer needs several policy documents and a join/comparison. A global top 5 list can be dominated by one topic even though each individual clause is answerable. Time/version: verify the 2026 revision and regional exception are both active for the requested date. The absence of a superseded or unauthorized chunk is correct behavior, not a retrieval regression. 3. Root cause debugging sequence A. Query preservation: compare the raw text with text delivered to the embedder and BM25 engine. Log a hash plus safe diagnostic excerpt of the critical fields, token count and truncation position; detect accidental summarization, stopword/negation deletion, long query maximums and encoder side truncation. A 1,200 token input does not fit a hypothetical 512 token limit; whether it fails or truncates depends on provider configuration. B. Eligibility: check ACL, tenant, effective date, country, role and current version filters against each expected answer bearing chunk. Distinguish domain constraints such as “contractor only” from access control policies; never relax access filters in production. C. Direct known chunk lookup: verify each expected document/version/chunk exists and is search visible. This is a sanity check, not proof that the natural language query ranks it highly . D. Exact rank versus ANN: obtain a trusted exact/flat nearest neighbor reference on the same active, filtered corpus. If the right chunk is exact rank 137, increasing HNSW ef cannot make its semantic similarity intrinsically better. If exact rank 3 but ANN omits it, investigate ANN settings/readiness rather than query decomposition. E. Branch level retrieval: run the original full question, stripped essential question variant, position swapped variant, BM25, dense retrieval and 2–4 faithful subqueries side by side. Check whether each required facet gets at least one authorized support passage. F. Candidate and fusion budgets: log per branch top k, deduplication, reciprocal rank fusion/score normalization and reranker input. One branch may exhaust the global budget, while a passage for another facet is retrieved but pruned. G. Reranking and context: if all required passages reach the candidate pool but vanish later, investigate cross encoder input truncation, long query score bias, reranker top N, answer context limits and misplaced citation/source grouping. Distinguish candidate retrieval failure from answer synthesis failure. H. Quality of query transformations: compare rewritten/decomposed requests against the original's essential constraint checklist; reject any rewrite that silently drops “contractor,” “Q3 2026,” “Berlin,” exclusion/negation, or an exact policy identifier. 4. Carefully choose how to repair retrieval First fix lost query bytes , broken query compression, unintended tokenizer truncation or incorrectly applied hard filters. Do not add an LLM decomposition layer to hide an input loss bug. For multi facet questions that genuinely require different documents, test facet aware decomposition : preserve the original full query, derive a bounded number of focused subqueries, retrieve evidence per facet, deduplicate/merge and rerank under the original constraints. Build a coverage checklist so one facet doesn't crowd out others. However, decomposition can weaken the relationships between constraints : for questions requiring one document to satisfy every condition, separate retrieval branches may independently find partial matches. Test a monolithic query retrieval + constraint aware rerank path as an alternative. Recent stage aware retrieval research reports that decomposition is not universally beneficial at first stage retrieval. Use keyword/hybrid retrieval for policy titles, exact dates, locations, document codes and phrases whose literal identity matters. Where the engine supports it, separate verified metadata constraints from semantic intent; apply both in authorized search. Improve long query embeddings only after establishing the failure mechanism: use a suitable model/context budget, meaningful query condensation that preserves all hard constraints, or query encoder prompting required by the chosen model. Validate query/document model compatibility. When evidence spans multiple documents, keep document and citation provenance for each fact. Return an uncertainty/coverage note when a required regional exception or approval policy cannot be verified; never fill a missing branch with a plausible general policy passage. 5. Worked numerical example Evaluate 200 short, single intent queries and 200 long, multi condition queries with verified active, authorized answer evidence. The short set succeeds for 184/200 = 92% ; the long set has complete evidence for 104/200 = 52% . Each long query requires three evidence facets (policy cap, regional exception, approval requirement), so 200 queries require 600 facet level evidence hits . The baseline single vector top 10 retrieves 390 of 600 required facet instances = 65% facet coverage , but only 104 of 200 = 52% questions have all three facets. These are different metrics. Instrument one failing query: raw request 780 tokens ; the embedding endpoint (in this hypothetical configuration) silently retains only the first 512 . The final 268 tokens contain “contractor,” “2026 regional exception” and an exclusion. A position swap test places these constraints first and changes the candidate set sharply, identifying truncation as one cause. After fixing input preservation and using a tested constraint aware query representation, suppose all facet question coverage rises to 142/200 = 71% , with original 200 questions held constant. Missing evidence persists for queries whose facts span separate documents. Add a controlled four branch retrieval including the original query, a two facet comparison branch, a regional exception branch and an approval branch; keep top 12 candidates per branch , deduplicate, and rerank a bounded candidate pool. Assume the test reaches 172/200 = 86% complete evidence coverage, with 548/600 ≈ 91.3% facet hits. These hypothetical values require measurement, not an assumed gain from decomposition. End to end search and rerank p95 increases from 180 ms to 320 ms and the average number of retrieval calls rises from 1 to 4 . Absolute complete evidence improvement is 86 − 52 = 34 percentage points ; relative improvement is 34/52 ≈ 65.4% . Accept the result only if the latency/cost budgets and security checks pass. An A/B test also measures the monolithic + constraint aware reranker alternative; if it meets the same coverage with fewer calls, prefer it. The point is isolating and testing a remedy , not prescribing decomposition for every long question. 6. Engineering solution and rollout Define a versioned diagnostic schema: raw query hash, token count, embedder input tokens, truncated, intent facets, hard constraints, rewrites, retrieval branches, branch top k, fusion output, reranker input, facet coverage and citation ids. Add constraint preservation tests : every rewritten or subquery plan must retain or consistently inherit validated tenant, date, role, geography, document type, negation and exact identifiers. Avoid treating LLM inferred user attributes as authoritative filters. Establish a two stage candidate pipeline: broad first stage authorized retrieval (monolithic plus carefully selected extra branches when necessary) followed by source aware, constraint aware reranking and evidence coverage validation. Include diversity/coverage controls (such as one candidate per required facet before final reranking) when evidence truly spans sources; keep grouping by authoritative document/version to prevent mixed citations. Shadow evaluate length bins, number of facets, token position permutations, multi hop cases, high selectivity filters and negative/unsatisfiable constraints. Monitor Recall@k, all facet evidence coverage , citation accuracy, p95/p99, token costs, false inclusions and unauthorized retrieval violations. Canary only the identified fix, with feature flags and rollback. A more complex retrieval path should activate selectively for validated multi facet cases, not automatically for every verbose question. 7. Trade offs and common mistakes Longer embedder context: avoids lost text but does not guarantee that a single pooled vector represents every required fact equally well. Query shortening: reduces noise and cost but can erase essential negations, dates or constraints; preserve a normalized constraint ledger and original query. Decomposition: improves breadth in some multi hop workloads but can dilute interdependent conditions, produce spurious partial matches and multiply latency. Higher top k: can increase candidate recall but costs more reranking and may still crowd out rare facets. Measure branch diversity and final all facet coverage. Hybrid and BM25: helps exact terms, while literal matching alone may miss paraphrased policy language. Use both when warranted. More metadata filters: narrow candidates but can accidentally exclude valid evidence due to missing fields or wrong types. Keep mandatory authorization invariant. Evidence aggregation: independently retrieved facts may disagree across effective dates or policies; resolve precedence before answering, and cite each authoritative version. Optimizing answer accuracy alone: an LLM can guess correctly without retrieving necessary evidence. Track retrieval stage coverage and supported citations independently. 8. Prevent recurrence Maintain a length and facet stratified golden set: short questions, verbose single intent questions, multi hop comparisons, long negated queries, exact policy codes and queries with key clauses at the end. Add deploy time probes for embedding context limits, truncation flags, rewrite constraint loss, search filters, per branch candidate pool sizes, reranker input length and citation coverage. Alert on performance gaps between short and long queries, position sensitive misses, per facet missing evidence, p99 and request cost, with tenant/security segmentation. Perform targeted fault injection: forcibly truncate the last 20% of the query, drop one subquery, invert a negation or lower rerank depth; the diagnostic traces should identify the first failed stage. Review a sample of long query production failures weekly and classify them as input loss, constraint parsing, representation, candidate coverage, fusion/rerank or synthesis . Tie each fix to a reproducible experiment. Interview takeaway: Debug long query RAG failures by separating input length from number of intents , verifying the exact embedded input and all mandatory constraints, then tracing facet level candidate survival. Use decomposition, monolithic retrieval, hybrid search or constraint aware reranking only after the first failing boundary has been demonstrated . Research references Ammann et al., Question Decomposition for Retrieval Augmented Generation , ACL SRW 2025: https://aclanthology.org/2025.acl srw.32/ Petcu et al., Query Decomposition for RAG: Balancing Exploration Exploitation , EACL 2026: https://aclanthology.org/2026.eacl long.322/ Yin et al., When Should Queries Be Decomposed? A Stage Aware Study of Query Decomposition for Multi Condition Retrieval , EMNLP 2026: https://aclanthology.org/2026.findings emnlp.1644/ Microsoft, Azure AI Search query rewriting and caveats: https://learn.microsoft.com/en us/azure/search/semantic how to query rewrite Microsoft, RAG information retrieval guidance: https://learn.microsoft.com/azure/architecture/ai ml/guide/rag/rag information retrieval
Search works well in English but performs poorly for Hindi-English mixed queries. How would you investigate?
Production incident (illustrative): A bilingual support assistant correctly answers the English query “What is the refund deadline after delivery?” but misses the same policy when a customer asks “Delivery ke baad refund kitne din mein milega?” or “डिलीवरी के बाद refund कितने दिन में मिलेगा?” . Instead of the current 14 day refund policy, retrieval surfaces a generic shipping FAQ. Overall Recall@10 looks healthy because the evaluation suite is 90% English. This is a code switching and cross script retrieval regression , not evidence that Hindi speaking users asked the wrong question. Key principle: Separate language , script , cross language alignment , lexical analyzer coverage , domain terminology , and pipeline transformations . A model claiming Hindi support does not automatically perform well on Romanized Hindi, spontaneous code switching, regional spelling, acronyms or business specific terminology. Diagnose the first retrieval boundary that fails using meaning equivalent, authorization equivalent query sets. 1. Triage the production failure Preserve a consent appropriate redacted query trace: the raw original string, Unicode code points for suspicious characters, detected languages/scripts (with confidence), text passed to the embedder, any transliteration/translation/rewrite, tokenizer counts, query model/version/input role, active vector space, BM25 analyzer, fusion and reranking outputs. Confirm the expected active policy chunk exists and is authorized for the user's tenant, geography, date and role. Check source document language(s), title, transliterated terms and OCR fidelity. Do not treat a missing or revoked policy as a multilingual model problem. Reproduce the same intent with controlled variants: English: “What is the refund deadline after delivery?”; Hindi/Devanagari: “डिलीवरी के बाद रिफंड कितने दिनों में मिलेगा?”; Hinglish/Latin: “Delivery ke baad refund kitne din mein milega?”; mixed script: “डिलीवरी के बाद refund kitne din mein milega?”. Keep corpus snapshot, user, ACLs, date/region filters, top k and search configuration fixed. Log each candidate list and score by lexical, dense, fusion and reranker stage. Changes to any of these other factors undermine language attribution. Do not translate, normalize away or rewrite a user's negative or precise constraint without checking that its meaning survives. Hindi “नहीं” and Romanized “nahi” are material; a dropped negation can reverse policy eligibility. 2. Isolate the failure boundary step by step A. Source and query preservation: inspect Unicode normalization (NFC/NFKC as appropriate), Devanagari combining marks, zero width characters, punctuation, mixed numeral forms, capitalization and the original Romanized spelling. Validate any OCR and text extraction separately from retrieval. B. Language/script routing: inspect language ID decisions on short code switched strings. A detector that labels the entire question “English” may choose an English only encoder or tokenizer, even if Hindi words dominate the intent. Test code switch aware routing or a multilingual default; do not rely on one brittle script heuristic. C. Embedding model coverage: compare exact document ID lookup, exact/flat nearest neighbor rank and ANN candidate inclusion for the same authorized corpus. If exact vector rank is already poor for Hinglish but good for English, HNSW tuning cannot fix deficient semantic alignment. D. Correct embedding roles and compatibility: verify document and query vectors use the intended model/revision/embedding dimensions and query versus document input types. An English only encoder may work for English and fail for Hindi; pin a properly benchmarked multilingual model rather than mixing incompatible vector spaces. E. Lexical analyzers: use an analyzer introspection API (for example Elasticsearch analyze) to compare emitted tokens for refund, रिफंड, refund kab, wapas, वापस. Hindi analyzers can normalize/stem Devanagari words, but do not automatically equate all Romanized spellings with Hindi terms or guarantee code switch understanding. F. Transliteration/translation: test a separate transliteration branch (Romanized Hindi to a canonical Hindi representation) or translation to English branch. Measure errors on named entities, brand and product names, negatives, numerals, monetary amounts, dates and technical English terms. Do not assume transliteration is a semantics preserving one to one mapping. G. Hybrid fusion and reranker: inspect lexical versus dense candidate depths and weights, language aware reranker behavior, score calibration and query rewrites. A relevant Hindi hit may appear at BM25 rank 8 yet vanish when English heavy dense scores dominate the merged top 5. H. Authorization and retrieval output: verify mandatory filters, active versions and citation targets stay identical across language routes. Never relax ACLs, document age requirements or tenant isolation when a language fallback is activated. 3. Root cause decision tree English works; both Hindi scripts fail in exact dense ranking: multilingual embedding/translation or underlying corpus language mismatch. Test a supported multilingual encoder plus compatible re embedding, or a faithful translation route evaluated against labels. Devanagari works; Romanized Hinglish fails: transliteration, Romanized spelling variation, tokenization, language ID, or insufficient Romanized training examples. Add Romanized query coverage and a robust controlled normalization branch. Romanized Hinglish works; Devanagari fails: inspect Unicode/OCR handling, Indic analyzer, Devanagari tokenizer and embedding path. Dense works but BM25 fails: diagnose analyzer/token mismatch; use tuned Unicode aware tokenization, Hindi analysis and approved transliteration/synonym mappings without corrupting exact identifiers. Each branch finds evidence but merged top k does not: examine hybrid fusion ranks, candidate limits, language biased rerankers and duplicate grouping. All retrieval branches return correct chunks, answer still wrong: inspect answer model multilingual comprehension, prompt language and citation use; that is downstream answer generation rather than first stage retrieval. Only one topic or dialect fails: inspect real user lexicon, domain specific Hinglish expressions, idioms and multilingual source coverage rather than replacing every model. 4. Worked numerical production example Construct 300 meaning equivalent test questions from 100 real support intents: 100 English, 100 Devanagari Hindi and 100 Romanized Hinglish . Each variant has an adjudicated authorized relevant policy chunk; all share the same corpus and security filters. In the baseline, relevant chunk appears in top 10 for English 92/100 = 92% , Hindi 64/100 = 64% , Hinglish 46/100 = 46% . Overall macro average across equally sized language groups is (92 + 64 + 46)/3 = 67.3% . A production dashboard weighted 90% toward English could mask the poor Hinglish segment. Known failing Hinglish query Delivery ke baad refund kitne din mein milega? has exact vector rank 47 for the correct document and ANN rank beyond top 10. Its accurate English paraphrase ranks 3 in exact search. This diagnoses semantic query representation, not HNSW graph health. A multilingual encoder experiment using a new matching document vector index improves illustrative Hindi Recall@10 to 82/100 and Hinglish to 72/100 while English holds at 91/100 . The model change alone still leaves 28 Hinglish misses. Add a constrained transliteration/translation query branch plus lexical search for domain keywords, then fuse bounded candidates and rerank under the original query and mandatory filters. Suppose test outcomes become English 93/100 , Hindi 86/100 , Hinglish 84/100 . Macro average = (93+86+84)/3 = 87.7% , an improvement of 20.4 percentage points versus 67.3%; Hinglish improves 38 points from 46% to 84%. Latency rises from p95 140 ms to p95 250 ms in this example, and indexing/storage increases for parallel vector spaces during migration. Approve only if the p95/p99, cost, security and citation quality gates remain acceptable. These metrics are illustrative, not measured MachinoAI production results. For the refund example, test that nahi/“नहीं”, 14 din/“14 दिन”, order identifiers and tenant restrictions remain unchanged. A top 10 retrieval gain is not a successful fix if negation flips or a revoked document is cited. 5. Engineering solution Golden evaluation suite: maintain semantic intent groups with English, Devanagari, Romanized Hinglish, mixed script, spelling variants, technical English loanwords, colloquialisms, negations, dates and exact identifiers. Track quality by language/script and intent rather than only aggregate scores. Model and index: compare current English only and verified multilingual query encoders on the same authorized retrieval tasks. If moving models, rebuild document embeddings consistently into a versioned collection/named vector; never send new model query vectors against an unrelated old model document space solely because dimensions match. Text handling: store original Unicode text and stable source IDs; maintain separate analyzed/normalized fields, never overwrite raw content or invoice/order identifiers with lossy transliteration. Lexical search: benchmark Elasticsearch's hindi analyzer and optional ICU analysis/transform components for actual query vocabulary. The ICU plugin supports Unicode normalization and transliteration, but use precisely tested filters and reindex when index time analyzer changes require it. Query branches: for mixed script queries, test original multilingual dense query, original lexical query and one validated alternate script/translation route. Deduplicate authorized candidates and use a multilingual reranker if evidence supports improvement. Use branch budgets and feature flags, not unbounded translation calls. Constrained fidelity checks: preserve dates, amounts, negations, IDs, product names, roles, effective versions and ACLs. Compare translated text to an entity/constraint ledger and reject unsafe rewrites; use original query as a final relevance/answer check. Rollout: shadow evaluate, canary by language/script, compare Recall@k, nDCG, correct citation rate, wrong policy rate, p95/p99, token/API costs and query rewriting failure. Roll back encoder, index alias, analyzer and fusion settings as a consistent bundle when gates fail. Observability: log protected per stage labels and candidate IDs, analyzer tokens, query language confidence, model route, translation error class, vector space version and final support citations. Sample safely; avoid storing sensitive multilingual text unnecessarily. 6. Production trade offs and common mistakes Multilingual embeddings: a single index can simplify cross language retrieval but may underperform specialized English models on some topics or fail on Romanized dialectal variants. Evaluate both and account for higher per query latency and reindex cost. Translation to English: leverages strong English corpus search, but can change meaning, especially negation, idioms, product codes and geography. Never use it as an unchecked replacement for the original question. Transliteration: helps Romanized to Devanagari matching but is ambiguous (kal may mean yesterday or tomorrow depending context) and is not the same as translation. Hindi lexical analyzers: improve analysis of Devanagari content but may not recover Romanized Hindi, misspellings or cross script synonyms without extra mappings. Multiple search branches: improve coverage but increase latency, complexity and fusion bias; monitor whether English results crowd out the needed policy. Low resource variety: colloquial spellings and English business terms vary across users; synthetic translations can overestimate real world code switching performance. Include native speaker reviews and actual consented query patterns. Over normalization: can erase phonetic distinctions or corrupt numerals, identifiers, legal names and names of customers. Keep original and normalized representations. Security first routing: a multilingual fallback must not bypass tenant/ACL/effective date restrictions, even when the original script query returns no results. 7. Prevent recurrence Make language/script specific Recall@10, nDCG, Hit@k, correct citation rate and zero unauthorized result gates part of every embedding, analyzer, OCR, translation, fusion and reranker deployment. Monitor English versus Devanagari Hindi versus Romanized/mixed Hinglish separately, including fallback rate, zero result rate, p95/p99, query language misclassification and constraint loss incidents. Continuously test minimal meaning pairs: refund kab milega versus refund nahi milega, 14 din versus 40 din, differing product IDs and role/date constraints. Run canary queries through both query and index generations; verify model compatibility, update freshness, current ACLs and source citations after any migration. Review error clusters with Hindi English bilingual reviewers; classify failures as representation, analyzer, transliteration, domain terminology, filter, fusion, reranker or generation. Document tested fallbacks: prefer faithful original query retrieval, optional constrained translation/transliteration, clear abstention when evidence remains insufficient, and no cross tenant disclosure. Interview takeaway: Treat code switching as a measurable retrieval path compatibility problem . Compare meaning equivalent English/Devanagari/Hinglish queries, inspect exact semantic ranks and analyzed tokens, then repair the first failing model, script, lexical or fusion stage. Deploy a multilingual solution only with validated language specific recall, latency, meaning preservation and authorization . Research references Cohere, Embed models and language support: https://docs.cohere.com/docs/cohere embed Cohere, Multilingual and cross lingual semantic search: https://docs.cohere.com/docs/semantic search with cohere Elasticsearch, Hindi language analyzer: https://www.elastic.co/docs/reference/text analysis/analysis lang analyzer Elasticsearch, ICU analysis plugin: https://www.elastic.co/docs/reference/elasticsearch/plugins/analysis icu Elasticsearch, ICU transform token filter: https://www.elastic.co/docs/reference/elasticsearch/plugins/analysis icu transform
A metadata filter unexpectedly reduces the result count from 100 to zero. How would you distinguish filtering errors from genuine data absence?
Production incident (illustrative): A legal policy RAG service returns 100 unfiltered candidates for "2026 contractor travel reimbursement rules". Enabling tenant id=acme AND region=APAC AND effective year=2026 AND document status=active unexpectedly returns zero . The team wants to remove metadata filters to restore results. That could expose documents from other tenants or outdated policies. A zero result set may be correct under legitimate constraints, a metadata typing/ingestion problem , a Boolean filter construction defect , a wrong collection/alias , or ANN post filter starvation . Establish which one with authorized exact counts and a known record witness. Core principle: 100 unfiltered vector hits are not evidence that any of those 100 match the complete predicate. Likewise 0 filtered ANN hits does not prove the filtered corpus is empty . Distinguish (1) eligible record count, (2) search visible eligible record count, and (3) ANN candidate coverage. Keep tenant and ACL enforcement invariant throughout. 1. Safely contain and reproduce Capture a reproducible request: tenant id, principal, query text/vector hash, target collection/alias, shard keys, replica/read consistency, model/version, top k, full serialized filter JSON, and response metadata. Protect private values and do not log sensitive raw payloads indiscriminately. Confirm mandatory tenant and ACL constraints independently of optional business filters. Never simply drop tenant id/authorization for a production query. If a policy cannot be retrieved safely, return an "insufficient authorized evidence" response. Retrieve one known expected active authorized document ID from the authoritative content system. Check its correct tenant, status, geographic applicability, effective date, version and actual stored payload types. A witness makes testing more informative than tweaking top k blindly. Confirm index identity: current alias, namespace, collection, embedding version, field schema, routing keys and shard/replica state. A stale read replica or old index version can contain vectors yet lack newly populated metadata. Build a protected test dataset or an isolated audit environment for comparing omitted predicates; perform any deliberately wider counts only with an authorized operator role, aggregate outputs and minimal retained information. 2. First determine if the eligible set is actually empty Use the authoritative database/source manifest to count active records satisfying the exact business predicate, with the same tenant and authorization conditions. Independently use the vector engine's filtered count API when supported, rather than relying on vector top k. Qdrant /points/count accepts filters and exact=true for exact counts; exact=false may be approximate during indexing. Elastic count can help isolate source/index membership. Compare counts at three stages: source eligible , index payload eligible , query visible eligible . If source count is 0 and predicate semantics are correct, genuine absence is plausible; confirm policy dates, scope and ACL before telling the user "no records." If the source count is positive but the index filtered count is 0, suspect missing/stale payload, field mapping, wrong alias/tenant route, type mismatch or delayed indexing—not ANN graph quality. If filtered exact count is positive but vector search returns 0, inspect ANN pre filter/post filter semantics, candidate budgets, threshold and timeouts. Check whether filtered exact/flat retrieval finds relevant points under the same predicates. 3. Clause by clause root cause debugging Start with mandatory authorization (tenant and ACL) unchanged, then add optional filters one at a time and record exact eligible counts. Do not compare an unsafe globally unfiltered count with a tenant filtered result as if they had the same population. Test tenant id and permitted principal against exact stored payload; string case, trailing spaces, ID serialization and hierarchy may differ. Treat a missing ACL field as deny by default, not an invitation to remove the filter. Inspect region: APAC vs Asia Pacific vs IN vs multi valued arrays. Decide whether "APAC" means direct label membership or a business defined region hierarchy ; enforce the correct mapping and include region map version. Inspect effective year: number 2026 versus string "2026". Match filter types to field schema; coercion differs across engines. For an effective date range , don't substitute a simple calendar year for an interval containment test. Test document effective from <= query date and effective to query date (or explicit open ended semantics) in a timestamp aware implementation. Inspect document status: ACTIVE, active, published or boolean flag; a new pipeline may have renamed the field without backfilling existing points. Determine whether the source genuinely marks the record inactive. Inspect missing and null: missing field, explicit JSON null, empty array and empty string can behave differently. Weaviate's null state filtering, for example, has indexing requirements and documents a convention that empty strings/arrays are treated as null. Avoid broad assumptions across providers. Inspect the Boolean expression: a must/AND clause requires every condition; should/OR needs only one; must not reverses inclusion. Check a mistakenly generated must not, two incompatible statuses, or an AND where business meaning needs OR. In Qdrant an array can require a nested condition if all constraints must hold on the same array element . Inspect field type and text analysis: equality queries on analyzed text may not mean exact key equality; use field mappings such as Elasticsearch keyword and verify field caps, mappings and diagnostic explain/profile as available. Inspect vector search specifics: post filtering a small top k may remove every candidate despite matching points elsewhere; pre filtering/exact eligible retrieval can test whether the set is nonempty. Also check a strict score threshold, search timeouts, shard partial responses and stale replicas. 4. Worked numerical production example One tenant has 1,000 authorized active vector points . Unfiltered within tenant semantic search returns top 100 , but that is only a ranked sample, not a filtered count. With tenant/ACL constraints always enforced, an exact count gives: 1,000 authorized; add region=APAC → 400 ; add status=active → 240 ; add effective year=2026 as an integer equality → 0 . The source manifest says 180 valid APAC/active documents for 2026. Inspecting a permitted witness shows payload {"effective year":"2026"} stored as a string ; the filter compares it to numeric 2026. The missing matches are caused by a schema/type contract violation , not a lack of policies. A controlled corrected string equality filter produces 180 eligible points , matching the authoritative manifest. Of the original unfiltered top 100 hits, suppose only 6 were among those 180; that is why "100 unfiltered" cannot be interpreted as 100 eligible. Hypothetical separate check: the retrieval pipeline performs dense top 10 first , then applies the APAC/year/status constraint; all 10 happen to be outside the eligible set, yielding 0 . Engine native pre filtering and an exact filtered search return matching documents, revealing post filter starvation in addition to the type defect. At the same authorized snapshot, corrected pre filtered ANN top 10 finds the labelled relevant chunk in 9 of 10 controlled queries versus 0 of 10 previously. These numbers are illustrative and too small to claim population wide recall improvement; validate using a larger representative golden set. Count difference = 180 source eligible − 0 broken index filter matches = 180 missing matches ; corrected index count = 180 , so count reconciliation passes. Yet a count agreement alone does not prove each ID/version or ACL is correct —compare exact IDs, source hashes and per tenant samples. Query p95 might rise from 95 ms to 130 ms after stricter native filtering due to extra search effort; this is an illustrative 35 ms trade off to evaluate against the service SLO. 5. Engineering solution and verification Define a versioned metadata contract : canonical tenant identifiers, ACL policy/version, region taxonomy, status enum, timestamp and numeric representations, null conventions, and nested array semantics. Generate filters from typed builders rather than unvalidated LLM strings. Add ingestion validators: reject or quarantine points with incompatible effective year types, missing required authorization metadata, invalid status, or ambiguous region codes. For existing data, backfill only correctly authorized active records from the source of truth and reconcile by stable point IDs. Rebuild or adjust relevant payload indexes or field mappings if required by the backend, after verifying compatibility and rollout resources. Avoid unnecessary full embedding recomputation when only metadata is wrong. Use authoritative count and vendor filter diagnostics before increasing ANN top k or ef search; these parameters cannot repair a logically impossible predicate or mismatched field type. Choose backend native pre filtering/eligible set search for selective metadata predicates, with sufficient ANN effort and a benchmark against exact filtered kNN . Avoid late post filtering when correctness requires all top k results to be eligible. Revalidate AND/OR grouping, range boundaries, missing value behavior and nested arrays. Store a normalized, parameterized filter representation with its schema version for reproducible traces. Shadow test the repaired path, perform a small canary with source to index count/ID reconciliation, then promote only when ACL safe retrieval, candidate coverage, latency and freshness gates pass. Roll back the typed metadata migration if it regresses query correctness—but never roll back to a state that exposes revoked data. 6. Trade offs, failures and common mistakes No matches vs retrieval miss: exact filtered count zero suggests no matching index payloads; ANN zero despite count positive suggests ranking, search execution or filter placement. Neither alone proves source truth. Relaxing filters: may increase hits but violates tenant isolation, policy effective dates or access controls. Diagnostic widening belongs only in a separate authorized audit environment; production mandatory filters remain enforced. Post filtering top k: cheap to implement, but can eliminate all candidates if the eligible subset was never in the original k. Larger k may lessen symptoms but does not guarantee exhaustive candidate coverage. Pre filtering: generally respects eligibility during candidate generation but can increase search cost for very selective predicates and depends on engine specific filtered ANN strategies. Type coercion: silent coercion may make some filters seem to work but can produce unexpected boundary and string comparisons. Typed contracts are safer. Dates: local time zones, inclusive/exclusive boundaries and effective to null semantics can introduce genuine exclusion even with consistent metadata. Missing/null: allowing records with missing ACL metadata is a security regression; deny by default and repair from the authoritative system. Indexes and replicas: a mapping change/backfill may not be visible instantly on every replica or generation; verify completion, consistency and alias identity before cutover. Counts: aggregate count equality hides which records are wrong; verify expected authorized witness IDs, version hashes and sample citations. 7. Prevent recurrence Maintain golden tests for tenant/ACL combinations, regional hierarchies, status values, year/date ranges, null and missing fields, numeric vs string types, nested attributes, and contradictory predicates. Dashboard a filter attrition funnel : authorized total → each business clause → exact eligible point count → filtered ANN hits → fused candidates → answer citation. Segment by tenant, index generation and query class. Alert on abrupt increases in nonempty source && zero filtered index, high rates of empty results on known authorized canaries, and metadata contract validation errors after deployments. Run source to vector ID level reconciliation , not just counts. Include versioned access and deletion events in every backfill/migration. Require negative security tests proving a filter rewrite never returns an unauthorized tenant or a superseded policy. Test pre and post filter behavior separately with high selectivity fixtures. Trace full filter JSON, schema version, index alias, exact count diagnostic and principal eligible snapshot without exposing confidential user data. Document a ten minute incident runbook: confirm source witness → count authorized corpus → count each clause → inspect typed payload/mapping → compare exact filtered search to ANN → fix first failing boundary → verify ACL and latency . Interview takeaway: An empty filtered result is a hypothesis, not an explanation . Compare authoritative eligible records with exact index filter counts, add clauses one at a time, inspect payload types/null/date and Boolean semantics, then distinguish actual empty data from filtered ANN candidate starvation. Never fix “zero results” by bypassing authorization. Research references Qdrant, Filtering conditions including must/should/nested and missing fields: https://qdrant.tech/documentation/search/filtering/ Qdrant, Count points API and exact counts: https://api.qdrant.tech/api reference/points/count points Elasticsearch, Troubleshoot searches, count, field capabilities: https://www.elastic.co/docs/troubleshoot/elasticsearch/troubleshooting searches Weaviate, Conditional filters, null and missing semantics: https://docs.weaviate.io/weaviate/api/graphql/filters Elasticsearch, kNN search and filtered search: https://www.elastic.co/docs/solutions/search/vector/knn