Retrieval-Augmented Generation

Retrieval-augmented generation connects a generative model to a searchable knowledge source. Instead of relying only on model weights, the system fetches relevant documents and places them into the prompt or context.

Two nearly identical queries produce completely different retrieval results. How would you assess retrieval instability?

Production incident (illustrative): A financial policy RAG assistant answers “Can contractors claim meals on international trips?” with the current contractor travel policy, but answers “Are meal expenses reimbursable for contractors traveling abroad?” using a superseded employee FAQ. A second incident appears when the exact same query alternates between the current policy and a stale replica's result list. Both look like “unstable retrieval,” but the causes—and remedies—are different. Core principle: First distinguish semantic paraphrase sensitivity (different wording, same intended facts) from identical request run to run variability (unchanged query, different outcomes). Then separate benign neighbor reorderings from failures in authorized evidence recall, policy versions, citations, and answer correctness. Approximate nearest neighbor (ANN) search need not return byte identical rankings; retrieval quality and security invariants are what matter. 1. Contain and define the incident Keep mandatory tenant, ACL, document version and effective date constraints enforced, even if a looser search appears to stabilize answers. Temporarily use a trusted authorized exact/BM25 lookup for high stakes policy questions when required evidence is missing; otherwise abstain. Record both raw questions and their normalized/rewritten forms, model/revision, embedding input role, full query vector hashes, active index alias, vector name, filters, top k, ANN effort, query timestamp, request routing, shard/replica ID, index generation, fusion settings, reranker version, response candidates and citations. Verify the pair is actually meaning equivalent . “May employees claim meals?” versus “May contractors claim meals?” is not a paraphrase; legal negation, geography, dates, role and exceptions materially change the authorized answer. Define the answer bearing eligible chunks, source version and relevancy labels before tuning. Two different chunks can both be correct evidence; low identifier overlap alone does not establish harmful instability. 2. Controlled experiment: two axes, not one Paraphrase axis: create meaning equivalent pairs preserving actor, scope, negation, year, entity IDs and information need. Hold corpus snapshot, ACL, index configuration and read replica fixed; compute one stable embedding per variant and compare first stage and end to end results. Repeatability axis: submit the identical query vector and exact same serialized search request 20–50 times to the same replica, then repeat across replicas and during controlled indexing. This isolates query embedding variation from ANN, index mutations and routing effects. Production axis: run across different time windows, client versions, shards, load and cache states. Differences appearing only here can indicate refresh lag, replica divergence, background segment optimization, changing aliases, time sensitive filters or concurrency dependent timeouts. Change one factor at a time: compare raw versus rewritten question, old versus new embedding model, ANN versus exact, lexical versus dense, filter on/off only in authorized test fixtures , fusion on/off and reranker on/off. Preserve fixed query IDs, versions, eligible source sets and exact search baselines. Without a frozen corpus and filter policy, natural ingestion or authorization changes may be mistaken for retrieval instability. 3. Metrics: stability is not identical to relevance Top k Jaccard similarity: |A k ∩ B k| / |A k ∪ B k|. Measures set overlap, ignoring rank; useful for paraphrase pairs and repeat runs. Low Jaccard can still be harmless if both sets contain relevant evidence. Rank sensitive overlap: use rank biased overlap (RBO) or measured top rank agreement to distinguish one swapped low ranked neighbor from replacement of the most important citation. Treat score ties as a separate bucket. Relevant document consistency: measure whether each semantically equivalent query hits at least one authorized correct chunk at k; also track pairwise both success , one success, neither success. This directly captures whether paraphrasing loses necessary evidence. Quality: labelled Recall@k, nDCG@k, MRR, citation support, answer equivalence and unsafe/superseded source hit rate. Compare metrics across query phrasing variants, query types and tenants. ANN approximation gap: compare approximate top k to exact/flat nearest neighbors over the same filtered corpus for each query vector. ANN Recall@k reveals search index error; it does not prove the semantic encoder ranks the right document. Embedding sensitivity: compute cosine similarity between the paired query embeddings only inside the same model space , then inspect exact nearest neighbor ranks. High query cosine similarity does not guarantee identical top k near decision boundaries. Confidence and sample size: bootstrap at the intent group level , not independently over correlated paraphrases; report uncertainty and query class breakdown. Never treat arbitrary Jaccard thresholds as universal release gates. 4. Root cause decision tree Same raw request, different query vector hash: inspect nondeterministic rewriting, chat history contamination, embedding model/router changes, stochastic normalization, hidden metadata and stale cache keys. Same vector, different exact results on frozen snapshots: inspect corpus/ACL/replica mismatch, distance metric config, score ties/tie breaking, version changes, inconsistent filters, shard errors and client request differences. Same exact reference, ANN rankings change: inspect HNSW approximation, hnsw ef/num candidates, segment compaction, quantization/rescoring, indexed only semantics, background updates and replica index generations. Qdrant explicitly warns ANN ranking can shift between requests and offset pagination can repeat/skip IDs. Identical request stable on each replica but varies across replicas: compare update propagation, read consistency, shard routing, optimizer readiness and effective document versions. Qdrant describes how default single replica reads can “blink” while replicas converge and supports stronger read consistency and version specific read affinity. Paraphrase results differ even for exact kNN: suspect query representation, domain vocabulary, negation handling, concept disambiguation, truncated query text or insufficient paraphrase robustness of the embedding model—not HNSW graph behavior. Dense is stable, hybrid or reranked outputs differ: audit BM25 analyzer/rewrites, dynamic fusion weights, candidate depth limits, RRF, deduplication, random tie resolution, reranker input truncation and result caching. Differences only under production load: check timeouts, partial shard responses, connection pooling, retry routing, caches and index updates. Determine whether the system returns incomplete candidates as successful responses. Differences are only score tied IDs near rank k: likely benign top k boundary variance, unless they remove a crucial source; use stable secondary sorting for exact deterministic paths when supported, without claiming ANN can always guarantee stable ranking. 5. Numerical worked example: stability versus correctness Use 100 business intents , each represented by two expert validated equivalent questions (200 searches), one known authorized policy answer target per intent. Every variant is tested on the same index snapshot. At baseline, 62/100 pairs retrieve valid evidence for both phrasings, 28/100 pairs succeed for only one phrasing, and 10/100 pairs fail for both. Thus both success rate = 62% ; at least one success rate = 90% ; per query success is (2×62 + 28) / 200 = 152/200 = 76%. The gap between 90% and 62% exposes paraphrase sensitivity invisible in single query evaluation. For one pair, top 5 ID sets are A={P1,P2,P3,P4,P5} and B={P1,P6,P7,P8,P9}. Their intersection has 1 ID and union has 9 , so Jaccard@5 = 1/9 ≈ 0.111 . If P1 is a valid current policy paragraph and both cite P1, set instability does not imply an answer failure; if B cites an expired FAQ instead, it is harmful. A controlled repeated query test holds the same query vector and backend filter fixed for 20 requests . The assigned replica returns an identical top 5 for 19/20 trials, but another replica with lagging policy updates returns different results in 15/20 trials. The exact numbers are illustrative; diagnose replica lag independently instead of claiming intrinsic ANN randomness. Using a verified updated replica and tuning the ANN candidate budget, repeat run consistency improves, but paraphrase pairwise both success remains below the target, indicating encoder/retriever sensitivity remains unresolved. A benchmarked semantic representation improvement plus carefully controlled lexical/dense candidate fusion raises both success to 84/100 pairs , one success to 12/100 , and neither success to 4/100 . Per query correct evidence success becomes (2×84 + 12)/200 = 180/200 = 90%. Pairwise both success improves 22 percentage points (62% → 84%). Suppose search p95 rises from 140 ms to 220 ms due to added candidate depth and reranking. Accept only after testing capacity, semantic relevance, evidence citations, authorization, and latency SLOs. These counts are illustrative , not measured MachinoAI traffic. 6. Engineering solution by identified failure mode Deterministic replay harness: log immutable query vector fingerprints, normalized query, request parameters, target collection generation, shard/replica mapping, source/filter snapshot and candidates. Run a fixed, rate limited reference benchmark periodically. Read path consistency: for genuinely stale replicas, repair replication/index readiness, route to sufficiently current peers, and select an appropriate supported consistency setting. Do not use stale replica results for policy/ACL changes or delete sensitive answers. ANN tuning: benchmark hnsw ef or per shard num candidates, quantization and rescoring against exact retrieval, while monitoring Recall@k and p95/p99. Qdrant exposes exact=true; Elasticsearch documents per shard num candidates. Higher effort may improve overlap but costs CPU/time. Representation fixes: evaluate paraphrase robust multilingual/domain embeddings; preserve original intent and all hard entities in rewrites. If encoder changes, re embed documents into a matching versioned vector space and switch query encoder and index coherently. Candidate aggregation: consider BM25/dense fusion for domain specific terms and a sufficiently wide, reproducible candidate pool. Use a deterministic, stable reranker where available. Do not average unrelated raw similarity scores. Evidence aware outputs: canonicalize equivalent source versions and document aliases for quality metrics; require current, authorized evidence and consistent citations rather than identical neighbor IDs. Operational rollout: shadow evaluate changes on an intent grouped golden set, canary across replicas and query types, gate on pairwise both success, Recall@k, unsafe citations, latency and error rate, then promote with a tested rollback bundle. 7. Trade offs and common mistakes Forcing exact search: improves reproducibility and diagnostic confidence but may be too slow/costly for millions of vectors. Apply to bounded diagnostic subsets or targeted, security critical requests. Increasing ANN effort: improves approximation accuracy in many settings but cannot repair a poor semantic embedding or lossy query rewrite. Stronger consistency: reduces stale replica disagreement but adds replica work, latency or availability trade offs; document the business need. Query rewriting/paraphrase expansion: sometimes improves coverage but risks changing dates, negation, roles or source scope and increases computation. Do not rewrite away mandatory filters. RRF and rerankers: may stabilize relevant evidence but can obscure whether the dense encoder, lexical branch, or fusion caused the original change. Keep branch level traces. Cache pinning/read affinity: can reduce visible blinking but may hide stale data rather than solve replication lag. Verify freshness and delete/ACL propagation first. Comparing only top k overlap: penalizes equivalent valid chunks and may reward a consistently wrong retrieval. Pair overlap with labelled relevance and version aware citation correctness. Ignoring near ties: tiny score differences can reorder boundary results harmlessly; do not overreact to a changed 10th neighbor if supported answers and ACLs are stable. 8. Prevent recurrence Build a meaning equivalent paraphrase evaluation suite covering domain synonyms, active versus expired policies, negative constraints, invoice IDs, multilingual variants, short/long queries and time effective source versions. Validate equivalence with knowledgeable reviewers. Monitor both success per intent , one sided paraphrase failures, Jaccard@k/RBO, ANN exact reference recall, relevant citation consistency and unauthorized/superseded result rates; segment by model version, corpus generation, tenant, query class and replica. Add same vector repeatability smoke tests across replicas, before and after index optimization, node upgrades, high ingest rates and consistency mode changes. Alert on sudden disparity between paraphrase variants, stale replica/document citations, growing non deterministic rewrite rates, unexpected partial shard responses or model/alias drift. Make each retrieval release auditable: pin versions, corpus/filter snapshot, embedding contract, index settings, fusion/reranker configuration and success thresholds. Rehearse regression rollback without resurrecting deleted records. Require causal diagnosis before adopting multi query consensus or deeper search; recent paraphrase stability research finds that diversity/consensus can improve coverage, but simple per query adaptive weighting may show little benefit. Interview takeaway: For two similar questions producing different evidence, test repeatability and paraphrase robustness separately . Freeze the query vector and corpus to identify backend/index/replica instability; compare validated meaning equivalent queries to isolate semantic sensitivity. Judge fixes using authorized relevant evidence and citation consistency , not top k identity alone. Research references Qdrant, Search API, exact search, ANN ranking stability: https://qdrant.tech/documentation/search/search/ Qdrant, Read consistency and replica affinity: https://qdrant.tech/documentation/scaling/consistency guarantees/ Elasticsearch, kNN num candidates and per shard merging: https://www.elastic.co/docs/reference/query languages/query dsl/query dsl knn query Mandiga et al., Paraphrase Induced Retrieval Instability: Systematic Measurement and the Limits of Inference Time Mitigation (2026): https://aclanthology.org/2026.uncertainlp main.34/

Retrieval quality improves in offline testing but becomes worse after production deployment. How would you identify differences in traffic, configuration, and indexing?

Production incident (illustrative): A support RAG team evaluates a new hybrid dense+lexical retrieval configuration on a labelled offline benchmark. Recall@10 rises from 82.0% to 87.2% , so the team promotes it. Within 24 hours, the live retrieval correct evidence rate falls from 62.0% to 53.6% , policy citations disappear on some newly updated documents, and complaints cluster among long multilingual requests. Dashboards show healthy query latency and successful ingestion. Was the experiment wrong, did real traffic change, or is the deployed retrieval system different from what was tested? Principle: An offline improvement is conditional on evaluation population, ground truth, corpus snapshot, index health, filters, model and configuration versions, and measurement method . Debug the offline to online contract by reproducing the same query across the same population and snapshots before blaming a model. A distribution shift and an actual serving defect can occur at the same time . 1. Contain the incident and define what regressed Stop expansion of the rollout; preserve the proven previous retrieval bundle and feature flag. For high impact policies, route to a safe authorized previous index or abstain if the previous index is too stale to respect deletions and ACL changes. Confirm the regression is retrieval and not solely answer generation: compare first stage candidates, reranker output, source versions, citation support, answer correctness, safety errors and downstream latency. A healthy HTTP 200 rate is not a retrieval quality metric. Freeze the observed time window, query sample and relevant source snapshots. Record request id, tenant, redacted query hash, consent safe query attributes, language/script, token length, intent class, exact identifier presence, filters, query embedding model/revision, reranker version, serving collection alias, search parameters, traffic cohort, shard and replica. Do not compare offline Recall@10 against unlabelled production clicks as though they were equivalent metrics. Clicks, answer thumbs and downstream events are selection biased, delayed and affected by UI . Use adjudicated online relevance labels or controlled replay for a like for like comparison. Prioritize security: never remove tenant, ACL or effective version restrictions just to improve recall. Assess new misses only against documents that were eligible at each query's event time. 2. Build an offline to production equivalence checklist Query population: Does offline data overrepresent short English questions and underrepresent Hindi English code switching, complex enterprise questions, misspellings, newly introduced product codes, rare tenants and seasonal traffic? Compute per slice shares and confidence intervals. Query transformations: Are query rewriting, conversation memory expansion, normalization, language detection, embedding truncation, hyphen/identifier processing, query prefixes or model input type identical? Compare the actual bytes embedded and vector fingerprint, not just original user text. Embedding and ranking versions: Pin both query/document embedding family and revisions, vector dimensions, distance metric, normalized vector policy, BM25 analyzers, hybrid fusion weights, score thresholds, reranker version, top k and candidate limits. Default settings often differ across local test scripts and production clients. Index and data snapshot: Inventory index generation/alias, shard availability, document versions, chunk templates, vector count, payload index mappings, delete/tombstone state, indexing completion, exact versus ANN behavior and replica lag. Did offline use a fully optimized snapshot while production is ingesting updates? Filter and authorization: Reproduce the exact serialized tenant/ACL, source effective date, region and status filters. A diagnostic test can use a privileged isolated fixture, but the production serving path must enforce current mandatory filters. Serving conditions: Compare read consistency, query timeout, partial shard responses, retry/routing policy, concurrency, cache state, CPU pressure, cold/warm index, segment optimizer status, indexed only mode and vector search candidate budget. Evaluation protocol: Check leaked near duplicate questions, use of future document revisions, benchmark answer keys mismatched to source versions, inconsistent top k and judge rubric drift, or reranker tuning on the same tiny set used to report improvement. 3. Step by step root cause debugging A. Validate the metric: sample production misses and have reviewers judge whether authorized relevant evidence existed at request time. Label with source version, tenant and factual support; distinguish retrieval misses from generator failures. B. Compare traffic mix: build the same strata offline and online (short/long, English/multilingual, common/rare intent, exact ID, tenant, source age). Compute old and new quality within each stratum and then standardize both results to the same reference population . A weighted aggregate can change even without a change in per stratum quality; do not claim degradation solely from a shifted aggregate. C. Replay production queries offline: on a frozen production source snapshot , execute old and candidate retrieval with the same authorized filters and identical query vectors. Compare candidates and citations. If candidate wins when fully reproduced, focus on serving differences , not just benchmark selection. D. Replay identical requests on production: use approved synthetic or safe audit queries with identical inputs and pin the model, alias, shard/replica, search settings and source snapshot as closely as the backend permits. Compare candidate IDs at every retrieval stage. Inspect partial shard responses, retry/fallback paths and cache divergence. E. Check query vector contract: verify model name/revision, correct query/document input roles, normalization, dimension, actual input string and any feature flag routing. Mixed query/document embedding spaces are unsafe even when their dimensions match. F. Compare exact and approximate search: get exact nearest neighbor results on the same filtered corpus; measure ANN Recall@k. If exact search is good but ANN weak, check hnsw ef, num candidates, quantization, rescoring, optimizer backlog, indexed only mode, replica readiness and search thresholds. G. Trace filtered and hybrid candidates: log dense and lexical branch ranks, per branch budgets, source version, metadata exclusion reasons, RRF/fusion, reranker input and final top k. A production only top k=5 instead of offline top k=40 can hide a relevant chunk before reranking. H. Examine freshness and writes: compare source change log offset, accepted upserts, query visible points and active index alias. Elastic search visibility depends on refresh; Qdrant can temporarily hide fresh points using indexed only or an experimental defer indexing mode. I. Test load induced behavior: correlate errors with peak QPS, p95/p99, query budget, timeouts, partial shards and cold cache. A request that returns fewer candidates under a deadline should be treated as a quality degradation, not silently accepted as a valid full result. 4. Numerical example: distinguish population shift from a deployment bug Offline benchmark: 1,000 labelled queries: 800 common short English questions and 200 long/multilingual questions. Old system succeeds for 720/800 = 90% common and 100/200 = 50% long, total 820/1,000 = 82.0% . The candidate succeeds for 752/800 = 94% common and 120/200 = 60% long, total 872/1,000 = 87.2% . The claimed offline gain is +5.2 percentage points on this 80:20 mix. Production traffic shifts: only 300 of 1,000 live questions are common and 700 are long/multilingual. Assuming the old system retains its per slice rates, its expected live success is 270 + 350 = 620/1,000 = 62.0% . At unchanged offline candidate rates, the candidate should get 282 + 420 = 702/1,000 = 70.2% . The lower absolute aggregate compared with 87.2% is explained by the 30:70 production mix , but the candidate should still outperform old by 8.2 points . Actual candidate deployment: observed live success is 270/300 = 90% common and 266/700 = 38% long, total 536/1,000 = 53.6% . That is 8.4 points worse than the old live estimate. Traffic mix alone cannot explain it because long question performance fell from the candidate's 60% offline to 38% live. These are illustrative observed counts, not a randomized controlled comparison. Root cause verified with paired replay: the serving API accidentally uses a shallower vector candidate budget and a production query rewrite that truncates long queries; an indexed only search mode also withholds some newly ingested policy vectors. Inspecting actual parameters and controlled query traces confirms each mismatch rather than inferring it from aggregate metrics. After tested repairs , assume within slice candidate rates become 94% common and 60% long on the same audited production cohort and snapshot. Expected quality returns to 70.2% , +16.6 points versus the broken 53.6% and +8.2 points versus the old 62.0% . These are illustrative outcomes; real impact requires adequate labelled sample sizes and a controlled online experiment. Operational cost: restoring full candidate depth might raise p95 retrieval latency from 140 ms to 225 ms and increase reranking calls. Approve only if latency, cost, freshness and authorization SLOs remain within defined limits. Do not optimize recall at the cost of unauthorized citations. 5. Engineering solution and production rollout Version the entire retrieval bundle as one deployment unit: query transformation, embedding model/revision/input role, target vector space/index alias, index generation, filter builder/schema, ANN search parameters, lexical analyzer, branch depth, fusion and reranker configuration. Use typed config manifests and log one generation ID per request. Run a same query/same snapshot replay harness on sanitized or explicitly authorized production like traffic. Keep a source manifest and labelled relevance set; diff candidate IDs at stage boundaries. Store outputs with separate metric definitions and confidence intervals. Implement guarded shadow traffic : duplicate retrieval internally to the candidate without showing its result, using a consistent query and permitted snapshot. Shadow evaluation measures relevance under realistic traffic but does not directly measure user behavior, so follow with controlled canary. Canary 1% → 5% → 25% → 100% only after quality, security, freshness and p95/p99 gates pass. Use representative strata so rare high risk cases are not missed; compare cohorts randomized by stable user/tenant assignment when feasible, avoiding cache contamination or interference. For index freshness: use the vendor's documented refresh/visibility mechanism rather than brute force refreshing every write. Elastic refresh=wait for waits for refresh without forcing an immediate refresh. Qdrant indexed only prioritizes latency but can temporarily exclude freshly updated points; monitor visibility lag when choosing it. Correct production only feature flags and defaults, increase candidate budget only after tracing where relevant chunks were dropped, and fix missing payload indexes or unsafe filters without bypassing tenant ACLs. Promote only when per slice Recall@k/nDCG, citation support, no unsafe result gate, freshness, latency, throughput and cost all pass; maintain a safe traffic rollback that includes current deletions and ACL updates, rather than restoring a stale index blindly. 6. Trade offs, failure scenarios and common mistakes Representativeness vs benchmark stability: a fixed offline benchmark is reproducible, but becomes stale as new languages, products and tenants appear. Maintain both fixed holdout and recently labelled production query suites. Population shift vs regression: Simpson's paradox can make aggregates misleading; reweight to a reference population and compare within slice quality. Reweighting does not repair an actual config/index defect. Shadow vs A/B: shadow is safer for observing retrieval and infrastructure cost, but user behavior and downstream answers under candidate routing aren't directly measured. Live A/B offers stronger evidence but needs randomization, sample size and guardrails. Strong read consistency: can improve freshness but costs additional latency/replica work and may lower availability under partitions. ANN depth/recall: higher ef search or num candidates can improve candidate recall while increasing latency and compute. Tune against exact retrieval baselines on representative filters. Indexing lag: an offline static snapshot has no newly unindexed points; a live index with frequent writes may. More frequent refresh or larger optimizer budget can increase write/search contention. Offline label leakage: benchmarks built after the incident may accidentally include future documents or answers, making results unrepresentative of historical online decisions. Metric mismatch: clicks and user satisfaction are not identical to Recall@10. Account for delayed labels, selection bias, time of day effects and evolving source version eligibility. Unsafe rollback: replacing green with a stale blue collection can resurrect deleted or unauthorized documents. Keep a current source side deny gate and mutation replay. 7. Prevent recurrence Maintain an offline–online parity dashboard for tokenizer and embedding inputs, model/alias generation, filters, top k, ANN effort, lexical/fusion settings, reranker, replica health, index visibility, cache and partial results. Compare performance by query class, language, length, tenant, exact ID presence, source age and filter selectivity , with standardized aggregates on a reference mix and alerting on harmful slice level drops. Add synthetic canary queries and recent document updates to test source to search visibility; record time from write acknowledgment to first authorized retrieval. Require a signed deployment manifest, reproducible replay test, frozen corpus snapshot and labelled production like query holdout before promoting any ranking or index changes. Instrument step wise candidate survival: raw query → embedded input → dense/BM25 branch candidates → filtered candidates → fusion → reranker → cited source. Fail safe on missing shards and unauthorized evidence. Rehearse counterfactual cases: drifted traffic mix, stale index alias, failed refresh, wrong vector space routing, production only low candidate limits, score threshold change, high load timeouts and filter schema changes. Set explicit rollback triggers and audit version switches; verify current ACL/deletion eligibility during fallback. Investigate incidents with paired replay instead of inferring root cause from one aggregate score. Interview takeaway: When offline retrieval improves but production worsens, first match the populations and measurement definitions , then run the same queries through the same indexed corpus and security rules while diffing query transformations, model/index generations, ANN/fusion budgets and freshness. A shifted query mix may explain a lower aggregate, but it cannot excuse a verified within slice deployment regression. Research references Microsoft Azure Architecture Center, RAG information retrieval and reranking: https://learn.microsoft.com/en us/azure/architecture/ai ml/guide/rag/rag information retrieval Microsoft Foundry, retrieval evaluators and labelled relevance metrics: https://learn.microsoft.com/en us/azure/ai foundry/concepts/evaluation evaluators/rag evaluators Qdrant, indexed only and index optimizer visibility: https://qdrant.tech/documentation/search/low latency search/ Elasticsearch, refresh parameter and visibility: https://www.elastic.co/docs/reference/elasticsearch/rest apis/refresh parameter Qdrant, production vector search behavior: https://qdrant.tech/documentation/deploy and scale/vector search in production/

Search for technical product codes works during testing but fails for new code formats introduced in production. How would you prevent this regression?

Production incident (illustrative): An enterprise hardware support RAG assistant reliably finds manuals for product codes like AX 4200 in pre release tests. A manufacturer then launches codes AX/42.00 R2, AX 4200 EU and 00 AX 4200. Users searching those exact strings get unrelated installation guides or zero results. The index is healthy, but an identifier extraction regex allows only uppercase letters, digits and hyphens, a text analyzer splits punctuation, a normalizer strips leading zeros, and the offline test suite contains only older formats. A production search quality regression arises from identifier schema drift , not from a general failure of vector similarity. Key principle: A product code is a structured identifier with a versioned grammar and an authoritative issuer. Semantic embeddings may help interpret associated product questions, but neither dense similarity nor general linguistic tokenization guarantees exact character fidelity. For exact code intent, correctness requires a source validated, tenant authorized identity match ; optional prefix/partial/fuzzy search must be a separately labelled fallback. 1. Contain the incident and confirm ground truth Identify affected code families, release dates, organizations, regional variants, known missing codes and user requests. Record full input/query path with redacted request id, exact query bytes, Unicode normalization, parser version, index alias/generation and candidate/citation IDs. Check the authoritative product catalog first: does AX/42.00 R2 exist, is it active for the requested region, is the account authorized, and which documents/versions should be linked? A nonexistent or inaccessible product must not be silently replaced with a nearby code. Reproduce source → extraction/OCR → canonical identifier → metadata field → indexed exact key → parser → exact term lookup → hybrid branch → reranking → citation. Locate the first failing boundary and keep all stages observable. Confirm whether user requests are literal code lookups (“manual for AX/42.00 R2”) or general troubleshooting (“AX 4200 overheating after upgrade”). In both cases preserve an explicit product identifier when present. If incorrect manuals are unsafe, disable nearest code substitution, require exact authorized catalog identity, and present a clear not found or clarification response until the mapping is repaired. 2. Root cause decision tree New code absent in source: check product catalog launch synchronization and allowed release/region constraints. Delay retrieval publication rather than manufacturing a record. Present in source, absent from search metadata: compare ingestion schema and event versions, OCR extraction, leading zeros, punctuation, long code truncation, malformed Unicode, source aliases, failed bulk items and reindex/refresh visibility. Metadata stored but exact lookup fails: inspect the actual indexed field mapping and query. Elasticsearch recommends keyword for product IDs, while term is unsuitable on analyzed text fields. Watch ignore above settings that can omit long keyword values from indexing. Parser fails to recognize new format: inspect anchored regex, whitelist, version gates, token boundaries and fallback classifier. A previously safe pattern such as [A Z]+ [0 9]+ rejects slashes, periods, underscores and leading numeric segments; a malformed detector may incorrectly route the entire query to vector similarity. Canonicalization causes collisions: avoid unconditionally removing leading zeros, underscores, hyphens, slashes and periods. AX 4200, AX/42.00 and AX4200 are not automatically equivalent. Equivalence is issuer/catalog specific and must be versioned. Exact lookup works; hybrid result misses: trace lexical candidate list, dense list, candidate cutoffs, RRF/weighting, deduplication, reranker and context assembly. A verified exact match for exact ID intent is a hard identity constraint , not a score bonus that a different product can overrule. Only new formats fail across regions: compare product taxonomy, locale, issuer and entity aliases, document publication delay, query parser model rollout and tenant routing. Multiple manufacturers can legitimately reuse an identical visible code. Partial code queries fail: test prefix, authorized field based wildcards or n gram strategies separately. Prefix search and substring matches are different user intents with different ambiguity and cost. 3. Map codes as typed, versioned product identities Maintain product code raw, product code canonical, issuer id, region, product family, format version, source product id, active version, tenant id and ACL/version metadata. Preserve raw exact bytes or a clearly documented Unicode preserving representation for auditing. Use an explicit keyword/exact field for equality; keep a separately analyzed text field for descriptive prose. Elasticsearch's keyword field is intended for structured IDs. Its term level lookup differs from a match query on tokenized text. Store verified manufacturer/catalog aliases as a many to one controlled mapping only when the issuer confirms equivalence , with effective dates and collision tests. Don't let regex normalization invent aliases. Where product codes need suffix/prefix or infix search, design a separate authorized prefix/wildcard/n gram field and measured performance bounds. OpenSearch offers a wildcard type for substring/regex scenarios, which should not replace exact product identity. Use typed query routing: extract possible code → validate against versioned grammar(s) → authorized catalog lookup → exact index term lookup → confirm source version → optionally retrieve matching product documentation → verify citations. If new syntax is unknown, keep the entire literal token, attempt an authorized exact lookup before assuming it is natural language, and emit a structured unknown format signal for monitoring. A regex is an optimization, not a hard gate to catalog lookup. Explicitly version mappings/dynamic templates. Elasticsearch dynamic mapping can infer surprising field types; dynamic=false can silently ignore unknown searchable fields, while dynamic=strict rejects such documents. Either failure should be observable. 4. Numerical worked example A pre release benchmark has 1,000 older format codes and reports 990 correct top 1 exact product identifications , so Hit@1 = 99% . This figure says nothing about new syntax absent from testing. After launch, production includes 700 old format and 300 new format product code queries. Old format Hit@1 remains 693/700 = 99% , but new format Hit@1 is 90/300 = 30% because only the fallback lexical/dense path finds some of the new codes. Overall is (693+90)/1000 = 783/1000 = 78.3%, even though the old evaluation suite still reports 99%. Controlled debug cases: AX 4200 parses and looks up correctly; AX/42.00 R2 fails the old regex; 00 AX 4200 is normalized to AX 4200 and incorrectly returns a different record; AX 4200 EU is tokenized into fragments in the descriptive text field. Deploy a catalog backed grammar version and dedicated exact keys. In an illustrative retest on the same authorized 1,000 query set , old format success remains 693/700 , and new format success rises to 288/300 = 96% . Overall becomes 981/1000 = 98.1% , a 19.8 percentage point improvement versus the 78.3% broken production baseline. Remaining 12 new format misses are classified as 5 ingestion/refresh delays, 4 unknown issuer aliases, and 3 OCR/canonicalization defects . Do not call the upgrade fully correct until each failure class is repaired or safely reported. Production p95 may rise from 80 ms to 105 ms due to the authoritative lookup, an illustrative 25 ms cost . Validate p95/p99 against the business SLO. The correctness gate is zero wrong product substitution and zero cross tenant disclosure , not simply a high candidate count. 5. Engineering solution: prevent future format changes from breaking retrieval Contract with product data owners: ingest a versioned catalog of known formats, sample codes, intended case rules, issuer namespaces, valid character sets and product aliases. Treat format additions as schema changes requiring integration tests and monitoring. Dual representation: retain exact literal product IDs and canonical validated keys independently of document embeddings, plus descriptive text for BM25/dense retrieval. Attach reliable product keys to all related chunks , not just a title or the first page. Parser resilience: use issuer specific grammars and unknown format fallback to raw exact catalog lookup; never discard a token because it fails the old regex. Maintain an audit reason code (regex unknown, catalog not found, indexed missing, ambiguous). Index lifecycle: validate field mapping via mapping and analyzer output via analyze; create/reindex a versioned exact field index when mapping/analyzer changes require it. Verify current active products, deleted/retired status and ACLs before alias switch. Do not blindly re embed unchanged descriptive content when only product code metadata is broken. Batch reconciliation: compare catalog active IDs against exact index IDs and related document/chunk lineage. Repair partial/failed writes and collect per issuer visibility lag. A zero result vector search is not evidence the product is missing from the authoritative catalog. Safe partial match UX: if the user supplies an incomplete code, present authorized candidates as suggestions with their full codes and model/region labels; require explicit user selection if ambiguous. Fuzzy candidate similarity must never silently change an exact product ID. Canary rollout: shadow test new format families before launch, then canary query routing by issuer/format; monitor exact hit@1, wrong product rate, unknown parser rate, authorization failures, p95 and publication lag. Keep an audited rollback, retaining security fixes and current source deletions. 6. Trade offs and failure scenarios Exact matching: reliable for known complete IDs, but strict when users mistype characters. Add a separate transparent suggestions workflow rather than weakening identity. Normalization: helps legitimate aliases but may collapse distinct model revisions, locales or manufacturer namespaces. Preserve leading zeros unless catalog rules explicitly state otherwise. Wildcard/substring indexing: assists partial lookup at extra storage and query cost; use bounded, tenant filtered queries and validate high cardinality performance. Regex only detection: fast for known formats but brittle when suppliers introduce punctuation or new prefixes. Prefer a catalog driven parser with monitored unknown format paths. Semantic fallback: supports troubleshooting language but can return plausible manuals for the wrong model. Do not let semantic scores override verified structured identities. Dynamic mappings: reduce upfront schema work but may infer incorrect data types or ignore new fields under certain configurations. Explicit mappings provide stronger production contracts. Full reindexing: may be necessary for mapping changes, but causes temporary storage, ingestion and I/O pressure; use staged migration and lineage reconciliation. Cross tenant and regional codes: identical visible product codes may refer to different items or access scopes. Use composite keys, enforce ACLs and avoid disclosing existence outside authorization. Code like input adversarialness: treat user provided code strings as data, not regex execution, code, SQL or generated filter expressions; validate length and construct parameterized queries. 7. Prevention and monitoring Maintain format aware golden cases for hyphen, slash, dot, underscore, mixed case, leading zeros, revisions, long alphanumeric codes, Unicode dashes, cross issuer collisions and region suffixes. Include negative tests for nonexistent and unauthorized codes. Partition evaluation by known versus newly launched formats , issuer, rollout date, tokenization/analyzer, query intent and region. Monitor exact ID Hit@1, wrong code rate, false not found, unknown format rate, ambiguous match count and correct citation rate . Add contract tests to catalog ingestion, OCR/extraction, keyword mappings, query parser, normalization and RAG citation assembly. Run representative future format fuzz tests using valid characters and delimiter variants without assuming any one generated format is real. Alert when newly observed punctuation or code prefixes exceed a baseline, when parsing falls back to semantic search unexpectedly, or when source active IDs are absent from the exact index. Use a release handshake: product data owner registers new grammar, ingestion canary verifies source/metadata, search canary verifies exact term lookup, and the application checks retrieved product identity before returning cited guidance. Review real error samples with product SMEs to catch revisions and aliases missed by regexes; keep source catalog and deletion/ACL policies authoritative. Interview takeaway: A product code regression after launch is usually a format contract failure : an old parser, normalizer, field mapping or test dataset assumes yesterday's syntax. Preserve literal IDs, use authoritative exact match lookups, version the grammar, and test unseen formats before deployment. Hybrid and dense search should support explanations, not override product identity. Authoritative references Elasticsearch, Keyword field for structured/product identifiers: https://www.elastic.co/docs/reference/elasticsearch/mapping reference/keyword Elasticsearch, Term query exact value behavior: https://www.elastic.co/docs/reference/query languages/query dsl/query dsl term query Elasticsearch, Explicit and dynamic mappings: https://www.elastic.co/docs/manage data/data store/mapping Elasticsearch, Dynamic mapping controls: https://www.elastic.co/docs/reference/elasticsearch/mapping reference/dynamic OpenSearch, Wildcard field for substring/code search: https://docs.opensearch.org/latest/mappings/supported field types/wildcard/

Your hybrid search returns many duplicate passages and suppresses source diversity. How would you diagnose the fusion and candidate-selection stages?

Production incident (illustrative): A compliance RAG assistant must answer, “Which 2026 travel reimbursement rules apply to Singapore contractors, and what is the exception approval path?” The dense retriever returns seven near identical chunks from the same global travel policy, while BM25 returns six copies of that policy plus one relevant regional exception memo. Hybrid fusion presents eight passages, but seven originate from one policy revision and the regional exception never reaches the LLM. The assistant gives a plausible but incomplete answer. Ingestion is healthy and the retriever has nonzero Recall@10, so the operational failure is source concentration and candidate selection loss , not necessarily missing documents. Principle: Hybrid retrieval balances semantic and lexical evidence; RRF/score fusion does not automatically guarantee passage uniqueness, independent sources, or coverage of all required facets . Diagnose at the point/chunk → logical passage → document version → authoritative source levels, then inspect where relevant minority evidence is crowded out. Preserve authorization, current version and effective date eligibility while testing diversity. 1. Reproduce and instrument the incident Capture a fixed query, user permissions, tenant, corpus snapshot, index alias, query embedding, BM25 text/analyzer, authorized filters, source effective date, search parameters, per branch candidate counts, rank window, fusion settings, reranker input and final cited passages. Collect every stage separately: dense candidates before fusion; BM25/sparse candidates before fusion; unified identities; fusion scores/ranks; dedup/grouping decisions; rerank results; final top k; and answer context/citations. Tag each result with point id, chunk id, parent document id, source id, document version, section id, content hash, ingestion generation, effective date, and ACL classification. A point ID alone is insufficient to detect replicas of the same passage under different IDs. Verify what counts as a genuinely distinct source . Different neighboring paragraphs may be valuable complementary evidence even from one policy; mirrored copies and repeated chunk overlap are usually not. Build a labelled evidence checklist for multi source questions: e.g. global reimbursement cap + regional exception + approval authority. Measure whether at least one authorized, active chunk supports each required fact. Do not force unrelated sources just to increase diversity. Distinguish source level deduplication from source relevance : an answer requiring one authoritative document may legitimately benefit from several nearby chunks. Diversification should be conditional on user intent and coverage requirements. 2. Root cause debugging: find the first stage that loses diversity A. Source to index duplication: inspect repeated ingestion jobs, unstable chunk IDs, overlapping chunk windows, archival copies, identical text across versions and mirrored connectors. Compare exact content hash plus source/version lineage. If source copies are indexed repeatedly with different IDs, fix ingestion idempotency and versioning first; re ranking cannot permanently cure this. B. Document representation: detect whether one long policy was split into hundreds of largely overlapping chunks while regional memos have only one each. Per chunk retrieval can favor the verbose source because it receives more opportunities to match. Check chunk granularity, overlap, parent IDs and which passages truly answer distinct facets. C. Branch candidate pools: compare the unique source id counts of dense versus BM25/sparse. Determine whether all branches independently favor the same source, or whether the regional memo appears in BM25 rank 12 but is clipped by branch top k=10. D. Identity reconciliation: review deduplication keyed by record ID rather than canonical source+version+content. The same paragraph emitted under different point IDs or both lexical and vector branches may count as distinct when it is not. Conversely, collapsing everything by parent document too early can discard two distinct relevant sections. E. Fusion semantics: inspect exactly which branch result ranks/scores fed the fusion stage, rank window size, rank constant, branch weights, score normalization, top k and the point of truncation. RRF boosts documents appearing near the top in multiple branches; this improves consensus but is not a diversity constraint. A source with many similar chunks can still dominate. F. Normalization and version merging: inspect whether relativeScoreFusion is skewed by score range outliers, whether lexical score dominates, and whether a reranker or fusion groups active and superseded versions together. Weaviate documents distinct rank based and relative score fusion methods; don't assume their outputs are interchangeable. G. Selection/diversification: ask if source aware caps, MMR (maximal marginal relevance), coverage quotas or parent grouping are applied after an adequately wide candidate pool but before irreversible top k truncation . If a memo is already discarded at branch retrieval, no downstream MMR can recover it. H. Context packing: inspect whether final packing pulls several overlapping passages from the same parent, exhausting token budget before an exception appears. This is downstream of retrieval; log both candidate and actually included context. I. Eligibility: verify the missing regional memo is active, authorized and effective for the query. Do not include a revoked or unrelated source simply to meet a “diversity” target. 3. Production metrics: distinguish duplicates, source diversity, and correctness Unique source ratio@k = number of distinct relevant/eligible source id in the final k ÷ k. With chunk based search, this is a diagnostic: it can be low on a correctly answered single source question. Duplicate passage rate@k = redundant or canonically equivalent passages beyond the first copy ÷ k. Define redundancy by normalized text/content hash and source/version/section identity; near duplicate detection uses bounded similarity and human reviewed thresholds. Largest source share@k = maximum count of passages from one source ÷ k. A high share can indicate domination, but apply query facet context before alerting. Evidence facet coverage = required relevant facets supported by at least one authorized chunk ÷ total required facets. Use complete evidence query rate where every required facet must be supported, not just mean facet recall. Source aware Recall@k: track whether authoritative relevant sources appear within candidate/final sets, separately from passage level Recall@k. Deduplicating many identical paragraphs may raise source coverage without changing vector similarity. MRR/nDCG and citation support: track relevance and quality simultaneously. Diversity that replaces the top authoritative evidence with irrelevant documents is not a win. Marginal context utility: after adding a passage, how much new evidence does it contribute, measured by an annotated facet or fact, not only its embedding distance? Latency/complexity: reranker input count, candidate memory overhead, multi source scoring, deduplication CPU, p95/p99, prompt tokens, generated wrong source citations and cross tenant leakage. 4. Numerical example: trace the stage that discards the memo For one query, dense top 10 includes 8 chunks from global policy G, 1 from FAQ F, 1 from exception memo E. BM25 top 10 includes 8 chunks from G and 2 from F, with E at BM25 rank 13 (outside the BM25 candidate window). Although E appears in the dense branch at rank 10 , final fused top 8 has 7 G chunks and 1 F chunk ; E disappears. Unique source ratio@8 = 2/8 = 25% . Largest source share = 7/8 = 87.5% . Required evidence facets are G: reimbursement cap , E: regional exception , and A: approval authority . The current final context supports only G, so facet coverage = 1/3 ≈ 33.3% . Another support chunk A was never retrieved within either branch's top 10. A simple RRF illustration with k=60: a G passage at dense rank 1 and BM25 rank 2 scores 1/(60+1)+1/(60+2) ≈ 0.03226. E at dense rank 10 and absent from the BM25 top 10 scores 1/(60+10) ≈ 0.01429. RRF naturally ranks the consensus G chunk higher; this is not a defect in the RRF formula. In actual systems, use the vendor's rank and window conventions. Controlled correction: expand each branch from top 10 to top 30, unify passage identities, preserve distinct answer bearing sections, perform source aware grouping and a measured relevance diversity rerank. The wider BM25 pool now contains E (rank 13), and a relevant approval chunk A appears at dense rank 18. Suppose final top 8 now contains 3 G, 2 E, 2 A, 1 F passages. Unique source ratio@8 = 4/8 = 50% and largest source share = 3/8 = 37.5% ; facet coverage = 3/3 = 100% . These values are illustrative , not observed production metrics, and selected chunks must be actual authorized evidence. On a held out 200 query set of multi source policy questions, imagine baseline complete evidence coverage 118/200 = 59% . After the tested change it becomes 170/200 = 85% , an absolute +26 percentage points . Rerank p95 rises from 130 ms to 205 ms , an additional 75 ms . Roll out only after assessing citation correctness, security and SLOs. Note: source diversity and facet coverage are not interchangeable . Four distinct sources can all repeat the same reimbursement cap; three chunks from one source can contain three different necessary rules. 5. Engineering remediation Canonical identity layer: distinguish point id, logical passage, source revision and canonical source. Derive stable content/version hashes, reconcile duplicate documents, preserve the current authoritative copy and respect tombstones/ACLs. Source aware fusion diagnostics: log rank positions, normalized scores and fusion contributions per candidate and branch ; make it possible to say “memo E was present in dense rank 10, then pruned after fusion.” Keep deterministic tie breaks where supported and pin all windows/configuration. Increase candidate coverage before diversification: widen BM25 and dense branch pools based on benchmarked recall, not arbitrarily. Elasticsearch RRF rank window size constrains which branch results reach fusion, and Qdrant hybrid prefetch limits are explicit. Deduplicate at the correct layer: collapse exact same passage across dense/BM25 branches as one candidate; use document/source canonicalization to identify near copy variants without collapsing genuinely complementary sections. Constrain source dominance when needed: after a sufficiently wide pool, use a tested source cap or group by with exception handling for one source questions. For multi facet tasks, prefer evidence coverage targets over a blind cap of one chunk per document. Diversify with MMR or other relevance diversity selection as supported by the backend or application. Elasticsearch's diversify retriever implements MMR over an inner result set. A high diversity setting can harm relevance, so benchmark across single source and multi source intents. Rerank remaining candidates against the original question and source effective constraints. Preserve source section and document version for citations, then pack a bounded evidence set by novel facet contribution. Validate backend semantics: Elasticsearch RRF supports collapse but not all retriever compositions or rescore operations; inspect your installed version and the supported order of grouping/reranking. In Weaviate, relative score and ranked fusion use different score treatments; pin the selected strategy. Deploy safely: use a feature flag, shadow comparisons, then a narrow canary. Gate on complete evidence query rate, source aware Recall@k, wrong citation rate, duplicate rate, p95/p99 and zero unauthorized sources. Retain a clean rollback path and consistent versioned index. 6. Trade offs and failure scenarios Ingestion deduplication vs retrieval time deduplication: fixing unstable source IDs reduces future storage and noise; retrieval time dedup still helps with overlapping chunks, mirrored content and dual branch appearances. Document level collapsing too early: may remove multiple answer bearing sections or cause outdated version selection. Use effective revision and section aware identity first. One source per result rule: improves surface diversity but can discard essential context when a policy has several independent rules. Cap intelligently or use facet aware selection. MMR: reduces redundancy but adds pairwise similarity computation and tuning; maximizing novelty alone can surface irrelevant sources. Wider candidates: improves opportunity to retrieve missing sources but raises memory, search latency and reranking costs. If E is not indexed or unauthorized, no branch window change can recover it. RRF consensus bias: a passage found in multiple branches is promoted; this is a useful signal in many cases, not an intrinsic bug. Relevance aware balancing and evidence coverage matter more than penalizing overlap automatically. Relative score fusion: can preserve score differences but needs comparable scale handling and is sensitive to branch score distributions; rank based methods have different failure modes. Rerank and context packing: a high quality reranker can still select repetitive evidence if its objective scores each chunk independently. Novel evidence rewards and citation constraints can help. Metrics and fairness: high source diversity isn't always beneficial; segment by task, authority, tenant and multi document need. Do not count unauthorized or obsolete sources as useful “diversity.” 7. Prevent recurrence Maintain gold queries that need (a) one authoritative source with multiple sections, (b) multiple sources to satisfy separate facets, (c) a regional exception overriding a global rule, and (d) exact version documents. Include mirrored copies, overlapping chunks and repeated ingestion. Instrument branch depth → unique passage count → unique source count → fused source share → reranker source share → final facet coverage , plus candidate removal reason codes. Alert when a source unexpectedly dominates multi source query classes or labelled exception evidence disappears. Add stable canonical IDs and content hashes to ingestion; audit new chunking/version strategies for changes in per source chunk count and skew. Compare RRF windows, branch limits, score fusion method, grouping order, deduplication thresholds and reranker settings as a single versioned retrieval manifest; reject changes that reduce labelled coverage or cite superseded policies. Run simulated duplicate amplification tests: insert 5–20 overlapping variants of one policy, then verify a known minority exception source remains retrievable and cited. Keep end to end authorization and policy effective version checks at both retrieval and citation time. A diversity feature must never include a source merely because it is novel when that source is forbidden or out of date. Review production incidents by which stage first eliminated a relevant distinct source , then fix the causal stage. Don't tune one score threshold blindly. Interview takeaway: When hybrid retrieval is dominated by repetitive passages, inspect branch candidates and source identities before and after fusion . Deduplicate equivalent passages, widen candidate pools if minority evidence was clipped, and select for relevant, authorized evidence coverage , using grouping or MMR carefully. Diversity is a means to complete, cited answers—not the goal itself. Authoritative references Elasticsearch, Reciprocal rank fusion and rank windows: https://www.elastic.co/docs/reference/elasticsearch/rest apis/reciprocal rank fusion Elasticsearch, Diversify retriever (MMR): https://www.elastic.co/docs/reference/elasticsearch/rest apis/retrievers/diversify retriever Qdrant, Hybrid queries and RRF/DBSF: https://qdrant.tech/documentation/hybrid queries/ Qdrant, Hybrid search and reranking: https://qdrant.tech/documentation/tutorials basics/reranking hybrid search/ Weaviate, Hybrid search and relative/ranked fusion: https://docs.weaviate.io/weaviate/concepts/search/hybrid search

A query planner routes questions to the wrong knowledge source. How would you identify routing failures and design fallback behavior?

Production incident (illustrative): An enterprise RAG assistant has four knowledge sources: HR POLICIES (leave, benefits and employee travel); FINANCE CONTRACTS (vendor payments and contractor reimbursements); SECURITY RUNBOOKS (access incidents); and PRODUCT DOCS (technical support). A user asks, “Who approves reimbursement for contractors travelling to Singapore under the 2026 vendor policy?” The query planner routes solely to HR POLICIES , because it matches “travel” and “reimbursement.” That index returns a credible employee travel policy excerpt, and the assistant confidently cites it. The authoritative answer is in FINANCE CONTRACTS , with a different approver. Source availability and HTTP success are normal: the incident is a wrong source routing decision masquerading as successful retrieval . Core principle: In multi source RAG, distinguish source selection from document retrieval within a source . Before tuning embeddings, ask: Did the planner consider the correct permitted source? Was it selected? Was a query actually sent? Did the correct source return authorized evidence? Was that evidence later dropped during fusion/reranking? A wrong answer may arise at any of these boundaries. A reliable fallback must preserve authorization and source authority , not just fetch more documents. 1. Contain the incident and establish ground truth Stop confidently answering policy questions using evidence from a source outside the requested domain. Require citations to an authorized, current, domain appropriate source ; if no such evidence exists, abstain or ask for disambiguation. Capture request trace: redacted user question, conversation context, tenant and principal, source registry version, source descriptions, allowed source list, generated query plan, selected sources, rejected sources/reasons, per source subqueries, tool arguments, responses, candidate IDs, reranker outputs and citations. Check source ownership with business SMEs. For the incident, vendor contractor reimbursements belong in FINANCE CONTRACTS; employee reimbursements belong in HR POLICIES. Source ownership is a maintained contract, not a guess inferred from a few overlapping keywords. Validate the target evidence in the authoritative source, its effective year, region, document version and ACL. Never debug by querying a forbidden tenant or by relaxing mandatory source permissions in production. Replay the same sanitized question with a frozen source registry, corpus snapshots and fixed user permissions. Verify whether the route changes with conversation history, source description edits, planner model versions or load. 2. Trace the first routing failure boundary A. Source registry: Enumerate permitted sources and their purpose, owner, freshness SLO, regions, document types, search capabilities, sensitivity tier and schema. Is the correct source registered, healthy and visible to the planner ? A disabled or stale source registry can force a misleading alternative. B. Source descriptions: Audit titles and retrieval/tool descriptions. Vague descriptions like “Search documents” cause overlap; specify unique data ownership (“Vendor contract terms, supplier invoices, contractor reimbursements; not employee HR travel policy”). Microsoft Azure Architecture guidance explicitly recommends precise tool descriptions and typed parameters. C. Intent extraction: Inspect planner interpretation of role, issuer, jurisdiction, date, named policy and required facts. “Contractor” must not be rewritten to “employee.” Test omitted chat history context, ambiguous acronyms, negation, polysemy, and the planner's confidence/calibration. D. Candidate source selection: Log the planner's eligible source candidates, ranks/scores when available, selected source set and budget. Check whether source top k=1 forces a choice between two relevant domains or whether hard coded precedence overrides the classifier. E. Tool invocation: A source may be selected but not queried because of a failed tool call, bad parameter, wrong index alias, missing tenant filter, timeout, rate limit, or query plan branch pruning. Selection logs alone are not proof of executed retrieval. F. Per source evidence: Inspect each source's returned IDs, scores, document versions, eligibility, citations and completeness flags. If the correct source is queried but yields no evidence, this is an in source retrieval/indexing issue , not necessarily a planner error. G. Merge/rerank: If the correct source returns evidence but it vanishes before the answer, investigate cross source score comparability, fusion windows, reranker input limits, source dominance, deduplication and context packing. Do not blame routing when the source was in fact queried successfully. H. Answer provenance: Check whether the final response cites the requested authority or merely a topically similar source. Validate domain ownership and source version requirements; citations alone do not prove the cited policy governs the user. 3. Root cause decision tree and controlled experiments Correct source not in allowed registry: configuration, permissions or registry freshness defect. Repair registration and authorization metadata; do not route around access control. Correct source allowed but not selected: routing classifier/LLM planner, vague descriptions, missing role/context, low source selection budget or over specific intent thresholds. Compare deterministic rules, descriptions and source ranking on labelled queries. Correct source selected but tool not called: planner to executor orchestration bug, wrong tool schema, timeout, routing guard or erroneous parallel call limit. Inspect execution activity and retry only when idempotent and safe. Correct source queried but no useful evidence: source specific search, index freshness, filters or query transformation issue. Run known ID and exact/lexical retrieval within that source; don't mask missing evidence by selecting an unrelated source. Correct source evidence retrieved but answer cites wrong source: fusion, rerank, source precedence, context assembly or grounded answer verification failure. Add domain authority constraints and citation checks. Router selects different sources for equivalent queries: test paraphrases against source labels; review non deterministic planning, changing registry snapshots, history and replica/tool availability separately. Query actually spans domains: use a bounded multi source plan with subqueries for each required fact. Prevent a single source planner from hiding cross domain requirements. Source data conflicts: rank by explicit business authority, applicability and effective version, not merely semantic score; surface unresolved conflicts rather than inventing a unified rule. 4. Worked numerical incident Assemble 500 expert labelled enterprise questions , with allowed sources and expected authoritative source labels. Assume 300 single source HR , 100 single source finance , 50 security and 50 genuinely cross source questions. Initially, the router correctly includes 450/500 = 90% of the needed source sets. However, a broad aggregate masks finance failures: for 100 finance only queries, it selects FINANCE CONTRACTS in 65 cases = 65% source selection recall ; it often picks HR POLICIES for contractor travel questions. For the failing question, a planner trace shows HR POLICIES selected at confidence 0.83 and FINANCE CONTRACTS at 0.69 , but source top k=1 drops finance. HR returns an employee policy with similarity 0.91 , which is irrelevant for a contractor . These scores are illustrative and not comparable to calibrated probabilities without validation. Expanding the authorized candidate source shortlist to 2 and adding a rule to keep FINANCE CONTRACTS eligible for vendor/contractor payment intent recovers the correct source in 94/100 finance only questions. A domain aware reranker and authority verifier retain the finance citation rather than the HR excerpt. On the same 500 question test, suppose complete required source recall rises from 450/500 = 90% to 485/500 = 97% , an absolute +7 percentage points . Finance only source recall rises from 65% to 94% ( +29 points ). Of the 15 remaining misses, classify failures by unregistered sources, ambiguous role terms, source timeouts and source specific retrieval, rather than calling all of them router mistakes. Average per request source calls increase from 1.1 to 1.6 , and retrieval p95 rises from 160 ms to 230 ms . These hypothetical numbers illustrate the quality versus latency trade off, not observed MachinoAI traffic. Validate source costs, permissions, accuracy and downstream citation support before rollout. Crucial distinction: a route recall gain does not prove final evidence recall or correctness; calculate source selection recall, retrieval success conditional on correct source , and authority supported answer rate separately. 5. Safe fallback design Primary route: parse request into a structured intent contract (domain, entity type, role, jurisdiction, effective date, document type, requires multiple sources) and enforce an authorized source allowlist from trusted policy. Bounded shortlist: where source ownership is ambiguous, select the top 2–3 permitted sources or route to specific rule based source sets. Use source descriptions/ownership to aid routing, not as a replacement for trust boundaries. Evidence sufficiency gate: evaluate whether returned evidence is current, authorized, domain applicable, supports the requested facts and passes an answer evidence threshold. A high vector score from the wrong domain should not satisfy the gate. Retry same source when indicated: when source selection is correct but retrieval is weak, first try a controlled lexical/hybrid query, validated rewrite or wider authorized candidate budget in the same authoritative source . A generic cross source fallback can conceal an indexing incident. Fallback to alternate allowed sources: if ownership is uncertain or the authoritative source is temporarily unavailable, query only sources explicitly permitted for this request. Mark alternate evidence as supporting , never silently promote it over a required legal/policy authority. Cross source questions: fan out bounded subqueries to sources that own the relevant facts, preserving tenant/ACL/date filters and merging evidence with provenance and authority precedence. Graceful abstention: if required authoritative data remains unavailable, state the limitation or offer the approved human escalation path. Never answer a finance policy from HR merely because the latter returns a confident passage. Circuit breakers and budgets: cap source calls, retries, total latency and token spend; distinguish timeout/no result/unauthorized/source not configured. Retries must not create runaway orchestration loops. Kill switch: maintain versioned source registry and router, route level feature flags and rollback to a tested policy. Verify rollback respects current deletions and permissions. 6. Engineering implementation and observability Maintain a versioned source registry describing business domain, owner, supported query classes, allowed principals, indexing/refresh SLA, authoritative precedence, source ID and schema. Include accurate negative descriptions (“does not govern contractors”) to reduce false route selection. Use typed tool schemas and explicit validation for query, filters, top k, source id and timeouts. Never let user provided natural language select internal source identifiers without authorization checks. Emit a trace per request: router model version, registry generation, selected sources, selection reason, eligible sources, source call status, source latency, source evidence ids, source authority, fallback trigger, fallback sources and final citation ids. Instrument source selection Recall@k, false route rate, wrong authority citation rate, required source coverage, fallback precision, retriever Recall@k conditional on correct route, p95/p99 and source tool error rate by domain, tenant and ambiguity class. Evaluate three separate layers: (1) route chosen; (2) evidence returned given route; (3) final grounded answer. A single end to end accuracy score cannot localize a failing planner. Build a held out labelled routing suite with overlapping language, multi source needs, negation, acronyms, contractor versus employee, sensitive domains, source outages, stale version and ambiguous user permissions. Use authoritative domain owner labels. Shadow test new router versions against the same authorized inputs and active source registry; then canary with strict wrong authority and unauthorized source guardrails. Compare source call cost, latency and final citation quality, not just classifier accuracy. Azure AI Search agentic retrieval documents knowledge source selection, query planning, execution activity and source references. Its source settings such as alwaysQuery are vendor specific: they can ensure particular sources are queried, but should not be configured globally for sensitive or irrelevant sources. 7. Trade offs and common mistakes Top 1 routing: cheapest and fastest, but brittle for overlapping enterprise domains or mixed intent. Top N can recover the authority at higher search cost and greater opportunity for noisy evidence. Always query policies: appropriate for a mandatory compliance authority when authorized, but expensive and possibly unsafe when applied indiscriminately. Source visibility and permissions still govern every call. Rule based routing: auditable for known IDs and clear policy domains, but needs continual maintenance as products and departments change. LLM/semantic planner: flexible for ambiguous natural language, but tool descriptions, context order, prompt drift and source registry updates change decisions. Use deterministic guardrails for mandatory source ownership and ACL. Fallback on zero results: may help when source availability is uncertain, but can convert a retrieval/indexing defect into a plausible wrong source answer. Prefer an evidence/authority aware fallback trigger. Confidence thresholds: planner scores may not be calibrated across source classes; test selective routing/abstention and per domain thresholds on representative data. Cross source aggregation: improves complete evidence coverage but introduces conflicting versions and scoring incomparable across sources. Rerank with source level provenance and explicit precedence. Source outage: an unauthorized or non authoritative alternative is not an acceptable fallback; show an appropriate limited service response. Version rollback: an old router may send traffic to retired indexes or sources. Roll back a consistent versioned configuration while retaining current security/retention rules. 8. Prevention checklist Treat each new source integration, source description update, tool schema change or source owner change as a routing release with labelled regressions and rollback. Add counterfactual route tests : “employee travel reimbursement” vs “contractor travel reimbursement,” “security access policy” vs “IT access troubleshooting,” and invoices vs vendor contracts; check that one word changes cause appropriate , not accidental, routing changes. Continuously reconcile source registry against actual available indexes/tool endpoints, latest document versions, ACL requirements, owner changes and source health. Alert on unknown source and failed route attempts. Monitor sudden wrong source citation spikes, large increases in fallbacks, source selection recall declines, tool timeouts and domain specific query shifts. Inject controlled failure scenarios: authoritative source returns zero results; planner source description is vague; a source times out; the same query needs both finance and legal facts; a user's permissions change mid session; a retired source still exists in the registry. Require an evidence sufficiency and authority gate before answer synthesis. If the required source is not reachable or evidence is incomplete, abstain and record the reason. Review source selection errors with source owners and make routing manifests, policy changes and fallback decisions auditable. Interview takeaway: To diagnose wrong source routing, trace allowed sources → planner choices → executed source calls → per source evidence → citation authority . Fix the first failing boundary . Design fallback as bounded, authorized, authority aware retrieval with abstention , not an unrestricted search across every database. Research references Microsoft, Develop an Agentic RAG Solution on Azure : https://learn.microsoft.com/en us/azure/architecture/ai ml/guide/rag/rag agentic Microsoft, Agentic Retrieval Overview : https://learn.microsoft.com/en us/azure/search/agentic retrieval overview Microsoft, What is a Knowledge Source? : https://learn.microsoft.com/en us/azure/search/agentic knowledge source overview Microsoft, Query Knowledge Base via API or MCP : https://learn.microsoft.com/en us/azure/search/agentic retrieval how to retrieve

The correct evidence was retrieved, but context trimming removed the sentence containing the answer. How would you identify and fix this?

Production incident (illustrative): A procurement RAG assistant is asked: “Under the 2026 contractor travel policy, who can approve an exception above the Singapore lodging cap?” Retrieval correctly returns section 4.6 of the current policy. The first two sentences describe the lodging limit; the third states “Exceptions above the cap require written approval from the Regional Finance Director before booking.” The retrieved chunk is 1,420 tokens, but the context assembler extracts only its first 900 tokens, discarding the approval sentence and its paragraph reference. The assistant cites the correct policy yet guesses that a line manager approves exceptions. Search Recall@10 looks healthy: the evidence was retrieved but not delivered to the model . Core principle: Treat retrieval and context assembly as separate correctness boundaries. A document in the retriever's candidate list does not imply the answer bearing span survived deduplication, compression, truncation, ordering, serialization, token budget enforcement, provider input limits, or final prompt construction. Investigate span survival before changing the embedding model, ANN index, top k or LLM prompt wording. 1. Contain the failure and freeze a trace For policy or financial decisions, do not allow an answer unsupported by the final model visible context. If the relevant sentence cannot be preserved under the budget, return an evidence insufficient response or use an approved evidence refetch path. Capture a permission protected request trace: query and principal/tenant, immutable source document/version, selected chunk IDs, search and reranker scores, original character/token offsets, raw retrieved text hashes, compressor outputs, context packing order, token counts per stage, final serialized model messages, exact model/version and provider truncation policy. Obtain the authoritative answer bearing sentence with its source span and active effective date. Validate that the user's ACL permits the full span. The key is not simply that the parent document is present. Replay the identical request with a frozen retrieval snapshot and deterministic packing parameters. Compare the actual model visible prompt bytes or a securely recorded hash/span inventory with candidate payloads. Avoid depending on a debug print of pre trim content. Separate this incident from later lessons: if the exact answer sentence is present in the final prompt but the model overlooks it, investigate position sensitivity/attention behavior ( 267); if present and cited but unsupported claims appear, inspect claim grounding ( 268). Here the sentence never reaches the model . 2. Instrument the evidence survival funnel Source truth: identify source id, document version, section id and the gold answer span [start offset,end offset); confirm the source is current, authorized and complete. Ingestion chunk: verify the answer span is included in the actual indexed chunk and that chunk offsets map correctly to original source text. The entire parent file existing is insufficient. Retriever output: inspect the raw text of the retrieved candidate at the retrieval API boundary. Compare chunk ID, source offset interval, text hash and gold span overlap. Reranker selected set: ensure the candidate remains after rerank and thresholding; a candidate dropped before the assembler is a candidate selection incident, not a trimming incident. Contextual compression: diff compressor input/output; inspect extractive selector, LLM summarization or text truncation, including whether the exception sentence and negation survived. A plausible summary that omits one approval requirement is unsafe. Deduplication and grouping: check whether overlaps, neighboring chunks, or multiple source versions are collapsed. Identity by document ID alone may accidentally discard distinct answer bearing sections. Token packer: enumerate each selected chunk's token cost, overhead for citations/source labels, reserved response tokens, conversation history, system instructions and examples. Record why and where each chunk or span was rejected : lower score, fit failure, maximum per document allocation, trim to N, dedup, or model hard limit. Final serialization/API: count tokens with the target model tokenizer and inspect the actual outgoing message list after SDK adapters, provider middlewares and any automatic truncation. Different tokenizers, message wrappers or hidden per field caps can produce an overflow even if application estimates pass. Answer/citation: verify that every asserted fact is supported by a model visible span, not just a matching source title or document reference. 3. Root cause debugging: find the first losing stage Gold sentence absent from indexed chunk: document extraction, OCR, chunking or versioning bug. Repair source to chunk mapping and reindex; expanding the inference context will not recover unindexed text. Present in indexed chunk but absent from retriever payload: query return field projection, text preview/length limit, metadata only retrieval or a mismatch between vector record and backing document store. Request the full authorized payload using stable IDs. Present after retrieval but absent after reranking: candidate cutoff or grouping bug. Revisit reranker top N and source/version aware grouping before changing context budget. Compressor drops the critical clause: overaggressive extractive threshold or lossy abstractive compression. Evaluate required span recall, reinsert verbatim evidence, or bypass compression for protected facts. Validate that summarization does not invert “must,” “unless,” or monetary/datetime qualifiers. Assembler slices text[:max chars] or tokens[:limit]: naive prefix trimming loses salient content near the end. Use a query aware, sentence boundary safe span selector and retain offsets/provenance. Packer removes lower ranked chunk that holds a necessary fact: a global greedy knapsack may favor redundant high score chunks; deduplicate repeats, reserve coverage for required facets, and re rank by evidence per token rather than raw similarity alone. Conversation history or examples crowd out evidence: enforce an explicit context budget and prioritize authorized answer bearing evidence. Summarize prior turns without dropping the active user constraints. Provider truncates after app packing: application tokens were counted with an inaccurate tokenizer or output reserve; target model maximum or an SDK setting differs from the environment. Fail closed on over budget rather than silently sending a clipped prompt. Gold sentence in final prompt but wrong answer: move to answer position/grounding diagnostics; no longer attribute the failure to trimming. 4. Beginner friendly numerical example Assume a model supports 8,192 total context tokens in the relevant request configuration. Allocate 900 system/instruction tokens , 650 conversation/query tokens , 900 reserved response tokens , and 200 serialized metadata/citation overhead tokens . The true retrieval context budget is 8192 − 900 − 650 − 900 − 200 = 5,542 tokens. First stage retrieval returns 10 chunks × 1,000 tokens = 10,000 tokens of raw content, including many duplicated summaries and one essential policy paragraph. The assembler naively packs the first five chunks (~5,000 tokens) and slices one remaining chunk to fit the last 542 token allowance. The exception sentence begins at token 730 of that sixth chunk, so it is not model visible , although its source document is marked “retrieved.” In a benchmark of 200 labelled policy questions , the answer bearing evidence occurs in the retrieved candidate pool for 184/200 = 92% . After naive context packing, only 132/200 = 66% final prompts contain all required answer spans. 52 questions lose evidence during assembly ; initial retriever Recall@k stays at 92%. A controlled remediation preserves the original relevant source and selects query relevant contiguous sentence spans with nearby qualification and approval sentences. After deduplicating overlapping summaries and reserving a 5,542 token budget for source labelled evidence, suppose 176/200 = 88% final prompts retain complete gold evidence. That is +22 percentage points of final prompt coverage versus 66%, while the 92% retrieval ceiling remains unchanged. At the illustrative answer stage, grounded correct answers rise from 124/200 = 62% to 168/200 = 84% , with no change to the embedding model. This does not mean the model always answers correctly whenever evidence survives; 176 evidence complete prompts versus 168 correct grounded answers leaves eight failures for other causes. Additional span extraction may increase p95 from 170 ms to 225 ms and add computation. Test whether 55 ms and extraction costs meet the SLO. Numbers are hypothetical, not live MachinoAI measurements. Treat a sentence's presence as verbatim exact normalized span containment or validated source offset overlap , not a fuzzy semantic match alone. Fuzzy matching can incorrectly declare a paraphrased but meaning changing summary “complete.” 5. Engineering solution Token budget contract: calculate available context = model context limit − instructions − query/history − serialization overhead − output reserve, with the tokenizer for the target model. Leave a safety margin for provider/tool wrappers. Treat a negative budget as a hard packing error, not a reason to silently trim. Span aware provenance: every selected text region carries source id, version, section, start/end offsets, content hash, token count, parent chunk id. Store a compact structured manifest of included and removed spans with reason codes; secure or minimize raw text traces. Answer span preservation: for evaluation cases, identify the gold sentence and its minimal essential surrounding context (e.g., approval authority, exceptions and effective date). In production, use a query aware evidence selector with abstention when evidence is uncertain; don't claim the system knows unseen gold spans perfectly. Contiguous neighbor expansion: if a relevant clause references “this exception,” include preceding policy definition, next sentence and footnotes as needed, not just one attractive isolated sentence. Bound expansion by verified source versions and token costs. Source aware deduplication: collapse exact overlapping portions but keep distinct approval conditions, exclusions and qualifiers. Deduplicate before aggressive packing to free tokens without information loss. Evidence per token optimization: compare constrained rerank/selection by expected information utility per token, with a minimum coverage requirement for each facet. Allow multi document coverage when needed. The 2026 controlled evidence fragmentation study found gains from contiguous span selection but also documented cases where it underperformed—benchmark rather than assume universal benefits. Compressor guardrails: when using an abstractive summary, keep the original source span retrievable and require entailment or literal fact checks for high stakes numeric, time and negation claims; on failure reinsert verbatim evidence. A second LLM summarizer is not inherently lossless. Pack then validate: construct the real model message object, tokenize it, and assert mandatory evidence manifest entries are present in the outgoing prompt. If too large, reduce redundant context, use parent/section refetch, stage the answer across batches, or safely decline. Citation verification: return citations tied to actual included spans and current source revisions. A title only citation to a retrieved but trimmed document must not count as supported evidence. Controlled rollout: shadow test packers on fixed authorized queries, then canary with checkpoints for gold span survival, complete evidence query rate, answer support, p95/p99 and token cost; keep a feature flag and rollback. 6. Practical failure scenarios and trade offs More context: may rescue removed sentences but raises cost/noise and can still exceed a hard limit. A larger context window does not excuse an incorrect packing policy. Smaller chunks: can prevent arbitrary mid sentence clipping but may split a qualification across chunks; preserve section/neighbor linkage. Longer chunks: retain neighboring context but compete harder for the budget and may be chopped by the assembler. Choose retrieval and delivery units independently when possible. Extractive compression: preserves exact wording for selected spans but can miss cross sentence references. Include neighboring clauses when required. Abstractive compression: saves tokens but risks dropping amounts, exclusions, approval authority and negative qualifiers. Preserve source and check fidelity. Greedy relevance first packing: prioritizes high scored text yet may repeatedly select the same source and drop minority facts; use diversity/facet coverage with relevance bounds. Token aware packing: safer than character counts but tokenizer versions and message wrappers can change; count the final serialized request. Preemptive truncation: avoids API errors but can turn a successful retrieval into a confident unsupported answer. Fail with explicit reason when critical evidence cannot fit. Full parent refetch: supplies missing context and offsets but adds network/IO and may surface superseded or unauthorized content without source version checks. Multiple inference passes: map reduce/iterative refinement retains more evidence with extra latency and intermediate error risks; compare to simpler span based packing first. 7. Monitoring and prevention Build a stage level evidence preservation dashboard : source present → indexed chunk contains span → retriever payload contains span → reranker keeps span → compressor preserves span → final prompt preserves span → answer supported by included span. Track final prompt evidence Recall@k, complete evidence query rate, gold span survival, truncation incidence, dedup loss, context tokens used versus budget, citation support rate , cost and p95/p99 separately from first stage retrieval Recall@k. Add tests with answer sentences at start, middle and end of a chunk; qualification one sentence after a boundary; adjacent document versions; duplicated boilerplate; changed conversation history size; and near limit token prompts. Record packing reason codes for each dropped chunk/span (not selected, duplicate, budget exceeded, per document cap, compressor removed, provider truncated) and reconcile manifests to the final serialized message. Canary every change to chat history summarization, tokenizers, model context limits, reranker depth, chunk size, context compressor or packing algorithm against an evidence complete labelled suite. Add a no grounded evidence → abstain or refetch guardrail. If a required span is absent from final context, the system must not claim the answer based on document title or latent model knowledge. Rehearse rollback while retaining active ACLs, deletions and source version rules; record both overall answer correctness and independently verified span provenance. Interview takeaway: When retrieval succeeds but the answer sentence disappears, diff the retriever's full payload against the final model visible prompt , attributing every discarded span to compression, deduplication, packing or provider truncation. Fix the first loss with token aware, span preserving selection and verify complete evidence in the outgoing prompt , not just retrieved document recall. Research references Microsoft Azure Architecture Center, RAG prompt engineering: token budgets, chunk selection, context overflow : https://learn.microsoft.com/en us/azure/architecture/ai ml/guide/rag/rag prompt engineering Microsoft Azure Architecture Center, RAG chunking phase : https://learn.microsoft.com/en us/azure/architecture/ai ml/guide/rag/rag chunking phase LangChain, Improving Document Retrieval with Contextual Compression : https://www.langchain.com/blog/improving document retrieval with contextual compression Aloteibi, Evidence Fragmentation Is a Distinct Failure Mode in Chunked Retrieval (Information 2026): https://www.mdpi.com/2078 2489/17/11/1129 Banerjee et al., The Bottleneck Is Retrieval, Not Reading (2026): https://arxiv.org/abs/2610.26951

The relevant evidence is in the middle of a long prompt, but the model repeatedly overlooks it. How would you test for position sensitivity?

Production incident (illustrative): A corporate travel RAG assistant correctly retrieves and includes the 2026 contractor policy paragraph stating that Singapore lodging exceptions require Regional Finance Director approval . The final model prompt contains 14 policy and FAQ passages totaling 11,600 tokens. The controlling paragraph is around token 6,100, surrounded by older employee travel FAQs. The assistant repeatedly answers “manager approval” despite a correct, current, authorized sentence in the model visible context. When engineers move the exact same paragraph to the beginning, answers improve. This is a candidate case of position sensitivity (“lost in the middle”) , not proof of bad embeddings or context trimming. Scientific grounding: Liu et al., Lost in the Middle: How Language Models Use Long Contexts (TACL 2024) showed that answer accuracy can depend strongly on where relevant text appears, commonly higher at the beginning or end of long contexts than in the middle. Later work investigated positional bias and evidence aware reordering. These are model and task dependent empirical effects , not a universal law that all models ignore their middle tokens. 1. Confirm this is a reading failure, not a retrieval/packing failure Check that the expected authoritative document was retrieved, remained eligible, survived reranking, survived compression and appears verbatim in the exact serialized prompt sent to the model . Use source IDs, version IDs, byte/token offsets and final message hashes. This distinguishes 267 from Q266, where trimming removed the answer sentence entirely. Preserve a verified answer bearing span with its surrounding qualification: approval role, exception scope, effective date and policy hierarchy. Ensure the span is actually applicable to the user and current for the relevant region and personnel category. Check whether the model is asked the same question each time, with the same system/developer instructions, chat history, generation settings, model release and tool responses. Otherwise apparent position sensitivity could be a changed task. Prevent unsupported high stakes answers by requiring model visible source support; when evidence is conflicting or missed, return uncertainty or retrieve focused current source snippets rather than guess. The prompt may contain a lot of data but still be within the model's published context window. Capacity to ingest N tokens does not guarantee equal attention or robust use of every position. 2. Design a controlled position swap experiment Construct 100–300 labelled questions with an authoritative answer span and several plausible distractor passages. Use an isolated, authorized fixed corpus snapshot and redact sensitive content from experiment logs. Fix the question, source passage text, distractor multiset, total prompt token count, model, decoding settings, instruction template and response format requirements . Only change the answer passage's position in the prompt. If a permutation changes total tokens because of headers or numbering, normalize wrappers and record actual token offsets. For each question, create placements at 0%, 25%, 50%, 75% and 95% of the retrieval context region (near start, early middle, center, late middle, near end). Each placement should preserve the answer sentence and enough neighboring text to remain interpretable. Randomize distractor ordering in a balanced way across placement conditions or use a Latin square/permutation design so a particular distractor is not always adjacent to the answer when it is centered. Keep the same authorized fact set and versions in all conditions. Repeat trials with different fixed seeds or multiple independent samples if the API is not deterministic. Even with temperature 0, server implementation or model updates may yield variation; log exact model identifier and evaluation period. Repeated runs also estimate noise. Counterbalance task semantics: include single fact lookup, exception application, multi document comparison, contradictory outdated policy, numeric extraction, exact document code and negative constraints. Test short vs long contexts with the same evidence to measure length position interaction. Use oracle placement tests with a gold passage deliberately inserted as a controlled benchmark, separately from normal retrieved RAG replay. The oracle test isolates reading ability; end to end RAG evaluation additionally includes retrieval and packing. When possible, run the same prompts across a second model to test whether the measured curve is specific to the current reader, not to all LLMs. 3. Collect useful measures rather than one accuracy number Position conditioned supported accuracy: correct and cited answers at position / evaluated questions at position. Grade source support separately from factual correctness. Middle penalty: accuracy at the average of beginning/end conditions minus accuracy at 50%. Report absolute percentage points and confidence intervals. Answer span recognition: does the model quote or reference the correct source span when asked to identify it? Compare with final policy decision accuracy to separate locating from reasoning over evidence. Wrong policy citation rate: incorrect evidence from outdated or inapplicable distractors; unsupported answer rate: claims not entailed by cited text; abstention rate: inability to answer when valid evidence is available. Facet/exception coverage: did the answer include required jurisdiction, role, date and exception approval? A response can name the approval officer but omit a critical eligibility condition. Token offset diagnostics: record absolute token position and relative position inside the retrieved content section, prompt length, source position/rank and distractor overlap; a passage at absolute 5,000 may be middle for 10k but near start for 40k. Statistical inference: because all positions use the same question, use paired bootstrap over question IDs or paired categorical tests, report confidence intervals and avoid treating 5 placements × repeated samples as fully independent. Failure localization: if the exact answer text is missing in one condition, this is prompt construction/truncation, not a pure positional effect. If a contradictory distractor moves relative to the answer, control for conflict and document authority. 4. Numerical worked example Test 200 labelled questions with a single gold paragraph and a fixed set of distractors, using identical 12,000 token retrieval sections and the same model. Every prompt includes the exact gold sentence at all placements. Suppose supported correct answers are: start 176/200 = 88% , 25% position 144/200 = 72% , middle 112/200 = 56% , 75% position 138/200 = 69% , end 170/200 = 85% . The mean edge accuracy is (88% + 85%)/2 = 86.5%. The middle penalty is 86.5% − 56% = 30.5 percentage points. The pattern resembles a U shaped lost in the middle curve; it does not show why a particular transformer behaves this way. In a control with a 2,000 token retrieval section, suppose start and middle are 90% and 86% . The much smaller 4 point penalty suggests a length × position interaction; use paired testing and uncertainty bounds before concluding. The original production prompt puts the crucial Regional Finance Director clause at the 50% position. Baseline on the 200 query set is 112/200 = 56% when gold is centered. A carefully tested ordering strategy that places the highest confidence, required, current evidence early yields 168/200 = 84% supported answers in a separate validation run ; absolute improvement 28 points . Do not claim this equals the 88% oracle start result: production evidence ranking and context effects differ. The same strategy increases average context packing work by 15 ms , and may not alter LLM inference cost if token count stays fixed. If it adds an extra reranker pass, overall p95 can increase; measure rather than assume. These figures are illustrative , not measured MachinoAI production data or a published paper result. A product decision needs its own task specific significance test, confidence interval, latency budget and no unsafe citation gate. 5. Root cause debugging decision tree Gold sentence absent at the final model input boundary: Q266 packing/trimming/serialization problem; fix span survival before testing position sensitivity. Gold sentence present, early and late succeed but middle fails, all else equal: credible position sensitivity. Compare context length, model revision, distractor placement and task complexity. All placements fail: wrong/ambiguous gold evidence, poor question formulation, model comprehension/grounding defect, conflicting authoritative sources or a broken instruction; moving context is unlikely to fix it. Only positions near conflicting old policies fail: distractor interference, source authority ambiguity or version sorting—not necessarily generic “middle blindness.” Preserve priority rules, annotate effective dates and test conflict controls. Long contexts fail everywhere but short ones pass: excess noise, answer context dilution, unsupported length scaling or truncation. Reduce/structure context or use query focused evidence extraction and compare across lengths. Correct span is recognized but final answer omits it: instruction following or multi step reasoning failure, perhaps a missed exception. Add explicit source grounding verification and inspect claim support, leading into 268 and 269. Repeated identical prompts produce different answers: assess stochasticity, server model changes, request routing or tool variability; don't attribute all variance to position. Answers cite the same document but wrong section: citation granularity and source version conflation; demand span level provenance. 6. Engineering solution and guarded rollout Evidence aware ordering: prioritize a small number of high confidence, current, authorized answer bearing passages near the start of the context and place complementary facts deliberately. Don't push all evidence to the ends without evaluating multi facet coverage and relevance. Query focused reduction: remove duplicated boilerplate and less useful distractors while retaining complete approval/exception qualifiers, citations and cross sentence definitions. A shorter faithful context may help more than clever reordering. Structured evidence envelope: make source ID, title, version, effective date, applicable role and relevant excerpts visually distinct; place explicit original user question near the evidence when helpful. Validate that structured formatting doesn't itself introduce measurement confounds. Span first retrieval: if current top chunks are long, refetch authorized relevant subsections with surrounding sentences and preserve source offsets. Avoid lossy summaries for legal, numeric or approval claims; keep the original span auditable. Two pass evidence check: in selected high risk cases, have a bounded first pass identify which cited spans answer each facet, then generate using those spans. Validate its extra latency and ensure first pass mistakes do not suppress required evidence. Position robust prompting: test prompts instructing the model to list supporting evidence IDs first, then produce an answer; do not assume these instructions eliminate underlying bias. Compare against a strong simple retrieval and rerank baseline before adopting a learned context planner. Citation and authority gate: verify the final answer's claims against included , current, authorized spans and permit abstention/refetch where support is missing. Never trust the model's stated citation without checking it. Deployment: shadow replay on balanced position tests, then canary with supported answer accuracy by position/length, critical exception omission, wrong policy citations, cost, p95/p99 and tenant/ACL guardrails. Feature flag the packer/orderer and retain reversible prior configuration. 7. Trade offs, experimental pitfalls and limits Reorder only: often cheap and reversible, but can move irrelevant content toward salient positions and may fail for tasks needing multiple distant facts. Shortening context: reduces distraction/cost but risks deleting qualifiers; verify exact relevant span survival separately, as Q266 establishes. Position conditioned benchmark: demonstrates sensitivity but not universal production gains. Crafted gold placements may be easier than real retrieval. Random permutations: test robustness and control confounds but expand experimental cost and complicate inference; use balanced assignments and paired statistics. Reranking: improves selection but may assign high scores to generic summaries or superseded policies. Include authority, coverage and provenance criteria. Learned context planning: extra models and training may add complexity without beating strong simple retrieval; use held out, leakage free evaluations. Alternative model: one long context model may be less position sensitive than another, but pricing, latency, source grounding and data governance constraints can outweigh the gain. Near end placement: can improve access to a fact but risks placing essential content after instructions whose priority is unclear; always respect system and security instructions. Large context window: a model supporting 100k+ tokens does not guarantee it uses centered evidence equally well across tasks. Model updates: position profiles may change when provider models or tokenizer behavior changes; rebaseline after upgrades. 8. Prevention checklist Add a balanced position swap regression suite with gold facts at beginning, 25%, center, 75% and end, across short/medium/long contexts and known authorized source versions. Track supported answer accuracy, middle position penalty with uncertainty, first stage retrieval coverage, final prompt span presence, wrong citation rate, exception omission and per position context length. Require final prompt manifests with source IDs, section offsets and relative positions; log exact model/config versions in secure traces so shifts are diagnosable. Include adversarial distractors: obsolete policies, near identical region rules, multi step approvals, contradictory source revisions and repeated boilerplate. Verify the correct version outranks inapplicable text. Check both single source and multi document questions; prioritizing one gold span can harm questions needing several distinct documents. Alert on new position gaps after context packer, reranker, source ordering, model, summarizer or context budget changes; use offline position controls and online citation review. Audit security and freshness regardless of ordering: moving an unauthorized or superseded paragraph to the top is a critical failure, not an accuracy improvement. Interview takeaway: First verify the answer sentence is inside the exact model visible prompt . Then perform a controlled position swap experiment that changes only where the same authorized evidence appears while keeping the question, distractors, tokens, model and settings fixed. Measure supported accuracy by position, confirm a statistically credible middle penalty, and test evidence aware reordering or shorter faithful context with citation and latency guardrails. References Liu et al. (2024), Lost in the Middle: How Language Models Use Long Contexts , TACL: https://aclanthology.org/2024.tacl 1.9/ Hsieh et al. (2024), Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization : https://aclanthology.org/2024.findings acl.890/ Boytsov et al. (2025), Positional Bias in Long Document Ranking : https://aclanthology.org/2025.ijcnlp long.46/ Yi et al. (2026), Attention Basin: Why Contextual Position Matters in Large Language Models : https://aclanthology.org/2026.acl long.1198/ Li and Chen (2026), When Learned Context Planning Fails to Beat Strong Retrieval : https://aclanthology.org/2026.insights 1.11/

Your assistant cites the correct document but makes a claim that the cited passage does not support. How would you detect this automatically?

Production incident (illustrative): A company procurement assistant answers, “For Singapore contractors, expenses above SGD 500 automatically qualify for finance approval within two days,” citing the correct 2026 Contractor Expense Policy. But the cited paragraph says “Claims above SGD 500 must be reviewed by the regional finance team; no automatic approval or fixed turnaround time is guaranteed.” The source document is correct, current, authorized and present in the prompt. The generated answer changes “review” to “automatically approved,” invents a two day deadline, and drops an explicit denial. A document level citation check passes while a claim to evidence entailment check would fail. Key principle: A citation is an address, not proof. Distinguish citation validity (document/version exists and is authorized), passage applicability (the cited span governs this entity/date/region), and claim support (the exact cited span logically entails the statement in the response). A claim can be factually plausible—or even true in some other source—yet still be unsupported by its attached citation . Detect this at the smallest meaningful claim/span level, not only the full answer or document level. 1. Contain the incident and establish a reliable evidence trace For consequential finance, legal, safety or policy answers, gate unsupported factual claims before release. Either regenerate using verified evidence, remove the unsupported statement, or explicitly abstain; never retain a confident wrong citation because the document title looks relevant. Trace the user question, tenant/principal, active document version, retrieval snapshot, final model visible spans, generated answer, citation markers and the exact passage text referenced by each citation . Record canonical source ID, version, section, offsets, content hash and effective date. Confirm retrieval and packing did not lose the clause. Q266 covers lost spans during context trimming; Q267 covers position effects despite spans being present. Here the supporting source is available, but the answer overclaims or contradicts the cited sentence. Validate that the citation resolver maps markers to the correct chunk and current version. A citation to a file's landing page or title is not enough for automatic support verification. Store only necessary, access controlled evaluator inputs; document text and user questions may contain confidential material. Grounding checks must reuse the user's authorized evidence set, never search across another tenant to validate a claim. 2. Decompose the answer into atomic, testable claims Split compound sentences into discrete propositions without losing subjects, negation, modalities, quantities, and conditions. Example answer: “Claims over SGD 500 are automatically approved within two days.” Atomic claim A: Claims over SGD 500 are automatically approved . Atomic claim B: Approval is completed within two days . Optionally distinguish claim C (threshold SGD 500) from A because the source supports the threshold while denying automatic approval. Normalize referents using local question context: “they” means Singapore contractors ; “it” means a reimbursement claim. Preserve identity, time range, jurisdiction and policy edition in each structured claim. Keep attribution: link each claim to the citation(s) attached to its original sentence; if citations are only listed at paragraph end, define a consistent conservative citation scope and mark ambiguous mapping for review. Exclude purely subjective style statements from factual groundedness denominators, but include legal/financial instructions and all implied factual commitments. Avoid artificially inflating faithfulness by splitting “SGD 500” into trivial supported fragments. Treat quoted text specially: verify byte/normalized text agreement and source offsets for direct quotations, alongside entailment of surrounding interpretation. Exact quote matching alone does not validate an inference. Candidate claim schema: claim id, answer sentence id, subject, predicate, object, qualifiers, time, region, modality, negation, citation ids. Keep stable identifiers for regression evaluation and audit. 3. Resolve exact citation spans before judging entailment Use the same model visible, authorized citation evidence attached to the response, not all text in the source document. This distinguishes “somewhere in the document” from “supported where cited.” Resolve each citation to source id + version + section + [start,end) and verify the text hash against the evidence in the final prompt. Reject broken links, mismatched revisions and stale/unauthorized spans as invalid citation before attempting semantic support. Fetch a small bounded adjacent window only when the cited sentence refers to definitions, exceptions or tables immediately around it; maintain original provenance and permissions. Report separately whether the precise cited window versus expanded validated window supports the claim. For multi hop or multi citation claims, allow a bounded evidence set if the inference requires combining two passages. Record which claim is supported by which citations, and flag missing prerequisites. Do not declare an unsupported claim “supported” merely because some unrelated passage in the full retrieved corpus could conceivably support it. Check that evidence applies to the user's role, geography, date, product version and scope; an expired employee policy cannot support a current contractor instruction, even when wording is similar. 4. Automatic claim to evidence classification For each claim, cited span, question context pair run an evidence conditioned NLI or LLM as judge check with three labels : SUPPORTED (entailed), CONTRADICTED (the evidence entails its negation), and INSUFFICIENT EVIDENCE (neither support nor contradiction). A fourth INVALID CITATION status comes from provenance resolution. Force judges to output minimal quoted supporting/conflicting evidence , exact offsets, preserved qualifiers and a structured label. Verify judge provided quotes against the span rather than trusting fabricated quotations. Require conservative treatment of quantified and modal terms: “may,” “must,” “can request review,” “automatically approved,” “within,” “at least,” “up to,” “unless,” and “not” are materially different commitments. Use deterministic validation where possible: numeric threshold comparisons, ISO dates, currency/units, exact product codes, status enums, source version, and quote substrings. A semantic judge should not override a clearly conflicting structured value. For claim A, the policy says “must be reviewed” and “no automatic approval” . The proper classification is CONTRADICTED , not supported just because “SGD 500” and “finance” appear in the excerpt. For claim B, “no fixed turnaround time is guaranteed” contradicts a guaranteed two day turnaround. If the cited passage simply omitted timing entirely, B would be INSUFFICIENT EVIDENCE , not contradicted. The distinction matters for remediation. Never equate embedding cosine similarity or lexical overlap to logical entailment; two contradictory sentences can be lexically and semantically similar. A groundedness score at whole answer level can mask one dangerous unsupported assertion. Use independent human calibrated evaluation labels for ambiguous claims. LLM judges can be overconfident, sensitive to wording, or wrong about negation and arithmetic; do not describe an automated judge as perfect proof. 5. Root cause debugging decision tree Citation marker resolves to an old/incorrect passage: citation formatter, source version resolver, document relocation or span offset bug. Repair provenance first, not prompting. Marker resolves correctly but a key exception sentence was omitted from the final prompt: Q266 context loss; fix delivery. Claim supported elsewhere in the prompt, not by attached citation: citation to claim alignment defect. Move/add the correct precise citation only after verifying all constraints; otherwise remove the unsupported citation. Source and cited passage are correct but claim contradicts them: generation/reasoning issue, negation/modality misread, overgeneralization or blended memory. Reject/regenerate with exact span and stricter claim validation. Evidence is silent about claimed timing, quantity or authority: unsupported extrapolation. Seek an additional authorized authoritative source or express uncertainty; do not invent details. Claim combines facts from two policies without applicable join: multi source reasoning and source authority failure. Confirm role, effective dates and policy precedence. Grounding judge reports failure on a true paraphrase: evaluation false positive; audit the judge rubric, necessary preceding definitions and local context. Re evaluate using adjudicated examples. Judge reports support for a wrong number or negation: evaluation false negative; add structured equality/range tests and adversarial negative examples. Answer contains several facts but only one is unsupported: revise that claim rather than marking every supported claim invalid. However, for high stakes operational actions a single critical unsupported claim may require withholding the entire answer. 6. Worked numerical production example Evaluate 200 answers containing 1,000 human annotated factual claims . Gold labels: 850 supported , 90 contradicted , 60 insufficient evidence , so 150 genuinely unsupported or contradicted in total. Assume citation IDs and source versions are valid for this controlled test. Before the verifier, all 200 answers are returned. Suppose only 126/200 = 63% are completely claim supported; this answer level rate is a separate annotation and must not be inferred from claim counts alone. An automated claim checker flags 135 of the 150 truly non supported claims ( recall = 135/150 = 90% ) and incorrectly flags 34 of the 850 supported claims ( false positive rate = 34/850 = 4% ). Its precision on flagged claims is 135/(135+34) = 135/169 ≈ 79.9%; 15 truly unsupported claims remain unflagged. These numbers show why automatic detection should not be sold as perfect safety assurance. In a controlled remediation, take 169 flagged claims for targeted removal, regeneration with cited span, or human review depending on risk. Assume 120 of the 135 true positives are successfully repaired without introducing new errors. On the same 1,000 claim set, genuine unsupported claims fall from 150 to 30 (150 120=30), or 15.0% to 3.0% . Of the remaining 30, 15 were missed and 15 flagged but unrepaired . The 34 false positives should not be automatically deleted as “false facts”; route ambiguous or business critical ones to review. Suppose rerun human grading confirms 178/200 = 89% fully supported answers, versus 126/200 = 63% baseline. That is a +26 point illustrative improvement, not an algebraic consequence of the claim repairs. A claim verifier adds p95 180 ms → 310 ms in this hypothetical setup, plus evaluation tokens. For a high volume low risk assistant, use risk tiered checks; for approval policy guidance, stricter verification may justify latency. These counts are illustrative , not live MachinoAI measurements. Real deployment must separately calibrate detection recall, precision, changes in answer completeness, abstentions, costs and customer outcomes. 7. Engineering solution and runtime workflow A. Produce citeable spans: attach stable source/version/section offset IDs to the exact retrieved text used in the final prompt. Do not let the model invent source IDs or cite unseen document passages. B. Generate structured answer and provenance: request per sentence citation IDs, then post process claims and map them to supporting spans. A citation is only accepted after a trusted resolver validates its source. C. Decompose factual claims: use a lightweight constrained extractor, version its prompt/model, and test that no material assertion is lost or split incorrectly. Ensure negation, numerical comparisons and qualifiers survive decomposition. D. Verify claims: run deterministic rule checks for exact facts and an NLI/LLM judge for semantic entailment, including labels and source offsets. Where multiple citations are required, check the bounded combined evidence and record each bridge. E. Make risk aware decisions: supported claims pass; contradicted claims block; insufficient claims trigger authorized evidence refetch, clarification, conservative rewrite or abstention; invalid citations trigger source resolution repair. Escalate disputed high impact facts. F. Repair safely: regenerate only affected spans with the original verified text, then re verify repaired claims. Avoid infinite regeneration loops by limiting retries and refusing unsupported claims when a bounded repair fails. G. Log auditable artifacts: claim ID, original answer span, source passage hash/version, verifier model/version, label, measured confidence (if calibrated), deterministic checks, action and eventual human adjudication. Protect sensitive inputs and maintain retention limits. H. Canary with labelled evidence: compare claim level support, contradiction escapes, false positive rate, valid citation rate, whole answer support, missing critical fact rate, abstention, latency and cost. A system can become more “faithful” merely by answering nothing, so maintain answer utility and completeness measures independently. I. Handle policy change: invalidate/re check cached source mappings and answers when citation source versions, ACLs or document effective dates change. Grounding against stale content is not sufficient. 8. Evaluation design, trade offs and failure scenarios Groundedness vs factual truth: “true on the internet” is not equivalent to “entailed by this cited source.” The verifier is a citation support guard, not a universal truth oracle. Precision vs recall: aggressive blocking catches more unsupported claims but can suppress correct paraphrases; permissive verification improves coverage but lets harmful claims through. Set thresholds by the cost of false approvals, not a single global target. Claim splitting: smaller atomic units improve localization but can erase conditions if extraction drops a shared negation or temporal scope. Keep a structured qualifier ledger and evaluate decomposition quality. Passage level vs document level: broad document text may contain contradictory revisions or exceptions, making a wrong citation seem acceptable. Prefer the actual cited span and explicit version authority. Multi hop evidence: some claims require combining independent excerpts; scoring one passage alone can yield false unsupported labels. Bound permitted support sets to authoritative cited material. NLI/LLM judges: scalable but imperfect on numbers, modality, conditional logic, tables, multilingual text and domain jargon. Use calibrated human labels, deterministic guards and regression suites. Verifier feedback loops: if the same model generates and judges, shared biases may cause correlated errors. Compare independent verifier models and periodic human adjudication for critical slices. Self repair: can improve support but also remove necessary qualifications or invent new citations. Always re run support checks and preserve evidence trace. Abstention: necessary when the citation cannot substantiate a claim, but excessive abstention harms helpfulness; monitor supported answer completeness separately. Conflicting sources: selecting any text that entails the claim is unsafe when an authoritative current policy contradicts it. Enforce temporal, jurisdictional and document precedence. 9. Prevention checklist Maintain a golden claim evidence dataset with entailed, contradicted, insufficient and invalid citation examples. Include "same document but wrong paragraph" and "right quote but wrong effective date." Add adversarial transformations: reverse a negation, change 500→5,000, “may”→“must,” “review”→“automatically approve,” swap entities, change two days to ten, truncate an exception, and attach a real but irrelevant citation. Evaluate claim level unsupported rate, contradiction escape rate, supported citation precision, whole answer all claims supported rate , citation resolution, false positive rate, missed critical claims, completeness, costs and p95/p99. Segment by model, document version, answer length, citations per claim, domain, language, tables, multi document questions and sensitive actions. Alert on increases after prompt, reranker, packing or citation format changes. Audit the citation resolver separately from semantic verification. Ensure relocated files preserve stable IDs, spans, current ACLs and source versions. For high risk answers, sample daily human audits and feed adjudicated failure cases into the regression suite. Recheck labels after source updates rather than treating old judgments as timeless ground truth. Fail closed when a cited source is forbidden or missing; never widen retrieval across tenants to make a claim appear supported. Require a bounded regeneration budget and clear rollback criteria if an updated verifier increases false blocks, omissions or latency. Interview takeaway: Detect “right document, wrong claim” by linking each atomic claim to the exact cited, authorized passage and checking entailment, contradiction or insufficient evidence . Validate citation provenance first, protect numeric/negative qualifiers, measure verifier precision/recall on human labels, and reverify every repaired claim. A clickable citation is not evidence that the claim is grounded . Research references Es et al., RAGAS: Automated Evaluation of Retrieval Augmented Generation (EACL 2024): https://aclanthology.org/2024.eacl demo.16/ Min et al., FActScore: Fine grained Atomic Evaluation of Factual Precision in Long Form Text Generation (EMNLP 2023): https://aclanthology.org/2023.emnlp main.741/ Microsoft Foundry, RAG evaluators: https://learn.microsoft.com/en us/azure/foundry/concepts/evaluation evaluators/rag evaluators Microsoft Azure, Large Language Model End to End Evaluation Phase : https://learn.microsoft.com/en us/azure/architecture/ai ml/guide/rag/rag llm evaluation phase

A retrieved clause states an exception, but the generated response mentions only the general rule. How would you prevent missing qualifications?

Production incident (illustrative): An employee asks, “Can a Singapore contractor claim a hotel above the SGD 350 nightly cap for a conference?” The assistant retrieves the current policy and confidently answers: “Contractors may claim accommodation only up to SGD 350 per night.” It cites the right 2026 policy. But the same authorized, model visible clause says: “For a pre approved accessibility requirement, the cap may be exceeded with written Regional Finance Director authorization before booking.” The user has stated an accessibility related need. The assistant’s general rule is literally supported yet its response omits an applicable exception , potentially causing an incorrect denial. Key distinction: Faithfulness is not sufficient for completeness. The included sentence may be true, but the answer as a whole can misrepresent the policy by omitting a condition, exception, exclusion, negation, override, effective date restriction or approval path. Microsoft’s RAG evaluation guidance distinguishes groundedness (claims do not exceed the context) from response completeness (critical expected information is not missing). Evaluate whether a qualified decision rule was represented, not just whether each stated claim sounds supported. 1. Contain the incident and verify what the model actually saw Check the user question, entity/role, jurisdiction, date, requested action, source versions and authorization. Is the exception actually applicable, or is the user asking only for the default cap? Do not append every remote exception to every response. Trace the full evidence funnel: authoritative document → indexed chunk → retrieved evidence → reranker → compressor → final serialized prompt → answer and citations. Record the exact paragraph text with source/version/offset and each output stage’s hash. If the exception was dropped during compression or context packing, it is a Q266 context survival defect. If the exception survives in the final prompt but the model ignores it, this is Q269 answer qualification omission . If the model makes an unsupported claim, refer to Q268’s claim level citation checks. Establish authoritative precedence: which rule and exception are current; whether the exception overrides the limit or merely provides a process to request review; whether approval is prior to booking. A wrong exception can be more harmful than no exception. In consequential policy guidance, disable unconditional “never allowed” conclusions until applicability is checked. Return an uncertainty statement or escalate if controlling clauses cannot be reconciled. 2. Extract a structured rule plus exception contract Represent a rule as rule id, source id, version, scope, condition, default effect, exception id, exception condition, exception effect, required action, approving authority, effective period and precedence. Example default: role=contractor AND location=Singapore → hotel cap SGD350/night. Exception: preapproved accessibility need=true AND written Regional Finance Director approval before booking=true → may exceed cap. This is permission conditional on approval , not automatic entitlement. Make the condition scopes explicit. “Except in emergencies” may modify a time deadline rather than an amount cap; grammar and section layout matter. Preserve conditional verbs such as must , may , unless , only if , except , subject to , notwithstanding and provided that . Extract source sentence offsets for default and exception, plus neighboring definitions, headings, footnotes and referenced sections. When the exception refers to “the above threshold,” include enough nearby text to resolve the reference. Avoid blanket exceptions based on similar words from obsolete policy versions, another tenant, employee rather than contractor provisions, or an entirely different region. Treat structured extraction as a hypothesis: validate important legal/finance clauses with a deterministic field matcher or a calibrated independent reviewer, and record ambiguous cases for human policy ownership. 3. Root cause debugging sequence A. Evidence eligibility: confirm the exception clause is in an authorized, active, governing source and relates to the requested user. A legitimate exclusion by jurisdiction or date is not an omission bug. B. Context preservation: verify default and exception spans survive search, deduplication, rerank, compression and packing. A high scoring general policy summary may crowd out the only exception sentence. C. Source segmentation: check whether chunk boundaries separate “General rule” from “Exceptions,” whether a table footnote was dropped or a section heading misclassified. Index section/parent linkage and retrieve exception neighbors when needed. D. Intent binding: compare user attributes and explicit qualifiers to rule preconditions. The planner may identify a contractor hotel cap question but not the stated accessibility requirement. E. Retrieval/selection bias: inspect whether a short generic FAQ outranks the detailed exception. The answer model may prefer prominent, repetitive general rules; check source authority and facet level coverage. F. Generated answer audit: extract the answer’s unconditional claims and compare to rule schema. “The cap is always SGD350” conflicts with a possible approved exception; “Normally SGD350, with a documented accessibility exception subject to prior written approval” preserves the decision structure. G. Complementary omission evaluation: distinguish an answer that is supported but incomplete from one that contradicts an exception by saying “never.” Standard claim entailment precision tests may pass the former; add a required fact recall and condition coverage check. H. Repair loop: if the default is present but an applicable qualifier is omitted, regenerate with the validated default+exception spans and explicit conditional structure. Recheck both groundedness and completeness, then enforce a bounded retry limit. I. Later stage issue: if the exception is represented but combined with another person's facts or an incompatible reporting period, investigate entity/temporal consistency (Q270–Q271) rather than simply changing sentence order. 4. Engineer a qualification aware answer pipeline Rule graph/ledger: for higher risk policy sources, extract or author rule groups linking default → exception → prerequisite → approval → effective version. Preserve original text and provenance; use source owners to approve changes when necessary. Query to rule matching: derive trusted user constraints (role, region, effective date, purpose) from the question and permitted profile metadata. Classify as default only, exception potentially applicable, exception confirmed or insufficient attributes; avoid inferring sensitive conditions not stated. Evidence assembly: fetch default and candidate exception clauses together, including necessary definitions and relevant footnotes, while retaining source offsets. Use a bounded neighbor expansion instead of adding an entire unrelated policy. Structured generation: ask the answerer to complete default rule, applicability, exception, approval requirement, uncertainty, citations as internal checked fields before producing concise prose. The user facing response can stay short, but the decision must not silently discard controlling qualifiers. Dual validation: (1) groundedness/claim support—every expressed fact is backed by exact model visible cited spans; (2) completeness/qualifier recall —each applicable material condition in the evidence ledger is represented in the answer or explicitly marked unknown. Logic guards: deterministic checks for critical numbers, inequality signs, dates, approvers, policy versions, and strong absolute words (“always,” “never,” “automatically”). Mark a violation when an unconditional conclusion is generated while a governing exception exists. Abstention and clarification: if the exception requires a fact not supplied (e.g. pre approval already granted), say that the outcome depends on that fact; do not assume approval, deny it categorically or invent eligibility. Authority conflicts: where a global rule and local exception differ, apply documented precedence and effective dates rather than LLM preference for one text. If authority is unclear, flag human review. Safety controls: all source/neighbor fetches obey tenant, ACL, redaction, version and time restrictions. Do not show exceptions from protected documents solely because they exist in the same corpus. 5. Numerical example: why a faithful answer may still be wrong Build 200 expert labelled policy questions with verified active evidence. For 120 cases a general rule suffices; for 80 cases an exception or qualification is material to the question. Labels include exact default, applicable condition, override, required approval and citation. At baseline, suppose the response states a supported general rule in 188/200 = 94% cases. A naive “is the answer supported by the cited paragraph?” check reports high faithfulness, but only 28 of 80 exception dependent answers include the controlling qualification: qualification recall = 28/80 = 35% . Complete and legally applicable answers total 120 default only + 28 exception covered = 148/200 = 74% . The remaining 52 exception cases may be superficially grounded but still incomplete. This arithmetic assumes all 120 default only answers are correct in the controlled example. Add source linked exception extraction, query applicability matching and a final required qualifier coverage gate . On the same 80 cases, suppose 68 include the correct conditional exception and prior approval requirement: 68/80 = 85% qualifier recall . If the 120 default only answers remain correct, complete applicable answers become 120 + 68 = 188/200 = 94% , a 20 percentage point gain in complete answer rate versus 74%. Ten additional errors remain for specific root cause review. The extra gating pass adds a hypothetical p95 165 ms → 260 ms (+95 ms), and 14 of 200 initial answers may request clarification or be conservatively abstained where approval facts are unknown. Those abstentions must be reflected separately in answer utility evaluation rather than counted automatically as successes. All figures are illustrative , not measured MachinoAI operational metrics or claims from the cited research. Human adjudication and a carefully controlled holdout are needed before setting a deployment threshold. 6. Evaluation metrics and test design Applicable qualification recall: number of required, governing qualifiers represented correctly / number of required governing qualifiers across labelled questions. Evaluate applicability , not just mention counts. Complete answer rate: fraction of questions whose answers include the default and all material conditions and are valid for user scope. Record wrong effective date and wrong policy outcomes separately. False exception inclusion: percentage of answers incorrectly adding an exception that does not apply. Optimizing only exception recall can produce irrelevant or dangerous overqualification. Unconditional overclaim rate: outputs using “always/never/automatically” when the governing policy includes an applicable exception. Evidence coverage by stage: default+exception available at retrieval, present after context packing, identified in the rule ledger, then reflected in the final answer. Groundedness and citation support: verify included conditions are cited to the governing text; a complete sounding answer can still hallucinate policy details. Decision accuracy and clarification: evaluate whether the resulting action is appropriate, and whether the assistant asks for missing facts rather than assuming them. Use paired tests for policy rules with/without an exception, cases where a qualifier changes the outcome, negations, overlapping jurisdictions, dates, permission versus automatic entitlement, and user questions intentionally asking only for a simple baseline. Run separate source verification and output verification gold sets. The same model extracting, generating and grading may share mistakes; independently adjudicate a risk weighted sample. 7. Engineering trade offs and limitations Show every exception: technically easy but bloats answers, adds irrelevant caveats and may confuse users. Include only material or plausibly applicable qualifiers, with optional deeper details. Short answers: better UX but more likely to omit caveats. A concise two sentence “normally X; if Y, then Z with approval” can be both short and complete. Full document context: could contain all exceptions but increases noise and token costs. A bounded linked rule extraction is more targeted, provided offsets and scope remain correct. LLM rule extraction: flexible for prose and footnotes but imperfect for nested conditions, tables and modality. Validate high risk policies with rule ownership and exact text. Rigid templates: enforce coverage but may sound repetitive and fail unusual document structures; render natural language after the structured facts are verified. Prior approval: “may exceed with approval” does not mean “automatically reimbursable”; guard against modality inversion. Omitted versus contradicted: an incomplete general rule answer and an unconditional “never allowed” answer have different severity. Metrics and remediation should reflect that. Multiple exceptions: an exception can have sub exceptions or override conditions. Preserve a precedence chain and avoid flat keyword lists. Policy drift: newly added exceptions can change previously correct answers even if retrieval of default rules is stable; version and re evaluate the rule ledger. Evaluation bias: synthetic questions may make exceptions more obvious than real user queries. Include actual anonymized phrasing, reviewer validated hard negatives and ambiguous cases. Latency/cost: extraction, rule checks, and regeneration add work; tier by consequential decisions and measure p95/p99 plus abstention burden. 8. Prevention and production runbook Every indexed policy update gets a default/exception/condition audit . Changes in caps, prerequisites, approvals, negation or effective dates invalidate old rule schemas and cached answers. Maintain a clause aware regression suite: “usually” vs “always”; “unless” vs “except”; “may request approval” vs “approved”; boundary amounts, pre /post booking dates, missing permissions, wrong regions and retired versions. Emit secure trace fields: source version, rule id, exception id, default span id, qualifier span id, qualification applicable, qualifier in prompt, qualifier in answer, judge verdict, citation ids, and any clarification. Alert on a growing retrieved qualifier → omitted answer gap, increased unconditional overclaim rate, critical wrong policy decisions or new extraction errors after ingestion/prompt/model changes. Test at multiple levels: source extraction; retrieval of linked clauses; context budget survival; conditional inference; final answer completeness and citation verification. Canary rule aware generation on real production like query slices and evaluate complete answer accuracy, qualifier recall, false exceptions, abstention, latency and cost before rollout. Preserve rollback and current ACL enforcement. Require SME adjudication for critical policy families and disagreements between a general rule and purported local exception. When source precedence is unknowable, show uncertainty rather than silently choosing one. Interview takeaway: A RAG answer may be entirely faithful to the words it repeats yet still misleading because it omitted a controlling exception. Build source linked rule/exception extraction , test whether the exception applies to the specific query, and gate outputs on both claim support and required qualification recall . If the clause is in the prompt but absent from the answer, fix conditional reasoning and completeness—not vector search. Research references Microsoft Foundry, RAG evaluators (groundedness vs response completeness): https://learn.microsoft.com/en us/azure/foundry/concepts/evaluation evaluators/rag evaluators Microsoft Azure Architecture Center, RAG prompt engineering and self evaluation: https://learn.microsoft.com/en us/azure/architecture/ai ml/guide/rag/rag prompt engineering Microsoft Azure Architecture Center, RAG LLM evaluation phase and completeness: https://learn.microsoft.com/en us/azure/architecture/ai ml/guide/rag/rag llm evaluation phase Es et al., RAGAS: Automated Evaluation of Retrieval Augmented Generation (EACL 2024): https://aclanthology.org/2024.eacl demo.16/

Multiple retrieved chunks refer to the same entity using different names. How would you avoid combining facts from the wrong entity?

Production incident (illustrative): An enterprise finance RAG assistant is asked, “What was Orion Mobility India Pvt Ltd's FY2026 service revenue, and did the Orion Mobility European parent guarantee its lease?” Search retrieves a company registry record for Indian subsidiary IN CIN 001, a 2026 filing from the European parent EU LEI 002, and an older logistics business called Orion Mobility Services IN CIN 003. A retrieved subsidiary note uses the shorthand “Orion,” while the parent filing calls itself “Orion Group.” The assistant merges the subsidiary's revenue with a different company's lease guarantee because all chunks share the phrase “Orion Mobility.” Each sentence appears grounded in some cited document, but the entity joins are wrong . Core principle: Name similarity is not identity . Entity resolution must distinguish aliases of the same legal/entity identity , parent/subsidiary and predecessor/successor relationships , and different organizations with overlapping names . A model may use retrieved facts only after every factual subject is linked to the correct, authorized, time valid canonical entity. Similar embeddings, shared names and even a company group relationship are insufficient to merge financial facts. 1. Contain the incident and establish correct identity Suspend or qualify answers that combine numbers, liabilities or permissions from ambiguously identified organizations. For high impact financial or legal questions, ask the user to choose an entity or state the unresolved entity ambiguity rather than inventing a consolidated answer. Capture a privacy protected trace: raw query, source IDs, retrieved chunk IDs, source document titles, publisher, dates, entity strings, surrounding context, entity linker outputs, canonical IDs, relation types, scores, filters, reranker outputs, packed evidence and final per claim citations. Obtain authoritative identifiers from the right registry or internal master data service: company registration number, LEI, vendor id, legal entity id and any verified source record id. A human readable label is a display attribute, not a reliable join key. Establish whether the question seeks one legal entity , a consolidated group , a named subsidiary , or a comparison of related entities . These are distinct tasks. An answer about a subsidiary should not automatically use parent company totals or obligations. Confirm every candidate chunk is authorized under the user's tenant, ACL and document effective date. Never use an external registry match to bypass a source's access controls. 2. Diagnose the first entity mixing boundary A. Source extraction: review original documents, headings, legal names, internal cross references and named entities. OCR or document parsing may strip a legal suffix, table header, registration code or a footnote identifying the true subject. B. Ingestion metadata: validate source IDs, document version, tenant, publication date, issuer entity, subject entity and canonical entity ID. A report authored by a parent may contain facts about a subsidiary. Store these as separate roles. C. Entity detection: inspect mention spans such as “Orion,” “Orion Mobility,” “OM India,” or “the Company”; preserve their local sentences and table headers. Coreference is contextual: “it” may mean a subsidiary in one section and the group in another. D. Candidate linking: show the top possible canonical entities with match evidence, not just the winning display name. Compare legal registration identifiers, issuer, country, entity type, period, address, organization hierarchy, and source document role. E. Alias graph: an alternate name should be a verified alias edge with source and valid from/valid to timestamps, not an unconditional global synonym. A name shared across brands or jurisdictions can have many possible entities. F. Relationship semantics: distinguish SAME AS, ALIAS OF, PARENT OF, SUBSIDIARY OF, ACQUIRED BY and PREDECESSOR OF. Parent of is not same as ; ownership does not automatically transfer revenue, contractual guarantees or policy applicability. G. Retrieval selection: inspect whether query routing and metadata filters restrict to the intended canonical subject, and whether dense/hybrid retrieval pulls nearby names or parent reports. Retrieval can include candidates from several entities for disambiguation, but answer synthesis must not treat them as one. H. Fusion/context packing: check whether source headers, issuer metadata and section labels are dropped when adjacent chunks are packed. If entity IDs are removed, the generator can no longer distinguish superficially similar paragraphs. I. Answer composition: represent each extracted fact as (subject entity id, predicate, value, unit, period, source span, relationship scope). Reject a generated claim whose subject ID does not match the intended entity or an explicitly authorized relationship path. J. Citation audit: a citation supporting a revenue number for IN CIN 003 does not support a revenue claim about IN CIN 001. Citation correctness requires entity applicability , not only textual entailment. This is separate from 268's general support verification. 3. Root cause decision tree Two names, same stable registration/entity ID, compatible time: likely aliases or a legal rename; verify issuer records and time ranges. Link as the same canonical identity only if authoritative evidence supports it. Same trading name but distinct registration IDs: different entities. Do not merge even if semantic similarity is high. Ask for location, legal suffix, account, product or another discriminator if unresolved. Parent and subsidiary have different IDs: represent PARENT OF, preserving ownership percentage, effective dates and consolidation scope; never rewrite it to SAME AS. Historical merger or rename: model predecessor/successor and valid time relationship; financial statements can belong to different legal entities before/after an event. Document evidence must determine whether an event creates legal continuity. Entity ID present at retrieval but missing in final prompt: context serializer/metadata loss bug. Retain canonical subject and issuer labels in every included chunk and citation. Names correctly linked yet claims cross contaminated: generator level subject binding failure. Require typed fact extraction and constrain answer synthesis to entity specific evidence bundles. Low confidence match / conflicting evidence: preserve multiple hypotheses, abstain or ask a targeted clarification. Do not choose an alias because the LLM “recognizes” the company. Same entity name reused in different tenants: canonical keys need namespace/tenant scope. Enforce ACLs before matching and after citation; never leak another tenant's entity existence. Multiple entity types: Orion could denote a company, product line or legal group. Use document type, local syntax and query intent, not generic name only similarity. Entity and financial period both vary: check IDs here and periods separately; Q271 handles temporal consistency across reporting periods. 4. Worked numerical example An illustrative evaluation contains 200 multi source questions whose correct answers require matching a mentioned organization to a specific canonical legal entity. There are 120 clear ID backed mentions, 50 aliases/renames and 30 ambiguous parent subsidiary or same name cases. Baseline raw name matching attributes facts to the right entity for 112/120 clear , 30/50 aliases , and 10/30 ambiguous , totaling 152/200 = 76% entity attribution accuracy . The other 48 are entity mixing or unresolved wrongly guessed identities. Consider an answer requiring subsidiary revenue and a parent guarantee. Baseline retrieves 8 passages : 3 from subsidiary ID A, 3 from parent ID B, 2 from unrelated same name company ID C. A similarity only merger groups all 8 under the same display label and falsely attaches C's revenue to A. Introduce verified canonical IDs and relation aware grouping: A's facts remain with A, B's with B, and C's are excluded from claims about A. A parent guarantee is only reported when an actual guarantee relation or explicit contract span supports it, not merely because B owns A. On the same labelled set, a versioned entity resolution + evidence gate correctly attributes 118/120 clear , 46/50 aliases , 25/30 ambiguous , totaling 189/200 = 94.5% . This improves attribution by 18.5 percentage points , leaving 11 errors requiring audit. For claim mixing, suppose the baseline produces 32 answers with at least one wrong entity citation out of 200 ( 16% ). The guarded pipeline reduces these to 5/200 = 2.5% , but another 9 answers request clarification . Clarifications must be measured separately rather than treated as fully correct answers. Mean entity linker candidate checks rise from 1.0 to 2.4 per query and p95 retrieval plus linking latency from 160 ms to 235 ms . These hypothetical values show quality/cost/abstention trade offs; they are not observed MachinoAI production metrics . A critical additional check is false merge rate : for labelled distinct entity pairs, count how often the linker incorrectly declares identity. A high true alias recall cannot justify merging unrelated legal entities. 5. Engineering solution: entity aware evidence assembly Canonical registry: maintain stable, issuer or business authoritative IDs, namespace, legal name, aliases, verified registration attributes, entity type, jurisdiction, and relationships. Store valid from/valid to and a provenance ID on every alias or relationship edge. Typed matching: prioritize exact trustworthy identifiers; then apply configurable multi attribute rules; finally use probabilistic/fuzzy matching only to propose candidates. AWS Entity Resolution documents rule based and ML based workflows, stable match IDs and the need to order precise rules before broad rules. Don't assume transitivity: two weak edges through the same name can form a false giant cluster. A~B and B~C by fuzzy aliases does not prove A=C. Require a strong equivalence witness or review before transitive closure, particularly for legal entities. Model issuer versus subject: source publisher entity id and fact subject entity id can differ. A group annual report may mention subsidiaries; each statement must be linked to its own subject rather than automatically tagged with the publisher's name. Per chunk provenance: include document id, version, section, subject entity id, issuer entity id, source span, period, tenant, acl, entity resolution version and confidence/evidence. Allow a chunk to contain multiple entities with mention specific spans rather than one misleading global label. Query disambiguation: resolve requested entity from explicit IDs, legal suffix, jurisdiction, product context and conversation information; do not invent missing constraints. Ask for a single discriminating attribute if ambiguity is material. Entity scoped retrieval: query authorized corpora using candidate entity IDs, but include controlled cross entity evidence when the task asks for comparisons or relationships. Keep each candidate's subject identity and score independently. Fact ledger: extract subject id → fact → value → units → time → supporting span, then reason about joins. Allow joins between A and B only with a typed, supported relationship , not an untyped shared alias. Answer verification: check every factual claim's subject ID matches its cited passage; verify relationship edges, current source versions and financial periods. Q268 checks general claim support; add entity identity entailment. Rollout: ship a versioned registry/linker, run shadow comparisons on audited entity pairs, canary on high risk financial queries, and gate on false merges, wrong entity citations, entity precision/recall, abstentions, cost and latency. Preserve ACL and deletion restrictions in all fallback paths. 6. Trade offs and common mistakes Exact IDs: excellent precision but often absent from informal user queries; propose candidate matches without silently joining them. Name normalization: punctuation/case/legal suffix standardization can help recall but must not erase disambiguating country, suffix, generation or product version. Keep raw names. Fuzzy similarity: helps spelling and aliases but can merge distinct companies or people; use domain specific thresholds and authoritative attributes. Knowledge graph aliases: useful for cross document linking; uncontrolled synonyms and transitive unions cause cascading false merges. Audit edge provenance. Single entity per chunk: simple to index but incorrect when tables, comparative reports and footnotes mention multiple subjects. Use mention/span level identities. Aggressive entity filtering: reduces noise but can suppress genuine cross company questions, acquisition histories, or subsidiary specific clauses. Make relation aware retrieval selective. LLM based entity linker: flexible but may hallucinate familiar names, assume parent==subsidiary or ignore negative evidence. Calibrate on annotated namesakes and use explicit IDs. Document title as identity: a company group title does not imply every passage describes the group totals; a report publisher may describe several subsidiaries. Temporal changes: names, ownership and legal status can change; aliases and relationship validity need effective dates and source versions. Distinguish temporal consistency ( 271). Privacy and tenant boundaries: linking external or other tenant records can reveal sensitive entities or relationships. Apply authorization to matching candidates and all evidence returned. Abstention: reduces harmful false mergers but can lower answer rate; measure correct disambiguation requests separately and improve registry coverage. 7. Prevent recurrence Maintain an entity collision golden suite: identical names in distinct jurisdictions, subsidiaries versus parents, old/new legal names, acronyms, shared brands, multiple issuers, product/company ambiguity, renamed businesses, and historical acquisitions. Require negative pair tests showing closely named distinct entities remain separate. Measure pairwise precision, false merge rate, false split rate, entity attribution accuracy, and wrong entity citation frequency. Test multi entity documents whose title belongs to a parent but passages refer to a subsidiary, joint venture or separate product; assert issuer and subject are not conflated. Add numeric checks that company revenues, units, currencies, fiscal periods and entity IDs align with each cited fact. A correctly copied number attached to the wrong entity is a correctness failure. Log entity candidate alternatives, selected canonical IDs, match rule/version, confidence, supporting attributes, source span, relationship graph path and final claim subject under access controlled retention. On new aliases, mergers or registry data feeds, run differential linking tests and verify the effect on all indexed chunks and cached answers. Use stable entity IDs to prevent renamed entities from being re created as duplicates. Alert on rising wrong entity citations, large clusters formed from fuzzy edges, rapid changes in entity linkage after a registry update, cross tenant linkage attempts, and unusually low disambiguation rates on ambiguous queries. When no trustworthy identity is established, return an explicit clarification or separate attributed facts rather than an unsupported combined conclusion. Review sampled high risk failures with domain owners and keep a record of corrected relationships and source evidence. Interview takeaway: The right retrieval candidates can still yield a wrong answer if the system joins facts using a shared name instead of canonical identity . Link mentions to stable, time valid entity IDs; distinguish alias from parent/subsidiary relationships; keep entity specific provenance with every extracted fact; verify that claims and citations refer to the same entity; and abstain when identity cannot be resolved reliably. Research references AWS, Entity Resolution matching workflows and stable match IDs: https://docs.aws.amazon.com/entityresolution/latest/userguide/create matching workflow.html AWS, Rule based matching and match rule auditability: https://docs.aws.amazon.com/entityresolution/latest/userguide/creating matching workflow rule based.html AWS, Transitive matching and overmatching: https://docs.aws.amazon.com/entityresolution/latest/userguide/transitive matching.html Scharpf et al., Entity Linking with Wikidata: A Systematic Literature Review (ACM Computing Surveys, 2026): https://doi.org/10.1145/3795134 AWS, Entity Resolution input schema and unique IDs: https://docs.aws.amazon.com/entityresolution/latest/userguide/create schema mapping.html