The correct evidence was retrieved, but context trimming removed the sentence containing the answer. How would you identify and fix this?

Production incident (illustrative): A procurement RAG assistant is asked: “Under the 2026 contractor travel policy, who can approve an exception above the Singapore lodging cap?” Retrieval correctly returns section 4.6 of the current policy. The first two sentences describe the lodging limit; the third states “Exceptions above the cap require written approval from the Regional Finance Director before booking.” The retrieved chunk is 1,420 tokens, but the context assembler extracts only its first 900 tokens, discarding the approval sentence and its paragraph reference. The assistant cites the correct policy yet guesses that a line manager approves exceptions. Search Recall@10 looks healthy: the evidence was retrieved but not delivered to the model.

Core principle: Treat retrieval and context assembly as separate correctness boundaries. A document in the retriever's candidate list does not imply the answer-bearing span survived deduplication, compression, truncation, ordering, serialization, token-budget enforcement, provider input limits, or final prompt construction. Investigate *span survival* before changing the embedding model, ANN index, top-k or LLM prompt wording.

1. Contain the failure and freeze a trace

• For policy or financial decisions, do not allow an answer unsupported by the final model-visible context. If the relevant sentence cannot be preserved under the budget, return an evidence-insufficient response or use an approved evidence-refetch path.
• Capture a permission-protected request trace: query and principal/tenant, immutable source document/version, selected chunk IDs, search and reranker scores, original character/token offsets, raw retrieved text hashes, compressor outputs, context packing order, token counts per stage, final serialized model messages, exact model/version and provider truncation policy.
• Obtain the authoritative answer-bearing sentence with its source span and active effective date. Validate that the user's ACL permits the full span. The key is not simply that the parent document is present.
• Replay the identical request with a frozen retrieval snapshot and deterministic packing parameters. Compare the *actual model-visible prompt bytes* or a securely recorded hash/span inventory with candidate payloads. Avoid depending on a debug print of pre-trim content.
• Separate this incident from later lessons: if the exact answer sentence is present in the final prompt but the model overlooks it, investigate position sensitivity/attention behavior (#267); if present and cited but unsupported claims appear, inspect claim-grounding (#268). Here the sentence never reaches the model.

2. Instrument the evidence survival funnel

• Source truth: identify source_id, document_version, section_id and the gold answer span [start_offset,end_offset); confirm the source is current, authorized and complete.
• Ingestion chunk: verify the answer span is included in the actual indexed chunk and that chunk offsets map correctly to original source text. The entire parent file existing is insufficient.
• Retriever output: inspect the raw text of the retrieved candidate at the retrieval API boundary. Compare chunk ID, source-offset interval, text hash and gold-span overlap.
• Reranker-selected set: ensure the candidate remains after rerank and thresholding; a candidate dropped before the assembler is a candidate-selection incident, not a trimming incident.
• Contextual compression: diff compressor input/output; inspect extractive selector, LLM summarization or text truncation, including whether the exception sentence and negation survived. A plausible summary that omits one approval requirement is unsafe.
• Deduplication and grouping: check whether overlaps, neighboring chunks, or multiple source versions are collapsed. Identity by document ID alone may accidentally discard distinct answer-bearing sections.
• Token packer: enumerate each selected chunk's token cost, overhead for citations/source labels, reserved response tokens, conversation history, system instructions and examples. Record why and where each chunk or span was rejected: lower score, fit failure, maximum per-document allocation, trim-to-N, dedup, or model hard limit.
• Final serialization/API: count tokens with the target model tokenizer and inspect the actual outgoing message list after SDK adapters, provider middlewares and any automatic truncation. Different tokenizers, message wrappers or hidden per-field caps can produce an overflow even if application estimates pass.
• Answer/citation: verify that every asserted fact is supported by a model-visible span, not just a matching source title or document reference.

3. Root-cause debugging: find the first losing stage

• Gold sentence absent from indexed chunk: document extraction, OCR, chunking or versioning bug. Repair source-to-chunk mapping and reindex; expanding the inference context will not recover unindexed text.
• Present in indexed chunk but absent from retriever payload: query return-field projection, text preview/length limit, metadata-only retrieval or a mismatch between vector record and backing document store. Request the full authorized payload using stable IDs.
• Present after retrieval but absent after reranking: candidate cutoff or grouping bug. Revisit reranker top-N and source/version-aware grouping before changing context budget.
• Compressor drops the critical clause: overaggressive extractive threshold or lossy abstractive compression. Evaluate required-span recall, reinsert verbatim evidence, or bypass compression for protected facts. Validate that summarization does not invert “must,” “unless,” or monetary/datetime qualifiers.
• Assembler slices `text[:max_chars]` or `tokens[:limit]`: naive prefix trimming loses salient content near the end. Use a query-aware, sentence-boundary-safe span selector and retain offsets/provenance.
• Packer removes lower-ranked chunk that holds a necessary fact: a global greedy knapsack may favor redundant high-score chunks; deduplicate repeats, reserve coverage for required facets, and re-rank by evidence-per-token rather than raw similarity alone.
• Conversation history or examples crowd out evidence: enforce an explicit context budget and prioritize authorized answer-bearing evidence. Summarize prior turns without dropping the active user constraints.
• Provider truncates after app packing: application tokens were counted with an inaccurate tokenizer or output reserve; target model maximum or an SDK setting differs from the environment. Fail closed on over-budget rather than silently sending a clipped prompt.
• Gold sentence in final prompt but wrong answer: move to answer-position/grounding diagnostics; no longer attribute the failure to trimming.

4. Beginner-friendly numerical example

• Assume a model supports 8,192 total context tokens in the relevant request configuration. Allocate 900 system/instruction tokens, 650 conversation/query tokens, 900 reserved response tokens, and 200 serialized metadata/citation overhead tokens. The true retrieval-context budget is 8192 − 900 − 650 − 900 − 200 = 5,542 tokens.
• First-stage retrieval returns 10 chunks × 1,000 tokens = 10,000 tokens of raw content, including many duplicated summaries and one essential policy paragraph. The assembler naively packs the first five chunks (~5,000 tokens) and slices one remaining chunk to fit the last 542-token allowance. The exception sentence begins at token 730 of that sixth chunk, so it is not model-visible, although its source document is marked “retrieved.”
• In a benchmark of 200 labelled policy questions, the answer-bearing evidence occurs in the retrieved candidate pool for 184/200 = 92%. After naive context packing, only 132/200 = 66% final prompts contain all required answer spans. 52 questions lose evidence during assembly; initial retriever Recall@k stays at 92%.
• A controlled remediation preserves the original relevant source and selects query-relevant contiguous sentence spans with nearby qualification and approval sentences. After deduplicating overlapping summaries and reserving a 5,542-token budget for source-labelled evidence, suppose 176/200 = 88% final prompts retain complete gold evidence. That is +22 percentage points of final-prompt coverage versus 66%, while the 92% retrieval ceiling remains unchanged.
• At the illustrative answer stage, grounded correct answers rise from 124/200 = 62% to 168/200 = 84%, with no change to the embedding model. This does not mean the model always answers correctly whenever evidence survives; 176 evidence-complete prompts versus 168 correct grounded answers leaves eight failures for other causes.
• Additional span extraction may increase p95 from 170 ms to 225 ms and add computation. Test whether 55 ms and extraction costs meet the SLO. Numbers are hypothetical, not live MachinoAI measurements.
• Treat a sentence's presence as verbatim exact normalized span containment or validated source-offset overlap, not a fuzzy semantic match alone. Fuzzy matching can incorrectly declare a paraphrased but meaning-changing summary “complete.”

5. Engineering solution

• Token-budget contract: calculate available_context = model_context_limit − instructions − query/history − serialization_overhead − output_reserve, with the tokenizer for the target model. Leave a safety margin for provider/tool wrappers. Treat a negative budget as a hard packing error, not a reason to silently trim.
• Span-aware provenance: every selected text region carries source_id, version, section, start/end offsets, content_hash, token_count, parent_chunk_id. Store a compact structured manifest of included and removed spans with reason codes; secure or minimize raw-text traces.
• Answer-span preservation: for evaluation cases, identify the gold sentence and its minimal essential surrounding context (e.g., approval authority, exceptions and effective date). In production, use a query-aware evidence selector with abstention when evidence is uncertain; don't claim the system knows unseen gold spans perfectly.
• Contiguous-neighbor expansion: if a relevant clause references “this exception,” include preceding policy definition, next sentence and footnotes as needed, not just one attractive isolated sentence. Bound expansion by verified source versions and token costs.
• Source-aware deduplication: collapse exact overlapping portions but keep distinct approval conditions, exclusions and qualifiers. Deduplicate before aggressive packing to free tokens without information loss.
• Evidence-per-token optimization: compare constrained rerank/selection by expected information utility per token, with a minimum-coverage requirement for each facet. Allow multi-document coverage when needed. The 2026 controlled evidence-fragmentation study found gains from contiguous span selection but also documented cases where it underperformed—benchmark rather than assume universal benefits.
• Compressor guardrails: when using an abstractive summary, keep the original source span retrievable and require entailment or literal-fact checks for high-stakes numeric, time and negation claims; on failure reinsert verbatim evidence. A second LLM summarizer is not inherently lossless.
• Pack then validate: construct the real model message object, tokenize it, and assert mandatory evidence manifest entries are present in the outgoing prompt. If too large, reduce redundant context, use parent/section refetch, stage the answer across batches, or safely decline.
• Citation verification: return citations tied to actual included spans and current source revisions. A title-only citation to a retrieved-but-trimmed document must not count as supported evidence.
• Controlled rollout: shadow test packers on fixed authorized queries, then canary with checkpoints for gold-span survival, complete-evidence query rate, answer support, p95/p99 and token cost; keep a feature flag and rollback.

6. Practical failure scenarios and trade-offs

• More context: may rescue removed sentences but raises cost/noise and can still exceed a hard limit. A larger context window does not excuse an incorrect packing policy.
• Smaller chunks: can prevent arbitrary mid-sentence clipping but may split a qualification across chunks; preserve section/neighbor linkage.
• Longer chunks: retain neighboring context but compete harder for the budget and may be chopped by the assembler. Choose retrieval and delivery units independently when possible.
• Extractive compression: preserves exact wording for selected spans but can miss cross-sentence references. Include neighboring clauses when required.
• Abstractive compression: saves tokens but risks dropping amounts, exclusions, approval authority and negative qualifiers. Preserve source and check fidelity.
• Greedy relevance-first packing: prioritizes high-scored text yet may repeatedly select the same source and drop minority facts; use diversity/facet coverage with relevance bounds.
• Token-aware packing: safer than character counts but tokenizer versions and message wrappers can change; count the *final* serialized request.
• Preemptive truncation: avoids API errors but can turn a successful retrieval into a confident unsupported answer. Fail with explicit reason when critical evidence cannot fit.
• Full parent refetch: supplies missing context and offsets but adds network/IO and may surface superseded or unauthorized content without source-version checks.
• Multiple inference passes: map-reduce/iterative refinement retains more evidence with extra latency and intermediate-error risks; compare to simpler span-based packing first.

7. Monitoring and prevention

• Build a stage-level evidence-preservation dashboard: source present → indexed chunk contains span → retriever payload contains span → reranker keeps span → compressor preserves span → final prompt preserves span → answer supported by included span.
• Track final-prompt evidence Recall@k, complete-evidence query rate, gold-span survival, truncation incidence, dedup loss, context tokens used versus budget, citation-support rate, cost and p95/p99 separately from first-stage retrieval Recall@k.
• Add tests with answer sentences at start, middle and end of a chunk; qualification one sentence after a boundary; adjacent document versions; duplicated boilerplate; changed conversation-history size; and near-limit token prompts.
• Record packing reason codes for each dropped chunk/span (not_selected, duplicate, budget_exceeded, per_document_cap, compressor_removed, provider_truncated) and reconcile manifests to the final serialized message.
• Canary every change to chat-history summarization, tokenizers, model context limits, reranker depth, chunk size, context compressor or packing algorithm against an evidence-complete labelled suite.
• Add a no-grounded-evidence → abstain or refetch guardrail. If a required span is absent from final context, the system must not claim the answer based on document title or latent model knowledge.
• Rehearse rollback while retaining active ACLs, deletions and source-version rules; record both overall answer correctness and independently verified span provenance.

Interview takeaway: When retrieval succeeds but the answer sentence disappears, diff the retriever's full payload against the final model-visible prompt, attributing every discarded span to compression, deduplication, packing or provider truncation. Fix the first loss with token-aware, span-preserving selection and verify complete evidence in the outgoing prompt, not just retrieved-document recall.

Research references

• Microsoft Azure Architecture Center, *RAG prompt engineering: token budgets, chunk selection, context overflow*: https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/rag/rag-prompt-engineering
• Microsoft Azure Architecture Center, *RAG chunking phase*: https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/rag/rag-chunking-phase
• LangChain, *Improving Document Retrieval with Contextual Compression*: https://www.langchain.com/blog/improving-document-retrieval-with-contextual-compression
• Aloteibi, *Evidence Fragmentation Is a Distinct Failure Mode in Chunked Retrieval* (Information 2026): https://www.mdpi.com/2078-2489/17/11/1129
• Banerjee et al., *The Bottleneck Is Retrieval, Not Reading* (2026): https://arxiv.org/abs/2610.26951