The relevant evidence is in the middle of a long prompt, but the model repeatedly overlooks it. How would you test for position sensitivity?

Production incident (illustrative): A corporate travel RAG assistant correctly retrieves and *includes* the 2026 contractor-policy paragraph stating that Singapore lodging exceptions require Regional Finance Director approval. The final model prompt contains 14 policy and FAQ passages totaling 11,600 tokens. The controlling paragraph is around token 6,100, surrounded by older employee travel FAQs. The assistant repeatedly answers “manager approval” despite a correct, current, authorized sentence in the model-visible context. When engineers move the exact same paragraph to the beginning, answers improve. This is a candidate case of position sensitivity (“lost in the middle”), not proof of bad embeddings or context trimming.

Scientific grounding: Liu et al., *Lost in the Middle: How Language Models Use Long Contexts* (TACL 2024) showed that answer accuracy can depend strongly on where relevant text appears, commonly higher at the beginning or end of long contexts than in the middle. Later work investigated positional bias and evidence-aware reordering. These are model- and task-dependent empirical effects, not a universal law that all models ignore their middle tokens.

1. Confirm this is a reading failure, not a retrieval/packing failure

• Check that the expected authoritative document was retrieved, remained eligible, survived reranking, survived compression and appears verbatim in the exact serialized prompt sent to the model. Use source IDs, version IDs, byte/token offsets and final-message hashes. This distinguishes #267 from Q266, where trimming removed the answer sentence entirely.
• Preserve a verified answer-bearing span with its surrounding qualification: approval role, exception scope, effective date and policy hierarchy. Ensure the span is *actually applicable to the user* and current for the relevant region and personnel category.
• Check whether the model is asked the same question each time, with the same system/developer instructions, chat history, generation settings, model release and tool responses. Otherwise apparent position sensitivity could be a changed task.
• Prevent unsupported high-stakes answers by requiring model-visible source support; when evidence is conflicting or missed, return uncertainty or retrieve focused current source snippets rather than guess.
• The prompt may contain a lot of data but still be within the model's published context window. Capacity to ingest N tokens does not guarantee equal attention or robust use of every position.

2. Design a controlled position-swap experiment

• Construct 100–300 labelled questions with an authoritative answer span and several plausible distractor passages. Use an isolated, authorized fixed corpus snapshot and redact sensitive content from experiment logs.
• Fix the question, source passage text, distractor multiset, total prompt token count, model, decoding settings, instruction template and response-format requirements. Only change the answer passage's position in the prompt. If a permutation changes total tokens because of headers or numbering, normalize wrappers and record actual token offsets.
• For each question, create placements at 0%, 25%, 50%, 75% and 95% of the retrieval-context region (near-start, early-middle, center, late-middle, near-end). Each placement should preserve the answer sentence and enough neighboring text to remain interpretable.
• Randomize distractor ordering in a *balanced* way across placement conditions or use a Latin-square/permutation design so a particular distractor is not always adjacent to the answer when it is centered. Keep the same authorized fact set and versions in all conditions.
• Repeat trials with different fixed seeds or multiple independent samples if the API is not deterministic. Even with temperature 0, server implementation or model updates may yield variation; log exact model identifier and evaluation period. Repeated runs also estimate noise.
• Counterbalance task semantics: include single-fact lookup, exception application, multi-document comparison, contradictory outdated policy, numeric extraction, exact document code and negative constraints. Test short vs long contexts with the *same* evidence to measure length-position interaction.
• Use oracle-placement tests with a gold passage deliberately inserted as a controlled benchmark, separately from normal retrieved-RAG replay. The oracle test isolates reading ability; end-to-end RAG evaluation additionally includes retrieval and packing.
• When possible, run the same prompts across a second model to test whether the measured curve is specific to the current reader, not to all LLMs.

3. Collect useful measures rather than one accuracy number

• Position-conditioned supported accuracy: correct_and_cited_answers_at_position / evaluated_questions_at_position. Grade source support separately from factual correctness.
• Middle penalty: accuracy at the average of beginning/end conditions minus accuracy at 50%. Report absolute percentage points and confidence intervals.
• Answer-span recognition: does the model quote or reference the correct source span when asked to identify it? Compare with final policy-decision accuracy to separate locating from reasoning over evidence.
• Wrong-policy citation rate: incorrect evidence from outdated or inapplicable distractors; unsupported-answer rate: claims not entailed by cited text; abstention rate: inability to answer when valid evidence is available.
• Facet/exception coverage: did the answer include required jurisdiction, role, date and exception approval? A response can name the approval officer but omit a critical eligibility condition.
• Token-offset diagnostics: record absolute token position and relative position inside the retrieved-content section, prompt length, source position/rank and distractor overlap; a passage at absolute 5,000 may be middle for 10k but near-start for 40k.
• Statistical inference: because all positions use the same question, use paired bootstrap over question IDs or paired categorical tests, report confidence intervals and avoid treating 5 placements × repeated samples as fully independent.
• Failure localization: if the exact answer text is missing in one condition, this is prompt construction/truncation, not a pure positional effect. If a contradictory distractor moves relative to the answer, control for conflict and document authority.

4. Numerical worked example

• Test 200 labelled questions with a *single gold paragraph* and a fixed set of distractors, using identical 12,000-token retrieval sections and the same model. Every prompt includes the exact gold sentence at all placements.
• Suppose supported correct answers are: start 176/200 = 88%, 25% position 144/200 = 72%, middle 112/200 = 56%, 75% position 138/200 = 69%, end 170/200 = 85%.
• The mean edge accuracy is (88% + 85%)/2 = 86.5%. The middle penalty is 86.5% − 56% = 30.5 percentage points. The pattern resembles a U-shaped lost-in-the-middle curve; it does not show why a particular transformer behaves this way.
• In a control with a 2,000-token retrieval section, suppose start and middle are 90% and 86%. The much smaller 4-point penalty suggests a *length × position* interaction; use paired testing and uncertainty bounds before concluding.
• The original production prompt puts the crucial Regional Finance Director clause at the 50% position. Baseline on the 200-query set is 112/200 = 56% when gold is centered. A carefully tested ordering strategy that places the highest-confidence, required, current evidence early yields 168/200 = 84% supported answers in a *separate validation run*; absolute improvement 28 points. Do not claim this equals the 88% oracle-start result: production evidence ranking and context effects differ.
• The same strategy increases average context-packing work by 15 ms, and may not alter LLM inference cost if token count stays fixed. If it adds an extra reranker pass, overall p95 can increase; measure rather than assume.
• These figures are illustrative, not measured MachinoAI production data or a published paper result. A product decision needs its own task-specific significance test, confidence interval, latency budget and no-unsafe-citation gate.

5. Root-cause debugging decision tree

• Gold sentence absent at the final model-input boundary: Q266 packing/trimming/serialization problem; fix span survival before testing position sensitivity.
• Gold sentence present, early and late succeed but middle fails, all else equal: credible position sensitivity. Compare context length, model revision, distractor placement and task complexity.
• All placements fail: wrong/ambiguous gold evidence, poor question formulation, model comprehension/grounding defect, conflicting authoritative sources or a broken instruction; moving context is unlikely to fix it.
• Only positions near conflicting old policies fail: distractor interference, source-authority ambiguity or version sorting—not necessarily generic “middle blindness.” Preserve priority rules, annotate effective dates and test conflict controls.
• Long contexts fail everywhere but short ones pass: excess noise, answer-context dilution, unsupported length scaling or truncation. Reduce/structure context or use query-focused evidence extraction and compare across lengths.
• Correct span is recognized but final answer omits it: instruction-following or multi-step reasoning failure, perhaps a missed exception. Add explicit source-grounding verification and inspect claim support, leading into #268 and #269.
• Repeated identical prompts produce different answers: assess stochasticity, server model changes, request routing or tool variability; don't attribute all variance to position.
• Answers cite the same document but wrong section: citation granularity and source-version conflation; demand span-level provenance.

6. Engineering solution and guarded rollout

• Evidence-aware ordering: prioritize a small number of high-confidence, current, authorized answer-bearing passages near the start of the context and place complementary facts deliberately. Don't push all evidence to the ends without evaluating multi-facet coverage and relevance.
• Query-focused reduction: remove duplicated boilerplate and less useful distractors while retaining complete approval/exception qualifiers, citations and cross-sentence definitions. A *shorter faithful context* may help more than clever reordering.
• Structured evidence envelope: make source ID, title, version, effective date, applicable role and relevant excerpts visually distinct; place explicit original user question near the evidence when helpful. Validate that structured formatting doesn't itself introduce measurement confounds.
• Span-first retrieval: if current top chunks are long, refetch authorized relevant subsections with surrounding sentences and preserve source offsets. Avoid lossy summaries for legal, numeric or approval claims; keep the original span auditable.
• Two-pass evidence check: in selected high-risk cases, have a bounded first pass identify which cited spans answer each facet, then generate using those spans. Validate its extra latency and ensure first-pass mistakes do not suppress required evidence.
• Position-robust prompting: test prompts instructing the model to list supporting evidence IDs first, then produce an answer; do not assume these instructions eliminate underlying bias. Compare against a strong simple retrieval-and-rerank baseline before adopting a learned context planner.
• Citation and authority gate: verify the final answer's claims against *included*, current, authorized spans and permit abstention/refetch where support is missing. Never trust the model's stated citation without checking it.
• Deployment: shadow replay on balanced position tests, then canary with supported-answer accuracy by position/length, critical-exception omission, wrong-policy citations, cost, p95/p99 and tenant/ACL guardrails. Feature-flag the packer/orderer and retain reversible prior configuration.

7. Trade-offs, experimental pitfalls and limits

• Reorder-only: often cheap and reversible, but can move irrelevant content toward salient positions and may fail for tasks needing multiple distant facts.
• Shortening context: reduces distraction/cost but risks deleting qualifiers; verify exact relevant-span survival separately, as Q266 establishes.
• Position-conditioned benchmark: demonstrates sensitivity but not universal production gains. Crafted gold placements may be easier than real retrieval.
• Random permutations: test robustness and control confounds but expand experimental cost and complicate inference; use balanced assignments and paired statistics.
• Reranking: improves selection but may assign high scores to generic summaries or superseded policies. Include authority, coverage and provenance criteria.
• Learned context planning: extra models and training may add complexity without beating strong simple retrieval; use held-out, leakage-free evaluations.
• Alternative model: one long-context model may be less position-sensitive than another, but pricing, latency, source grounding and data-governance constraints can outweigh the gain.
• Near-end placement: can improve access to a fact but risks placing essential content after instructions whose priority is unclear; always respect system and security instructions.
• Large context window: a model supporting 100k+ tokens does not guarantee it uses centered evidence equally well across tasks.
• Model updates: position profiles may change when provider models or tokenizer behavior changes; rebaseline after upgrades.

8. Prevention checklist

• Add a balanced position-swap regression suite with gold facts at beginning, 25%, center, 75% and end, across short/medium/long contexts and known authorized source versions.
• Track supported-answer accuracy, middle-position penalty with uncertainty, first-stage retrieval coverage, final-prompt span presence, wrong-citation rate, exception omission and per-position context length.
• Require final prompt manifests with source IDs, section offsets and relative positions; log exact model/config versions in secure traces so shifts are diagnosable.
• Include adversarial distractors: obsolete policies, near-identical region rules, multi-step approvals, contradictory source revisions and repeated boilerplate. Verify the correct version outranks inapplicable text.
• Check both single-source and multi-document questions; prioritizing one gold span can harm questions needing several distinct documents.
• Alert on new position gaps after context-packer, reranker, source-ordering, model, summarizer or context-budget changes; use offline position controls and online citation review.
• Audit security and freshness regardless of ordering: moving an unauthorized or superseded paragraph to the top is a critical failure, not an accuracy improvement.

Interview takeaway: First verify the answer sentence is inside the exact model-visible prompt. Then perform a controlled position-swap experiment that changes only where the same authorized evidence appears while keeping the question, distractors, tokens, model and settings fixed. Measure supported accuracy by position, confirm a statistically credible middle penalty, and test evidence-aware reordering or shorter faithful context with citation and latency guardrails.

References

• Liu et al. (2024), *Lost in the Middle: How Language Models Use Long Contexts*, TACL: https://aclanthology.org/2024.tacl-1.9/
• Hsieh et al. (2024), *Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization*: https://aclanthology.org/2024.findings-acl.890/
• Boytsov et al. (2025), *Positional Bias in Long-Document Ranking*: https://aclanthology.org/2025.ijcnlp-long.46/
• Yi et al. (2026), *Attention Basin: Why Contextual Position Matters in Large Language Models*: https://aclanthology.org/2026.acl-long.1198/
• Li and Chen (2026), *When Learned Context Planning Fails to Beat Strong Retrieval*: https://aclanthology.org/2026.insights-1.11/