Retrieval works correctly for short questions but fails for long, detailed enterprise queries. How would you isolate the cause?

Production incident (illustrative): A procurement RAG assistant answers “What is our contractor travel reimbursement cap?” correctly. But it fails on: “For a Singapore-based contractor traveling to Berlin in Q3 2026, compare the reimbursement cap and required VP approval under the 2026 Global Travel Policy and the regional exception, excluding employees and expired policy revisions.” Search retrieves generic reimbursement FAQs; the dated regional exception and approval clause never reach the answering model. The vector database is healthy and fast. The failure appears only when a query includes several conditions, document references and a comparison.

Diagnosis principle: Long queries can fail for *different* reasons: the embedding input may be truncated; one dense vector can dilute multiple intents; query rewriting may remove hard constraints; mandatory metadata filters may exclude evidence; relevant facts may be scattered across documents; and a shallow retrieval or fusion pool may discard needed passages. Do not assume every complex query should be decomposed. Isolate the earliest failing stage with a controlled, length- and complexity-stratified evaluation.

1. Contain the incident and capture a reproducible trace

• Preserve the raw user question, query after chat-history compression, normalized and rewritten queries, actual embedding input, tokenizer token count, embedding model/revision and maximum accepted length, input truncation mode, and the resulting vector dimensions/norm. Avoid storing raw confidential user text in broadly accessible logs.
• Record the authorized tenant, ACL, geography, document versions, effective dates, and other hard constraints *as separate validated fields*. Never infer authorization by generating a looser semantic query.
• Capture all candidate sets at each boundary: keyword/BM25, dense retrieval, per-subquery branches, fusion, reranking, final context assembly and cited answer. Include per-stage ranking, candidate budget and query latency.
• Build a paired reproducer: (A) short core question; (B) same question with legitimate qualifiers; (C) same question plus verbose but irrelevant background. Use the same user permissions and corpus snapshot; compare the first stage where required evidence disappears.
• If the query asks for a policy decision without sufficient current evidence, return an appropriately qualified abstention or request a missing critical constraint instead of citing a policy for another geography/personnel category.

2. Isolate length versus semantic complexity

• Length-only effect: add harmless context while holding the *same target answer* fixed; if results fail only once the input exceeds a token threshold, inspect truncation, positional sensitivity, model max length, query-compression code and request-size limits.
• Multi-intent effect: keep token count roughly constant but vary the number of independent subquestions, comparisons and conditions. If retrieval drops with facet count rather than tokens, the problem may be intent dilution or insufficient evidence coverage.
• Constraint sensitivity: move an essential identifier, country, year or “except/not” clause from the beginning to the end of the question. If the last clause vanishes, inspect input truncation or preprocessing; if it survives but isn't applied, debug structured parsing/filtering.
• Document distribution: establish whether one chunk contains the answer or whether the answer needs several policy documents and a join/comparison. A global top-5 list can be dominated by one topic even though each individual clause is answerable.
• Time/version: verify the 2026 revision and regional exception are both active for the requested date. The absence of a superseded or unauthorized chunk is correct behavior, not a retrieval regression.

3. Root-cause debugging sequence

• A. Query preservation: compare the raw text with text delivered to the embedder and BM25 engine. Log a hash plus safe diagnostic excerpt of the critical fields, token count and truncation position; detect accidental summarization, stopword/negation deletion, long-query maximums and encoder-side truncation. A 1,200-token input does not fit a hypothetical 512-token limit; whether it fails or truncates depends on provider configuration.
• B. Eligibility: check ACL, tenant, effective date, country, role and current-version filters against each expected answer-bearing chunk. Distinguish domain constraints such as “contractor only” from access-control policies; never relax access filters in production.
• C. Direct known-chunk lookup: verify each expected document/version/chunk exists and is search-visible. This is a sanity check, not proof that the natural-language query ranks it highly.
• D. Exact rank versus ANN: obtain a trusted exact/flat nearest-neighbor reference on the same active, filtered corpus. If the right chunk is exact rank 137, increasing HNSW ef cannot make its semantic similarity intrinsically better. If exact rank 3 but ANN omits it, investigate ANN settings/readiness rather than query decomposition.
• E. Branch-level retrieval: run the original full question, stripped essential-question variant, position-swapped variant, BM25, dense retrieval and 2–4 *faithful* subqueries side by side. Check whether each required facet gets at least one authorized support passage.
• F. Candidate and fusion budgets: log per-branch top-k, deduplication, reciprocal-rank fusion/score normalization and reranker input. One branch may exhaust the global budget, while a passage for another facet is retrieved but pruned.
• G. Reranking and context: if all required passages reach the candidate pool but vanish later, investigate cross-encoder input truncation, long-query score bias, reranker top-N, answer context limits and misplaced citation/source grouping. Distinguish candidate retrieval failure from answer synthesis failure.
• H. Quality of query transformations: compare rewritten/decomposed requests against the original's essential constraint checklist; reject any rewrite that silently drops “contractor,” “Q3 2026,” “Berlin,” exclusion/negation, or an exact policy identifier.

4. Carefully choose how to repair retrieval

• First fix lost query bytes, broken query compression, unintended tokenizer truncation or incorrectly applied hard filters. Do not add an LLM decomposition layer to hide an input-loss bug.
• For multi-facet questions that genuinely require different documents, test facet-aware decomposition: preserve the original full query, derive a bounded number of focused subqueries, retrieve evidence per facet, deduplicate/merge and rerank under the original constraints. Build a coverage checklist so one facet doesn't crowd out others.
• However, decomposition can weaken the relationships between constraints: for questions requiring one document to satisfy every condition, separate retrieval branches may independently find partial matches. Test a monolithic-query retrieval + constraint-aware rerank path as an alternative. Recent stage-aware retrieval research reports that decomposition is not universally beneficial at first-stage retrieval.
• Use keyword/hybrid retrieval for policy titles, exact dates, locations, document codes and phrases whose literal identity matters. Where the engine supports it, separate verified metadata constraints from semantic intent; apply both in authorized search.
• Improve long-query embeddings only after establishing the failure mechanism: use a suitable model/context budget, meaningful query condensation that preserves all hard constraints, or query-encoder prompting required by the chosen model. Validate query/document model compatibility.
• When evidence spans multiple documents, keep document and citation provenance for each fact. Return an uncertainty/coverage note when a required regional exception or approval policy cannot be verified; never fill a missing branch with a plausible general-policy passage.

5. Worked numerical example

• Evaluate 200 short, single-intent queries and 200 long, multi-condition queries with verified active, authorized answer evidence. The short set succeeds for 184/200 = 92%; the long set has complete evidence for 104/200 = 52%.
• Each long query requires three evidence facets (policy cap, regional exception, approval requirement), so 200 queries require 600 facet-level evidence hits. The baseline single-vector top-10 retrieves 390 of 600 required facet instances = 65% facet coverage, but only 104 of 200 = 52% questions have *all three* facets. These are different metrics.
• Instrument one failing query: raw request 780 tokens; the embedding endpoint (in this hypothetical configuration) silently retains only the first 512. The final 268 tokens contain “contractor,” “2026 regional exception” and an exclusion. A position-swap test places these constraints first and changes the candidate set sharply, identifying truncation as one cause.
• After fixing input preservation and using a tested constraint-aware query representation, suppose all-facet question coverage rises to 142/200 = 71%, with original 200 questions held constant. Missing evidence persists for queries whose facts span separate documents.
• Add a controlled four-branch retrieval including the original query, a two-facet comparison branch, a regional-exception branch and an approval branch; keep top-12 candidates per branch, deduplicate, and rerank a bounded candidate pool. Assume the test reaches 172/200 = 86% complete-evidence coverage, with 548/600 ≈ 91.3% facet hits. These hypothetical values require measurement, not an assumed gain from decomposition.
• End-to-end search-and-rerank p95 increases from 180 ms to 320 ms and the average number of retrieval calls rises from 1 to 4. Absolute complete-evidence improvement is 86 − 52 = 34 percentage points; relative improvement is 34/52 ≈ 65.4%. Accept the result only if the latency/cost budgets and security checks pass.
• An A/B test also measures the monolithic + constraint-aware reranker alternative; if it meets the same coverage with fewer calls, prefer it. The point is *isolating and testing a remedy*, not prescribing decomposition for every long question.

6. Engineering solution and rollout

• Define a versioned diagnostic schema: raw_query_hash, token_count, embedder_input_tokens, truncated, intent_facets, hard_constraints, rewrites, retrieval_branches, branch_top_k, fusion_output, reranker_input, facet_coverage and citation_ids.
• Add constraint preservation tests: every rewritten or subquery plan must retain or consistently inherit validated tenant, date, role, geography, document type, negation and exact identifiers. Avoid treating LLM-inferred user attributes as authoritative filters.
• Establish a two-stage candidate pipeline: broad first-stage *authorized* retrieval (monolithic plus carefully selected extra branches when necessary) followed by source-aware, constraint-aware reranking and evidence coverage validation.
• Include diversity/coverage controls (such as one candidate per required facet before final reranking) when evidence truly spans sources; keep grouping by authoritative document/version to prevent mixed citations.
• Shadow-evaluate length bins, number of facets, token-position permutations, multi-hop cases, high-selectivity filters and negative/unsatisfiable constraints. Monitor Recall@k, all-facet evidence coverage, citation accuracy, p95/p99, token costs, false inclusions and unauthorized retrieval violations.
• Canary only the identified fix, with feature flags and rollback. A more complex retrieval path should activate selectively for validated multi-facet cases, not automatically for every verbose question.

7. Trade-offs and common mistakes

• Longer embedder context: avoids lost text but does not guarantee that a single pooled vector represents every required fact equally well.
• Query shortening: reduces noise and cost but can erase essential negations, dates or constraints; preserve a normalized constraint ledger and original query.
• Decomposition: improves breadth in some multi-hop workloads but can dilute interdependent conditions, produce spurious partial matches and multiply latency.
• Higher top-k: can increase candidate recall but costs more reranking and may still crowd out rare facets. Measure branch diversity and final all-facet coverage.
• Hybrid and BM25: helps exact terms, while literal matching alone may miss paraphrased policy language. Use both when warranted.
• More metadata filters: narrow candidates but can accidentally exclude valid evidence due to missing fields or wrong types. Keep mandatory authorization invariant.
• Evidence aggregation: independently retrieved facts may disagree across effective dates or policies; resolve precedence before answering, and cite each authoritative version.
• Optimizing answer accuracy alone: an LLM can guess correctly without retrieving necessary evidence. Track retrieval-stage coverage and supported citations independently.

8. Prevent recurrence

• Maintain a length- and facet-stratified golden set: short questions, verbose single-intent questions, multi-hop comparisons, long negated queries, exact policy codes and queries with key clauses at the end.
• Add deploy-time probes for embedding context limits, truncation flags, rewrite constraint loss, search filters, per-branch candidate pool sizes, reranker input length and citation coverage.
• Alert on performance gaps between short and long queries, position-sensitive misses, per-facet missing evidence, p99 and request cost, with tenant/security segmentation.
• Perform targeted fault injection: forcibly truncate the last 20% of the query, drop one subquery, invert a negation or lower rerank depth; the diagnostic traces should identify the first failed stage.
• Review a sample of long-query production failures weekly and classify them as input loss, constraint parsing, representation, candidate coverage, fusion/rerank or synthesis. Tie each fix to a reproducible experiment.

Interview takeaway: Debug long-query RAG failures by separating input length from number of intents, verifying the exact embedded input and all mandatory constraints, then tracing facet-level candidate survival. Use decomposition, monolithic retrieval, hybrid search or constraint-aware reranking only after the first failing boundary has been demonstrated.

Research references

• Ammann et al., *Question Decomposition for Retrieval-Augmented Generation*, ACL SRW 2025: https://aclanthology.org/2025.acl-srw.32/
• Petcu et al., *Query Decomposition for RAG: Balancing Exploration-Exploitation*, EACL 2026: https://aclanthology.org/2026.eacl-long.322/
• Yin et al., *When Should Queries Be Decomposed? A Stage-Aware Study of Query Decomposition for Multi-Condition Retrieval*, EMNLP 2026: https://aclanthology.org/2026.findings-emnlp.1644/
• Microsoft, Azure AI Search query rewriting and caveats: https://learn.microsoft.com/en-us/azure/search/semantic-how-to-query-rewrite
• Microsoft, RAG information-retrieval guidance: https://learn.microsoft.com/azure/architecture/ai-ml/guide/rag/rag-information-retrieval