A query planner routes questions to the wrong knowledge source. How would you identify routing failures and design fallback behavior?

Production incident (illustrative): An enterprise RAG assistant has four knowledge sources: HR-POLICIES (leave, benefits and employee travel); FINANCE-CONTRACTS (vendor payments and contractor reimbursements); SECURITY-RUNBOOKS (access incidents); and PRODUCT-DOCS (technical support). A user asks, “Who approves reimbursement for contractors travelling to Singapore under the 2026 vendor policy?” The query planner routes solely to HR-POLICIES, because it matches “travel” and “reimbursement.” That index returns a credible *employee* travel-policy excerpt, and the assistant confidently cites it. The authoritative answer is in FINANCE-CONTRACTS, with a different approver. Source availability and HTTP success are normal: the incident is a wrong-source routing decision masquerading as successful retrieval.

Core principle: In multi-source RAG, distinguish source selection from document retrieval within a source. Before tuning embeddings, ask: Did the planner consider the correct permitted source? Was it selected? Was a query actually sent? Did the correct source return authorized evidence? Was that evidence later dropped during fusion/reranking? A wrong answer may arise at any of these boundaries. A reliable fallback must *preserve authorization and source authority*, not just fetch more documents.

1. Contain the incident and establish ground truth

• Stop confidently answering policy questions using evidence from a source outside the requested domain. Require citations to an authorized, current, domain-appropriate source; if no such evidence exists, abstain or ask for disambiguation.
• Capture request trace: redacted user question, conversation context, tenant and principal, source registry version, source descriptions, allowed-source list, generated query plan, selected sources, rejected sources/reasons, per-source subqueries, tool arguments, responses, candidate IDs, reranker outputs and citations.
• Check source ownership with business SMEs. For the incident, vendor contractor reimbursements belong in FINANCE-CONTRACTS; employee reimbursements belong in HR-POLICIES. Source ownership is a maintained contract, not a guess inferred from a few overlapping keywords.
• Validate the target evidence in the authoritative source, its effective year, region, document version and ACL. Never debug by querying a forbidden tenant or by relaxing mandatory source permissions in production.
• Replay the same sanitized question with a frozen source registry, corpus snapshots and fixed user permissions. Verify whether the route changes with conversation history, source-description edits, planner model versions or load.

2. Trace the first routing failure boundary

• A. Source registry: Enumerate permitted sources and their purpose, owner, freshness SLO, regions, document types, search capabilities, sensitivity tier and schema. Is the correct source registered, healthy and *visible to the planner*? A disabled or stale source registry can force a misleading alternative.
• B. Source descriptions: Audit titles and retrieval/tool descriptions. Vague descriptions like “Search documents” cause overlap; specify unique data ownership (“Vendor contract terms, supplier invoices, contractor reimbursements; not employee HR travel policy”). Microsoft Azure Architecture guidance explicitly recommends precise tool descriptions and typed parameters.
• C. Intent extraction: Inspect planner interpretation of role, issuer, jurisdiction, date, named policy and required facts. “Contractor” must not be rewritten to “employee.” Test omitted chat-history context, ambiguous acronyms, negation, polysemy, and the planner's confidence/calibration.
• D. Candidate source selection: Log the planner's eligible source candidates, ranks/scores when available, selected source set and budget. Check whether source_top_k=1 forces a choice between two relevant domains or whether hard-coded precedence overrides the classifier.
• E. Tool invocation: A source may be selected but not queried because of a failed tool call, bad parameter, wrong index alias, missing tenant filter, timeout, rate limit, or query-plan branch pruning. Selection logs alone are not proof of executed retrieval.
• F. Per-source evidence: Inspect each source's returned IDs, scores, document versions, eligibility, citations and completeness flags. If the correct source is queried but yields no evidence, this is an in-source retrieval/indexing issue, not necessarily a planner error.
• G. Merge/rerank: If the correct source returns evidence but it vanishes before the answer, investigate cross-source score comparability, fusion windows, reranker input limits, source dominance, deduplication and context packing. Do not blame routing when the source was in fact queried successfully.
• H. Answer provenance: Check whether the final response cites the requested authority or merely a topically similar source. Validate domain ownership and source-version requirements; citations alone do not prove the cited policy governs the user.

3. Root-cause decision tree and controlled experiments

• Correct source not in allowed registry: configuration, permissions or registry freshness defect. Repair registration and authorization metadata; do not route around access control.
• Correct source allowed but not selected: routing classifier/LLM planner, vague descriptions, missing role/context, low source selection budget or over-specific intent thresholds. Compare deterministic rules, descriptions and source ranking on labelled queries.
• Correct source selected but tool not called: planner-to-executor orchestration bug, wrong tool schema, timeout, routing guard or erroneous parallel call limit. Inspect execution activity and retry only when idempotent and safe.
• Correct source queried but no useful evidence: source-specific search, index freshness, filters or query transformation issue. Run known-ID and exact/lexical retrieval within that source; don't mask missing evidence by selecting an unrelated source.
• Correct source evidence retrieved but answer cites wrong source: fusion, rerank, source precedence, context assembly or grounded-answer verification failure. Add domain authority constraints and citation checks.
• Router selects different sources for equivalent queries: test paraphrases against source labels; review non-deterministic planning, changing registry snapshots, history and replica/tool availability separately.
• Query actually spans domains: use a *bounded multi-source plan* with subqueries for each required fact. Prevent a single-source planner from hiding cross-domain requirements.
• Source data conflicts: rank by explicit business authority, applicability and effective version, not merely semantic score; surface unresolved conflicts rather than inventing a unified rule.

4. Worked numerical incident

• Assemble 500 expert-labelled enterprise questions, with allowed sources and expected authoritative source labels. Assume 300 single-source HR, 100 single-source finance, 50 security and 50 genuinely cross-source questions.
• Initially, the router correctly includes 450/500 = 90% of the needed source sets. However, a broad aggregate masks finance failures: for 100 finance-only queries, it selects FINANCE-CONTRACTS in 65 cases = 65% source-selection recall; it often picks HR-POLICIES for contractor travel questions.
• For the failing question, a planner trace shows HR-POLICIES selected at confidence 0.83 and FINANCE-CONTRACTS at 0.69, but source_top_k=1 drops finance. HR returns an employee policy with similarity 0.91, which is irrelevant for a *contractor*. These scores are illustrative and not comparable to calibrated probabilities without validation.
• Expanding the *authorized* candidate-source shortlist to 2 and adding a rule to keep FINANCE-CONTRACTS eligible for vendor/contractor payment intent recovers the correct source in 94/100 finance-only questions. A domain-aware reranker and authority verifier retain the finance citation rather than the HR excerpt.
• On the same 500-question test, suppose complete required-source recall rises from 450/500 = 90% to 485/500 = 97%, an absolute +7 percentage points. Finance-only source recall rises from 65% to 94% (+29 points). Of the 15 remaining misses, classify failures by unregistered sources, ambiguous role terms, source timeouts and source-specific retrieval, rather than calling all of them router mistakes.
• Average per-request source calls increase from 1.1 to 1.6, and retrieval p95 rises from 160 ms to 230 ms. These hypothetical numbers illustrate the quality-versus-latency trade-off, not observed MachinoAI traffic. Validate source costs, permissions, accuracy and downstream citation support before rollout.
• Crucial distinction: a route recall gain does not prove final evidence recall or correctness; calculate source-selection recall, retrieval success *conditional on correct source*, and authority-supported answer rate separately.

5. Safe fallback design

• Primary route: parse request into a structured intent contract (domain, entity_type, role, jurisdiction, effective_date, document_type, requires_multiple_sources) and enforce an authorized source allowlist from trusted policy.
• Bounded shortlist: where source ownership is ambiguous, select the top 2–3 permitted sources or route to specific rule-based source sets. Use source descriptions/ownership to aid routing, not as a replacement for trust boundaries.
• Evidence sufficiency gate: evaluate whether returned evidence is current, authorized, domain-applicable, supports the requested facts and passes an answer-evidence threshold. A high vector score from the wrong domain should not satisfy the gate.
• Retry same source when indicated: when source selection is correct but retrieval is weak, first try a controlled lexical/hybrid query, validated rewrite or wider authorized candidate budget in the *same authoritative source*. A generic cross-source fallback can conceal an indexing incident.
• Fallback to alternate allowed sources: if ownership is uncertain or the authoritative source is temporarily unavailable, query only sources explicitly permitted for this request. Mark alternate evidence as supporting, never silently promote it over a required legal/policy authority.
• Cross-source questions: fan out bounded subqueries to sources that own the relevant facts, preserving tenant/ACL/date filters and merging evidence with provenance and authority precedence.
• Graceful abstention: if required authoritative data remains unavailable, state the limitation or offer the approved human escalation path. Never answer a finance policy from HR merely because the latter returns a confident passage.
• Circuit breakers and budgets: cap source calls, retries, total latency and token spend; distinguish timeout/no-result/unauthorized/source-not-configured. Retries must not create runaway orchestration loops.
• Kill switch: maintain versioned source registry and router, route-level feature flags and rollback to a tested policy. Verify rollback respects current deletions and permissions.

6. Engineering implementation and observability

• Maintain a versioned source registry describing business domain, owner, supported query classes, allowed principals, indexing/refresh SLA, authoritative precedence, source ID and schema. Include accurate negative descriptions (“does not govern contractors”) to reduce false route selection.
• Use typed tool schemas and explicit validation for query, filters, top_k, source_id and timeouts. Never let user-provided natural language select internal source identifiers without authorization checks.
• Emit a trace per request: router_model_version, registry_generation, selected_sources, selection_reason, eligible_sources, source_call_status, source_latency, source_evidence_ids, source_authority, fallback_trigger, fallback_sources and final_citation_ids.
• Instrument source-selection Recall@k, false-route rate, wrong-authority citation rate, required-source coverage, fallback precision, retriever Recall@k conditional on correct route, p95/p99 and source tool error rate by domain, tenant and ambiguity class.
• Evaluate three *separate* layers: (1) route chosen; (2) evidence returned given route; (3) final grounded answer. A single end-to-end accuracy score cannot localize a failing planner.
• Build a held-out labelled routing suite with overlapping language, multi-source needs, negation, acronyms, contractor-versus-employee, sensitive domains, source outages, stale version and ambiguous user permissions. Use authoritative domain-owner labels.
• Shadow-test new router versions against the same authorized inputs and active source registry; then canary with strict wrong-authority and unauthorized-source guardrails. Compare source-call cost, latency and final citation quality, not just classifier accuracy.
• Azure AI Search agentic retrieval documents knowledge-source selection, query planning, execution activity and source references. Its source settings such as alwaysQuery are vendor-specific: they can ensure particular sources are queried, but should not be configured globally for sensitive or irrelevant sources.

7. Trade-offs and common mistakes

• Top-1 routing: cheapest and fastest, but brittle for overlapping enterprise domains or mixed intent. Top-N can recover the authority at higher search cost and greater opportunity for noisy evidence.
• Always-query policies: appropriate for a mandatory compliance authority when authorized, but expensive and possibly unsafe when applied indiscriminately. Source visibility and permissions still govern every call.
• Rule-based routing: auditable for known IDs and clear policy domains, but needs continual maintenance as products and departments change.
• LLM/semantic planner: flexible for ambiguous natural language, but tool descriptions, context order, prompt drift and source registry updates change decisions. Use deterministic guardrails for mandatory source ownership and ACL.
• Fallback on zero results: may help when source availability is uncertain, but can convert a retrieval/indexing defect into a plausible wrong-source answer. Prefer an evidence/authority-aware fallback trigger.
• Confidence thresholds: planner scores may not be calibrated across source classes; test selective routing/abstention and per-domain thresholds on representative data.
• Cross-source aggregation: improves complete-evidence coverage but introduces conflicting versions and scoring incomparable across sources. Rerank with source-level provenance and explicit precedence.
• Source outage: an unauthorized or non-authoritative alternative is not an acceptable fallback; show an appropriate limited-service response.
• Version rollback: an old router may send traffic to retired indexes or sources. Roll back a consistent versioned configuration while retaining current security/retention rules.

8. Prevention checklist

• Treat each new source integration, source-description update, tool-schema change or source-owner change as a routing release with labelled regressions and rollback.
• Add counterfactual route tests: “employee travel reimbursement” vs “contractor travel reimbursement,” “security access policy” vs “IT access troubleshooting,” and invoices vs vendor contracts; check that one-word changes cause *appropriate*, not accidental, routing changes.
• Continuously reconcile source registry against actual available indexes/tool endpoints, latest document versions, ACL requirements, owner changes and source health. Alert on unknown-source and failed-route attempts.
• Monitor sudden wrong-source citation spikes, large increases in fallbacks, source-selection recall declines, tool timeouts and domain-specific query shifts.
• Inject controlled failure scenarios: authoritative source returns zero results; planner source description is vague; a source times out; the same query needs both finance and legal facts; a user's permissions change mid-session; a retired source still exists in the registry.
• Require an evidence sufficiency and authority gate before answer synthesis. If the required source is not reachable or evidence is incomplete, abstain and record the reason.
• Review source-selection errors with source owners and make routing manifests, policy changes and fallback decisions auditable.

Interview takeaway: To diagnose wrong-source routing, trace allowed sources → planner choices → executed source calls → per-source evidence → citation authority. Fix the *first failing boundary*. Design fallback as bounded, authorized, authority-aware retrieval with abstention, not an unrestricted search across every database.

Research references

• Microsoft, *Develop an Agentic RAG Solution on Azure*: https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/rag/rag-agentic
• Microsoft, *Agentic Retrieval Overview*: https://learn.microsoft.com/en-us/azure/search/agentic-retrieval-overview
• Microsoft, *What is a Knowledge Source?*: https://learn.microsoft.com/en-us/azure/search/agentic-knowledge-source-overview
• Microsoft, *Query Knowledge Base via API or MCP*: https://learn.microsoft.com/en-us/azure/search/agentic-retrieval-how-to-retrieve