How do you distribute retrieval across multiple nodes?

Distributed retrieval in RAG runs searches on multiple retrieval nodes and combines their results before generating an answer. Unlike sharding—which decides *where index data lives*—distributed retrieval focuses on query execution, coordination, fault handling, and globally correct result ranking.

Beginner-friendly example

Imagine 12 million chunks spread across 3 search nodes, each owning 4 million. A user asks: “Which deployment guide explains rolling restarts?”

1. A query coordinator authenticates the user, embeds the query, and applies access constraints.
2. The coordinator sends the query to the relevant nodes in parallel.
3. Each node searches its local index and returns its top 10 authorized candidates with document IDs and similarity scores.
4. The coordinator merges up to 30 candidates, deduplicates repeated documents, and selects or reranks the global top 5.
5. The generation service receives only the strongest source passages, then answers with citations.

This is a simplified flow. Returning only five from each node could exclude useful global candidates when deduplication or a later reranker changes the ordering.

Key components

• Coordinator/router: Selects shards and replica targets, handles authentication, sends parallel requests, and maintains a query deadline.
• Search workers: Hold or access local vector and keyword indexes and produce ranked local candidates.
• Replica selection: Chooses an available copy of each required shard for redundancy and load balancing.
• Aggregation layer: Deduplicates, merges compatible scores, applies fusion/reranking, and enforces final evidence limits.
• Telemetry: Records per-node latency, timeouts, selected shards, result counts, and final answer quality.

Important design decisions

1. Selective routing vs scatter-gather: Search one tenant/region shard when valid routing allows it; otherwise fan out to every relevant shard.
2. Preserve retrieval recall: Fetch sufficient candidates per shard. Final global Top-K can change after reranking or merging keyword and vector results.
3. Compare scores correctly: Raw scores from incompatible embedding spaces or search methods cannot simply be mixed. Normalize/calibrate or use rank fusion and reranking.
4. Handle stragglers: Set timeouts and latency budgets, use load-aware replicas, and avoid waiting indefinitely for a slow node.
5. Define partial-result behavior: If a shard fails, a partial answer might be acceptable for low-risk discovery, but should be labeled incomplete. For compliance or safety-critical questions, fail closed rather than hiding missing evidence.
6. Enforce access control: Apply tenant and document permissions at each search node, not just after aggregation.
7. Keep indexes consistent: Coordinate versioned updates, deletions, shard placement, and replica lag. Stale replicas can return outdated evidence.

Numerical latency example

Suppose three nodes finish their searches in 35 ms, 50 ms, and 90 ms. With parallel fan-out, retrieval completion is roughly driven by the slowest required node (90 ms), plus network and aggregation overhead—not the sum of 175 ms. Actual production p95 latency also depends on scheduling, retries, and coordinator load.

How to evaluate it

Load-test different shard counts and replica placements using realistic traffic. Track global recall@K, p95/p99 latency, queries per second, network fan-out, missing-shard rates, duplicate suppression, and end-to-end grounded answer quality.

Interview takeaway: Distributed retrieval means route → search nodes in parallel → merge and rerank → handle failures → ground the answer. Sharding supplies the partitions; query orchestration turns them into a reliable search service.