How should results from multiple retrievers be deduplicated?

Deduplication in multi-retriever RAG means combining results from lexical, vector, graph, or other search methods without repeating the same source passage. The objective is to preserve useful evidence and ranking signals while avoiding redundant context for the LLM.

How should it work?

1. Collect candidates: Run the authorized retrievers and collect each chunk's ID, document ID, score, rank, and source metadata.
2. Use stable identities: When the *same indexed chunk* appears in multiple lists, merge it using a canonical key such as tenant_id + document_id + document_version + chunk_id.
3. Keep retrieval signals: Record which retrievers returned the chunk and its score or rank in each list. Those signals can inform Reciprocal Rank Fusion (RRF) or calibrated score fusion.
4. Detect near-duplicates carefully: For different chunk IDs containing substantially overlapping text, compare normalized text, content hashes, or similarity. Keep important differences such as exceptions, dates, and document versions.
5. Fuse and rerank: Combine eligible candidates, rerank the unique evidence, and select passages within the context budget.
6. Preserve provenance: Retain source IDs, citations, version information, and access restrictions in the merged record.

Retriever A + Retriever B → canonical IDs → merge duplicates → fusion/reranking → unique evidence

Practical example

For “How do I fix PAY-409?”, BM25 returns chunks A, B, C and vector search returns B, D, A.

The combined list should be A, B, C, D—not six separate chunks. Both A and B retain evidence that two retrievers selected them, so an RRF-based fusion stage can reward their rankings appropriately.

Exact versus near-duplicate content

• Exact result duplicates: The same tenant, source version, and chunk ID. Merge deterministically.
• Duplicate text from different sources: Do not automatically erase provenance. Multiple sources may matter for authority or citation.
• Near-duplicates: Similar passages or overlapping chunks. Consider context-level redundancy removal, but avoid deleting unique conditions or facts.

Common pitfalls

• Deduplicating solely by plain text can collapse different versions, sources, or security scopes.
• Removing duplicate list entries before recording their retriever ranks loses useful fusion information.
• Summing raw BM25 and vector scores ignores incompatible scales.
• Running similarity-based deduplication across tenants can cross access boundaries; filter by authorization first.
• Aggressive similarity thresholds may drop the only passage containing a critical exception.

Interview takeaway

Deduplicate exact multi-retriever hits by stable source-and-chunk identity, preserve each retriever's ranking signals for fusion, and remove near-duplicate context cautiously. Always retain permissions, version information, and citation provenance.