Your hybrid search returns many duplicate passages and suppresses source diversity. How would you diagnose the fusion and candidate-selection stages?

Production incident (illustrative): A compliance RAG assistant must answer, “Which 2026 travel reimbursement rules apply to Singapore contractors, and what is the exception approval path?” The dense retriever returns seven near-identical chunks from the same global travel policy, while BM25 returns six copies of that policy plus one relevant regional-exception memo. Hybrid fusion presents eight passages, but seven originate from one policy revision and the regional exception never reaches the LLM. The assistant gives a plausible but incomplete answer. Ingestion is healthy and the retriever has nonzero Recall@10, so the operational failure is source concentration and candidate-selection loss, not necessarily missing documents.

Principle: Hybrid retrieval balances semantic and lexical evidence; RRF/score fusion does not automatically guarantee passage uniqueness, independent sources, or coverage of all required facets. Diagnose at the point/chunk → logical passage → document version → authoritative source levels, then inspect where relevant minority evidence is crowded out. Preserve authorization, current-version and effective-date eligibility while testing diversity.

1. Reproduce and instrument the incident

• Capture a fixed query, user permissions, tenant, corpus snapshot, index alias, query embedding, BM25 text/analyzer, authorized filters, source-effective date, search parameters, per-branch candidate counts, rank window, fusion settings, reranker input and final cited passages.
• Collect *every stage* separately: dense candidates before fusion; BM25/sparse candidates before fusion; unified identities; fusion scores/ranks; dedup/grouping decisions; rerank results; final top-k; and answer context/citations.
• Tag each result with point_id, chunk_id, parent_document_id, source_id, document_version, section_id, content_hash, ingestion_generation, effective_date, and ACL classification. A point ID alone is insufficient to detect replicas of the same passage under different IDs.
• Verify *what counts as a genuinely distinct source*. Different neighboring paragraphs may be valuable complementary evidence even from one policy; mirrored copies and repeated chunk overlap are usually not.
• Build a labelled evidence checklist for multi-source questions: e.g. global reimbursement cap + regional exception + approval authority. Measure whether at least one authorized, active chunk supports each required fact. Do not force unrelated sources just to increase diversity.
• Distinguish *source-level deduplication* from *source relevance*: an answer requiring one authoritative document may legitimately benefit from several nearby chunks. Diversification should be conditional on user intent and coverage requirements.

2. Root-cause debugging: find the first stage that loses diversity

• A. Source-to-index duplication: inspect repeated ingestion jobs, unstable chunk IDs, overlapping chunk windows, archival copies, identical text across versions and mirrored connectors. Compare exact content_hash plus source/version lineage. If source copies are indexed repeatedly with different IDs, fix ingestion idempotency and versioning first; re-ranking cannot permanently cure this.
• B. Document representation: detect whether one long policy was split into hundreds of largely overlapping chunks while regional memos have only one each. Per-chunk retrieval can favor the verbose source because it receives more opportunities to match. Check chunk granularity, overlap, parent IDs and which passages truly answer distinct facets.
• C. Branch candidate pools: compare the unique source_id counts of dense versus BM25/sparse. Determine whether all branches independently favor the same source, or whether the regional memo appears in BM25 rank 12 but is clipped by branch_top_k=10.
• D. Identity reconciliation: review deduplication keyed by record ID rather than canonical source+version+content. The same paragraph emitted under different point IDs or both lexical and vector branches may count as distinct when it is not. Conversely, collapsing everything by parent document *too early* can discard two distinct relevant sections.
• E. Fusion semantics: inspect exactly which branch result ranks/scores fed the fusion stage, rank_window_size, rank_constant, branch weights, score normalization, top_k and the point of truncation. RRF boosts documents appearing near the top in *multiple* branches; this improves consensus but is not a diversity constraint. A source with many similar chunks can still dominate.
• F. Normalization and version merging: inspect whether relativeScoreFusion is skewed by score-range outliers, whether lexical score dominates, and whether a reranker or fusion groups active and superseded versions together. Weaviate documents distinct rank-based and relative-score fusion methods; don't assume their outputs are interchangeable.
• G. Selection/diversification: ask if source-aware caps, MMR (maximal marginal relevance), coverage quotas or parent grouping are applied after an adequately wide candidate pool but before irreversible top-k truncation. If a memo is already discarded at branch retrieval, no downstream MMR can recover it.
• H. Context packing: inspect whether final packing pulls several overlapping passages from the same parent, exhausting token budget before an exception appears. This is downstream of retrieval; log both candidate and actually included context.
• I. Eligibility: verify the missing regional memo is active, authorized and effective for the query. Do not include a revoked or unrelated source simply to meet a “diversity” target.

3. Production metrics: distinguish duplicates, source diversity, and correctness

• Unique-source ratio@k = number of distinct relevant/eligible source_id in the final k ÷ k. With chunk-based search, this is a diagnostic: it can be low on a correctly answered single-source question.
• Duplicate-passage rate@k = redundant or canonically equivalent passages beyond the first copy ÷ k. Define redundancy by normalized text/content hash and source/version/section identity; near-duplicate detection uses bounded similarity and human-reviewed thresholds.
• Largest-source share@k = maximum count of passages from one source ÷ k. A high share can indicate domination, but apply query-facet context before alerting.
• Evidence-facet coverage = required relevant facets supported by at least one authorized chunk ÷ total required facets. Use *complete-evidence query rate* where every required facet must be supported, not just mean facet recall.
• Source-aware Recall@k: track whether authoritative relevant *sources* appear within candidate/final sets, separately from passage-level Recall@k. Deduplicating many identical paragraphs may raise source coverage without changing vector similarity.
• MRR/nDCG and citation support: track relevance and quality simultaneously. Diversity that replaces the top authoritative evidence with irrelevant documents is not a win.
• Marginal context utility: after adding a passage, how much new evidence does it contribute, measured by an annotated facet or fact, not only its embedding distance?
• Latency/complexity: reranker input count, candidate-memory overhead, multi-source scoring, deduplication CPU, p95/p99, prompt tokens, generated wrong-source citations and cross-tenant leakage.

4. Numerical example: trace the stage that discards the memo

• For one query, dense top-10 includes 8 chunks from global policy G, 1 from FAQ F, 1 from exception memo E. BM25 top-10 includes 8 chunks from G and 2 from F, with E at BM25 rank 13 (outside the BM25 candidate window).
• Although E appears in the dense branch at rank 10, final fused top-8 has 7 G chunks and 1 F chunk; E disappears. Unique-source ratio@8 = 2/8 = 25%. Largest-source share = 7/8 = 87.5%.
• Required evidence facets are G: reimbursement cap, E: regional exception, and A: approval authority. The current final context supports only G, so facet coverage = 1/3 ≈ 33.3%. Another support chunk A was never retrieved within either branch's top-10.
• A simple RRF illustration with k=60: a G passage at dense rank 1 and BM25 rank 2 scores 1/(60+1)+1/(60+2) ≈ 0.03226. E at dense rank 10 and absent from the BM25 top-10 scores 1/(60+10) ≈ 0.01429. RRF naturally ranks the consensus G chunk higher; this is not a defect in the RRF formula. In actual systems, use the vendor's rank and window conventions.
• Controlled correction: expand each branch from top-10 to top-30, unify passage identities, preserve distinct answer-bearing sections, perform source-aware grouping and a measured relevance-diversity rerank. The wider BM25 pool now contains E (rank 13), and a relevant approval chunk A appears at dense rank 18.
• Suppose final top-8 now contains 3 G, 2 E, 2 A, 1 F passages. Unique-source ratio@8 = 4/8 = 50% and largest-source share = 3/8 = 37.5%; facet coverage = 3/3 = 100%. These values are *illustrative*, not observed production metrics, and selected chunks must be actual authorized evidence.
• On a held-out 200-query set of multi-source policy questions, imagine baseline complete-evidence coverage 118/200 = 59%. After the tested change it becomes 170/200 = 85%, an absolute +26 percentage points. Rerank p95 rises from 130 ms to 205 ms, an additional 75 ms. Roll out only after assessing citation correctness, security and SLOs.
• Note: source diversity and facet coverage are not interchangeable. Four distinct sources can all repeat the same reimbursement cap; three chunks from one source can contain three different necessary rules.

5. Engineering remediation

• Canonical identity layer: distinguish point_id, logical passage, source revision and canonical source. Derive stable content/version hashes, reconcile duplicate documents, preserve the current authoritative copy and respect tombstones/ACLs.
• Source-aware fusion diagnostics: log rank positions, normalized scores and fusion contributions per candidate and branch; make it possible to say “memo E was present in dense rank 10, then pruned after fusion.” Keep deterministic tie-breaks where supported and pin all windows/configuration.
• Increase candidate coverage before diversification: widen BM25 and dense branch pools based on benchmarked recall, not arbitrarily. Elasticsearch RRF rank_window_size constrains which branch results reach fusion, and Qdrant hybrid prefetch limits are explicit.
• Deduplicate at the correct layer: collapse exact same passage across dense/BM25 branches as one candidate; use document/source canonicalization to identify near-copy variants without collapsing genuinely complementary sections.
• Constrain source dominance when needed: after a sufficiently wide pool, use a tested source cap or group-by with exception handling for one-source questions. For multi-facet tasks, prefer evidence coverage targets over a blind cap of one chunk per document.
• Diversify with MMR or other relevance-diversity selection as supported by the backend or application. Elasticsearch's diversify retriever implements MMR over an inner result set. A high diversity setting can harm relevance, so benchmark across single-source and multi-source intents.
• Rerank remaining candidates against the original question and source-effective constraints. Preserve source section and document version for citations, then pack a bounded evidence set by novel facet contribution.
• Validate backend semantics: Elasticsearch RRF supports collapse but not all retriever compositions or rescore operations; inspect your installed version and the supported order of grouping/reranking. In Weaviate, relative-score and ranked fusion use different score treatments; pin the selected strategy.
• Deploy safely: use a feature flag, shadow comparisons, then a narrow canary. Gate on complete-evidence query rate, source-aware Recall@k, wrong-citation rate, duplicate rate, p95/p99 and zero unauthorized sources. Retain a clean rollback path and consistent versioned index.

6. Trade-offs and failure scenarios

• Ingestion deduplication vs retrieval-time deduplication: fixing unstable source IDs reduces future storage and noise; retrieval-time dedup still helps with overlapping chunks, mirrored content and dual-branch appearances.
• Document-level collapsing too early: may remove multiple answer-bearing sections or cause outdated version selection. Use effective revision and section-aware identity first.
• One-source-per-result rule: improves surface diversity but can discard essential context when a policy has several independent rules. Cap intelligently or use facet-aware selection.
• MMR: reduces redundancy but adds pairwise similarity computation and tuning; maximizing novelty alone can surface irrelevant sources.
• Wider candidates: improves opportunity to retrieve missing sources but raises memory, search latency and reranking costs. If E is not indexed or unauthorized, no branch-window change can recover it.
• RRF consensus bias: a passage found in multiple branches is promoted; this is a useful signal in many cases, not an intrinsic bug. Relevance-aware balancing and evidence coverage matter more than penalizing overlap automatically.
• Relative-score fusion: can preserve score differences but needs comparable scale handling and is sensitive to branch score distributions; rank-based methods have different failure modes.
• Rerank and context packing: a high-quality reranker can still select repetitive evidence if its objective scores each chunk independently. Novel-evidence rewards and citation constraints can help.
• Metrics and fairness: high source diversity isn't always beneficial; segment by task, authority, tenant and multi-document need. Do not count unauthorized or obsolete sources as useful “diversity.”

7. Prevent recurrence

• Maintain gold queries that need (a) one authoritative source with multiple sections, (b) multiple sources to satisfy separate facets, (c) a regional exception overriding a global rule, and (d) exact-version documents. Include mirrored copies, overlapping chunks and repeated ingestion.
• Instrument branch depth → unique passage count → unique source count → fused source share → reranker source share → final facet coverage, plus candidate removal reason codes. Alert when a source unexpectedly dominates multi-source query classes or labelled exception evidence disappears.
• Add stable canonical IDs and content hashes to ingestion; audit new chunking/version strategies for changes in per-source chunk count and skew.
• Compare RRF windows, branch limits, score-fusion method, grouping order, deduplication thresholds and reranker settings as a single versioned retrieval manifest; reject changes that reduce labelled coverage or cite superseded policies.
• Run simulated duplicate amplification tests: insert 5–20 overlapping variants of one policy, then verify a known minority exception source remains retrievable and cited.
• Keep end-to-end authorization and policy-effective-version checks at both retrieval and citation time. A diversity feature must never include a source merely because it is novel when that source is forbidden or out of date.
• Review production incidents by which stage first eliminated a relevant distinct source, then fix the causal stage. Don't tune one score threshold blindly.

Interview takeaway: When hybrid retrieval is dominated by repetitive passages, inspect branch candidates and source identities before and after fusion. Deduplicate equivalent passages, widen candidate pools if minority evidence was clipped, and select for relevant, authorized evidence coverage, using grouping or MMR carefully. Diversity is a means to complete, cited answers—not the goal itself.

Authoritative references

• Elasticsearch, Reciprocal rank fusion and rank windows: https://www.elastic.co/docs/reference/elasticsearch/rest-apis/reciprocal-rank-fusion
• Elasticsearch, Diversify retriever (MMR): https://www.elastic.co/docs/reference/elasticsearch/rest-apis/retrievers/diversify-retriever
• Qdrant, Hybrid queries and RRF/DBSF: https://qdrant.tech/documentation/hybrid-queries/
• Qdrant, Hybrid search and reranking: https://qdrant.tech/documentation/tutorials-basics/reranking-hybrid-search/
• Weaviate, Hybrid search and relative/ranked fusion: https://docs.weaviate.io/weaviate/concepts/search/hybrid-search