One vector-search shard consistently responds much slower than the others. How would you identify and mitigate the hot shard?

Production incident (illustrative): A customer-support RAG system searches a 12-million-vector collection across six shards. At peak load, the search coordinator waits nearly a second for shard 4 while the other shards return in roughly 60–90 ms. Search p99 rises, and requests time out even though the cluster-wide average CPU looks normal. A proposal to add two nodes misses the central question: why is one shard hot?

Core principle: A hot shard can result from disproportionate query or write traffic, larger indexed data, expensive metadata filters, a slow host, or a lagging/compacting replica. Identify whether the problem follows a *logical shard*, a *physical node*, or a *query class* before moving data.

1. Contain the production incident

• Quantify affected search requests, p95/p99, timeout and error rates. Preserve representative traces and the placement map: collection → shard_id → replica_id → node_id.
• Keep tenant and ACL filters enabled. If safe, slow down background ingestion/optimizers, bound expensive-query concurrency, or route reads to healthy, sufficiently current replicas that meet the required consistency and authorization guarantees.
• Avoid immediately resharding or restarting the slow node. Both can increase disk I/O, trigger cold-cache penalties, or hide the original cause.
• Check whether a single slow shard actually lies on the critical path. Fan-out search often waits on the slowest participating shard, but execution and timeout semantics depend on the database.

2. Trace the bottleneck at shard and request level

• Instrument spans for client wait → coordinator planning → shard fan-out → per-replica queue → filter planning → ANN traversal → payload fetch → merge/rerank. Tag collection, shard, peer, request class, tenant (safely aggregated), top_k, filter selectivity, index parameters, retries, queue time and source/version eligibility.
• Compare per-shard QPS, read/write ops, active points, bytes/index footprint, HNSW and payload-index sizes, deleted-point ratio, optimizer progress, CPU throttling, RSS/page faults, memory pressure, disk read/write latency, cache misses and p50/p95/p99.
• Inspect data distribution: Did consistent hashing place a large tenant's records together? Does a user-defined shard key route almost all requests to one shard? Does a time-based scheme send nearly all recent-data queries to today's shard?
• Cross-check workload shape: A shard can contain the *same number of vectors* as its peers but receive many more queries, expensive predicates or high write/compaction volume.
• Examine replica placement and hardware: one slow replica may have degraded SSD I/O, CPU contention, poor network, stale index/cache or compaction backlog. Compare another replica of the same logical shard.
• Verify whether shard-selection keys are present. In Qdrant user-defined sharding, a query without the intended shard selector may search all shards; that broadens fan-out and can overload the system. Do not route solely on untrusted tenant input.

3. Root-cause decision tree

• Slow only on one physical peer: compare the same shard replica on other peers, hardware, cache, local indexing and maintenance; a targeted replica replacement/move may help.
• Slow logical shard across replicas: examine shard volume, dominant tenant/QPS, filter complexity, hot keys and compaction; moving its data unchanged may only move the hot spot.
• High queue wait, normal execution time: investigate burst concentration, connection pools, CPU scheduling and request routing. Cap or partition concurrency before raising hardware capacity.
• High ANN/filter execution time: inspect ef_search, top_k, selectivity, missing payload indexes and the query mix. Preserve retrieval recall and mandatory tenant/ACL filtering.
• High I/O and page-fault time: measure hot index working set, memory tier, cache churn, SSD latency and co-located ingestion. Test maintenance throttling or better I/O isolation.
• Write or optimizer hotspot: monitor update distribution and maintenance queue growth. Separate write-heavy tenants or stagger ingestion without violating freshness budgets.
• Network/coordinator bottleneck: inspect retries, cross-zone traffic and per-shard fan-out; shard movement won't cure a congested coordinator.
• Rule out misleading comparisons: use the same query mix, observation period, replica consistency setting and filters when contrasting shards.

4. Practical numerical example

• Six shards each hold 2 million vectors, giving 12 million total. Balanced storage does not guarantee balanced throughput.
• At peak, the cluster handles 1,200 searches/second. Shard 4 receives 600 shard-targeted searches/second (50%), while each of the other five receives 120 (10%). The average is 200/shard/second; shard 4's traffic is 3× the average and 5× each other shard.
• A fan-out or targeted-query trace shows shard 4 queue wait 650 ms + filter/ANN work 190 ms + payload fetch 60 ms = 900 ms. Another shard takes 15 + 55 + 10 = 80 ms. In requests requiring both shards, coordinator latency cannot beat the ~900 ms straggler, even before merge/network overhead. These are *individual illustrative traces*, not summed p99 percentiles.
• Logs reveal that one enterprise tenant uses a custom shard key and generates 480 of shard 4's 600 QPS, while other tenants contribute 120 QPS. Node CPU is high but raw vector counts are balanced.
• A controlled test partitions this tenant into four approved shard keys with stable document-to-key routing and a tested query fan-out/merge plan. Assume its 480 QPS is evenly split across four eligible partitions (120 QPS each), leaving the original shard's remaining 120 QPS. This requires true data repartitioning; adding read replicas alone increases read capacity but does not change shard-key distribution.
• After migration, illustrative maximum shard-targeted load falls from 600 to 240 QPS in the new balanced placement; slow-shard p99 falls from 900 ms to 180 ms and overall search p99 from 1,050 ms to 260 ms. Validate the actual partition map and load—these improvements are an example, not a guaranteed scaling result.
• Compute improvement: overall p99 reduction = (1,050 − 260)/1,050 × 100 ≈ 75.2%. Compare Recall@10, ACL correctness, ingestion freshness and storage cost before approving the cutover.

5. Engineering mitigation options

• Bad replica/host: shift search to a verified healthy replica and repair or move the faulty replica. In Qdrant, inspect collection cluster info, peer placement and shard transfer status; transfer mechanisms can preserve or rebuild indexes differently.
• Hot tenant/key: partition a large tenant's logical shard, choose a better custom-sharding strategy, or employ supported online resharding; document how reads combine results correctly across the newly assigned partitions. Retain versioned tenant-to-shard routing and audit ACLs.
• Oversized shard: rebalance/reshard with a snapshot and live change-log catch-up, checking that all vectors and deletions are accounted for. A move redistributes placement; a split redistributes logical work. They are not interchangeable.
• Filter-heavy workload: index the relevant metadata fields, tune ANN/filter planning and measure exact-vs-approximate recall with the same filters; never drop access predicates.
• Burst-induced queues: introduce per-tenant quotas, bounded concurrency, fair scheduling, request coalescing where safe and workload isolation. Monitor whether throttling raises user-facing 429s or freshness delays.
• Optimizer/I/O contention: rate-limit maintenance/ingestion or provision independent fast storage, with a measured freshness-versus-p99 budget.
• Roll out via canary traffic, watch per-shard QPS/queue/IO/latency and query correctness, and keep a secure rollback plan. On a physical move, verify target replication and read visibility before retiring the source. On logical repartition, retain consistent tenant routing plus old-key reconciliation until fully migrated.

6. Trade-offs and mistakes

• More replicas can increase read throughput and availability, but duplicate storage/RAM and may not fix hot-key routing, write hotspots or query planning.
• More shards distributes load when the shard-selection strategy permits it, but increases coordination, open segments, network fan-out and operational complexity.
• Moving a shard can cure a bad node but merely relocates a logically hot shard. Moving heavy shards also consumes disk/network resources and can impact p99 during migration.
• Per-tenant isolation improves fairness and predictable SLOs, but complicates onboarding, costs and cross-tenant aggregations. Never mix tenant data or relax authorization.
• Caching can offload repetitive searches but may serve stale/deleted content unless keyed and invalidated by tenant, ACL and document version.
• Lower ANN search effort improves latency at potential Recall@k cost; compare quality before changing it.
• Coarse shard averages hide bad replicas and hot query classes. Diagnose per-shard *and* per-replica histograms, with enough requests for stable percentiles.

7. Prevent recurrence

• Dashboard shard and replica QPS, in-flight/queued searches, p50/p95/p99, CPU and disk contention, index bytes, point count, cache hit rate, optimizer backlog, replica lag and dominant tenant/key shares.
• Alert on load skew (maximum shard QPS / mean shard QPS), queue-age SLO breaches, per-shard tail latency, missing shard selectors and rapidly growing shard sizes—not only average node CPU.
• Test tenant-heavy and time-skewed workloads with realistic top_k, filters, simultaneous writes and cold caches; verify routed-query results against an exact or trusted baseline.
• Establish sharding strategy and capacity planning rules that account for data distribution and request distribution; forecast tenant and time-window hotspots before production growth.
• Maintain rehearsed runbooks for replica isolation, safe shard move, logical split, resharding, write replay, canary/rollback and security validation.

Interview takeaway: Hot-shard diagnosis is about who sends work to which shard, where that shard is placed, and what consumes its time. Compare per-shard/replica traces, queue pressure and load distribution first. Rebalance a faulty host, but *split or reroute a truly hot logical partition* rather than simply moving its overload to another node.

References

• Qdrant, Distributed deployment: https://qdrant.tech/documentation/scaling/distributed_deployment/
• Qdrant, Monitoring: https://qdrant.tech/documentation/ops-monitoring/monitoring/
• Qdrant, Multitenancy: https://qdrant.tech/documentation/guides/optimize/
• Weaviate, Multi-tenancy operations: https://docs.weaviate.io/weaviate/manage-collections/multi-tenancy