Vector search p99 latency suddenly increases while p50 remains stable. How would you identify the responsible infrastructure bottleneck?

Production incident (illustrative): A customer-support RAG service queries 8 million product-document vectors. After the nightly ingestion batch grows, vector-search p50 stays at 45 ms, but p99 jumps from 180 ms to 1,250 ms. Most users still see fast responses; a small minority experiences timeouts. Raising the LLM timeout hides the symptom and doesn't identify the vector-search bottleneck.

Key distinction: p50 is the median request, whereas p99 is the latency at or below which approximately 99% of requests fall in a chosen observation window. Rising p99 with stable p50 suggests a slow subset, *not proof of a specific cause*. Investigate query shape, tenant, shard, node, queueing and storage behavior before changing index settings.

1. Contain and establish the boundary

• Check the customer-facing latency SLO, timeout rate, affected tenants and rate of bad requests; if necessary throttle bulk ingestion, bound request concurrency or shift traffic to a healthy replica through a controlled rollout.
• Confirm whether the alert measures vector DB server time, client-observed search time, or end-to-end RAG time. Record these separately; reranking, query embedding and LLM generation can affect end-to-end p99 even if vector search is healthy.
• Compare the same metric, route, traffic mix, time window, region and percentile aggregation with the previous baseline. Do not average p99 values computed across shards or time slices; use raw distributions/histogram aggregation where valid.
• Inspect changes to ingestion jobs, index settings, top-k, search ef, payload filters, quantization, shard placement, autoscaling, storage tier, deployment versions and traffic bursts.

2. Trace the actual slow requests

• Add a trace/correlation ID across API ingress → pool/queue wait → vector-client network → coordinator → scatter/gather → shard search → filter evaluation → HNSW traversal/rescore → payload fetch → response serialization. Log request size, top-k, ef, filter selectivity, collection, tenant, shard and target replica without leaking sensitive query text.
• Compare slow top 1% against normal requests on the same route, including a request-sampled profile, not just global CPU averages.
• Look at DB-native profiling where available. Weaviate, for example, exposes per-shard query profiling fields such as vector_search_took, filter allow-list construction, and HNSW traversal/rescore timings. Use the provider's actual supported telemetry rather than assuming these field names apply across vendors.
• Measure queue wait and concurrency separately from execution. A saturated client connection pool can inflate client-observed p99 even when server execution stays low.

3. Classify the responsible bottleneck

• Storage and cache misses: slow traces correlate with read I/O spikes, storage await, page faults, cache evictions or cold on-disk HNSW/payload access. Qdrant warns that colder index/payload tiers can increase disk I/O and latency. Compare repeated warm queries with first-time cold queries.
• CPU or background index contention: slow windows align with bulk upserts, segment compaction/optimization, HNSW rebuilds, CPU run queues or CPU throttling. Search and optimizer workers may compete for CPU, RAM and I/O. Stop or rate-limit the batch as a reversible experiment.
• Query planner/filter issue: slow requests use selective or unindexed metadata filters, unusually large top_k/offset, costly hybrid search or changed ANN search effort. Inspect whether the filter uses an index, builds an expensive candidate list, or triggers a full scan. Do not simply lower search effort before measuring recall.
• Network/coordination tail: one straggling replica, transient retries, cross-zone traffic, replica read inconsistency, unavailable shards, or coordinator fan-out magnifies p99. Inspect per-shard/peer traces and retry counts; a minority of slow shards is sufficient to hurt the slowest queries.
• Load imbalance and queueing: bursts concentrate on a tenant or shard, connection-pool wait rises, or a load balancer sends disproportionate traffic to one replica. Check per-node request rates, queue depths and utilization, not cluster averages.

4. Practical numerical root-cause walkthrough

• Over 10,000 comparable searches in a five-minute window, assume 9,800 finish near 45–80 ms while 200 filtered searches suffer significant delays. The slow 2% is sufficient to move p99 dramatically while p50 remains about 45 ms.
• Before incident: p50 = 45 ms, p99 = 180 ms. During incident: p50 = 46 ms, p99 = 1,250 ms. Tail inflation = 1,250 / 180 ≈ 6.94×, despite only 1 ms median increase.
• In a representative slow-request trace: API/queue 20 ms + coordinator/network 25 ms + filter planning 90 ms + vector search and payload I/O 1,070 ms + serialization 45 ms = 1,250 ms. This is a *single illustrative trace*, not a decomposition of the aggregated p99.
• Node metrics show a spike in disk read latency and vector I/O on one replica during heavy index optimization. With the ingestion batch paused for a short controlled test, representative slow-request vector I/O falls from 1,070 ms to 110 ms, while network and filter time remain similar. This correlation plus the isolated intervention implicates storage contention from ingestion/optimization, not the LLM or semantic-search score.
• Engineering fix: throttle indexing and cap concurrent writes, move hot index working sets to a suitable faster memory/storage tier if justified, and schedule expensive optimization work appropriately. Validate with the same replay traffic; suppose p99 falls to 210 ms and p50 remains 46 ms. Tail improves (1,250−210)/1,250 × 100 = 83.2%, though it has not returned to the 180 ms baseline.

5. Make the change safely

• Choose the smallest reversible mitigation backed by traces: cap ingestion concurrency, rebalance overloaded replicas, tune storage tiers or optimize a missing payload index. Treat index rebuilds or migrations as controlled operations.
• Re-run a representative benchmark with actual metadata filters, tenant mix, burst rate, cold starts, and write workload. Compare p50/p95/p99, throughput, failure rate, queue wait, CPU throttling, disk IOPS/latency and per-shard results.
• For changes to HNSW ef, quantization, top-k or filter strategy, simultaneously check Recall@k and correctness; a faster result can be worse retrieval. Watch query-time access control and tenant filtering.
• Roll out through canaries, with rollback thresholds; preserve a before/after trace set and configuration diff to confirm causality.

6. Trade-offs and edge cases

• More RAM / pinned index: fewer cache misses, higher cost; do not scale first if queueing or a bad filter dominates.
• Faster SSD / more IOPS: helps I/O-bound tails, but may do little for CPU or lock contention.
• Limit write concurrency: protects search p99 but increases ingestion-to-search freshness lag; set a freshness budget and reserve ingestion capacity.
• Replica routing: can isolate a straggler, but risks inconsistent freshness or ACL visibility if replicas lag. Validate consistency and capacity.
• Lower ef, top-k or aggressive quantization: can improve speed at the expense of retrieval recall and downstream answer quality.
• Timeouts and circuit breakers: prevent unlimited resource use; they may reduce successful-answer rate. Measure errors and abstentions alongside latency.
• Percentile caveats: if a sample contains fewer than ~100 observations, p99 is unstable; histogram bucket resolution, sampling bias, coordinated omission and mixed query classes can also distort the metric.

7. Prevent recurrence

• Instrument distributed spans, request histograms and per-peer/shard CPU, memory pressure, disk latency/IOPS, optimizations, queue waits, cache effectiveness, retries and request-class dimensions. Qdrant exposes node and collection metrics via /metrics; scrape every peer, since node endpoints are not cluster-wide.
• Define alert thresholds for p99 and p99/p50 ratio together with request volume, error rate, backlog growth and retrieval quality. Keep query-class, tenant and region breakdowns where cardinality and privacy budgets allow.
• Run continuous workload benchmarks under simultaneous reads/writes, cold and warm caches, burst traffic and expected metadata filters. Load-test new indices/configuration before promotion.
• Add guards for ingestion concurrency, query complexity and expensive filters; maintain on-call playbooks for diagnosing CPU, disk, network, shard and query-planner tails.
• After every incident, preserve an evidence-backed timeline and verify the fix against p99, freshness, cost and recall, not median latency alone.

Interview takeaway: When p99 explodes but p50 stays flat, isolate the slow request cohort and locate time in queue → coordinator → shard/filter → ANN → disk. Use traces, provider profiling and a reversible intervention to prove the bottleneck before increasing hardware or sacrificing recall.

References

• Qdrant, Monitoring & Telemetry: https://qdrant.tech/documentation/ops-monitoring/monitoring/
• Qdrant, Database Optimization troubleshooting: https://qdrant.tech/documentation/faq/database-optimization/
• Qdrant, Indexing and storage choices: https://qdrant.tech/documentation/manage-data/indexing/
• Weaviate, Query profiling: https://docs.weaviate.io/weaviate/search/query-profile