Your RAG API performs well for 10 concurrent users but becomes unstable at 1,000. How would you load-test and identify the first bottleneck?

Production incident (illustrative): A customer-support RAG endpoint works for 10 employees: p95 end-to-end latency is 1.8 seconds, few errors, and answers cite the right product manual. After a launch exposes the same endpoint to 1,000 connected users, p95 exceeds 9 seconds, 429/503 responses appear, retries multiply traffic, and a few answers become incomplete after timeouts. Engineers initially blame the LLM because generation is the most visible stage, but distributed traces reveal an exhausted vector-database connection pool and growing retrieval wait queue. The problem is neither fixed by increasing the API server pod count nor accurately characterized by saying “1,000 users equals 1,000 RPS.”

Core principle: Model a RAG request as a queueing network—gateway → authorization → query preprocessing/embedding → vector/lexical search → rerank/context assembly → generator/provider → streaming response, plus caches, storage, pool limits and asynchronous dependencies. Recreate the *arrival process, request mix, prompt/output token distributions, think times and concurrency*; observe each queue and stage before scaling. Distinguish users, in-flight requests, arrivals per second, completed requests per second, tokens per second, and sustainable capacity. A service with 1,000 users may see very different workloads depending on how frequently they ask questions and how many tokens each answer consumes.

1. Define service objectives, risk limits and realistic traffic

• Capture the current accepted baseline at 10 concurrent users and select explicit test SLOs: for illustration, p95 full-response latency ≤3 seconds, p99 ≤6 seconds, error fraction below 1%, zero unauthorized results and acceptable grounded-answer accuracy. These are example targets, not universal standards for RAG workloads.
• Separately define time to first token (TTFT), time to complete streamed response, time to first retrieved context, and a non-streaming end-to-end duration. A fast TTFT with a 12-second tail is not a 3-second full answer.
• Recreate realistic user behavior: login/auth path, tenant and policy filters, bursty arrivals, session histories, question complexity, cache-hit versus cold queries, top-k, number of candidate sources, prompt length, answer token count and streaming/cancellation behavior.
• Distinguish 1,000 *connected users* from 1,000 simultaneously outstanding requests, and from 1,000 requests/second. Estimate request rate using measured think times and completion times, then test both realistic sessions and defined arrival-rate limits.
• Build a sanitized representative workload: short factual lookups, multi-part questions, long documents, cold/warm caches, complex reranking, large outputs and negative queries. Include authorized tenant diversity and avoid actual customer data in shared staging.
• Establish cost and safety budgets before stress tests. Provider calls consume paid tokens, external QPS quotas, shared inference capacity and potentially trigger vendor protections; never flood a live shared tenant without a controlled window and stop conditions.
• Record revisions of API image, prompt, embedding/rerank/generator models, vector index, source snapshot, connection pool settings, replicas and rate limits. Only compare runs with controlled changes.

2. Choose the correct load generator and test shape

• Closed-loop user model: a virtual user sends a request, waits for completion, thinks for a specified interval, then sends another. It models many human sessions but its arrival rate falls as the system slows, potentially concealing overload (coordinated omission). Grafana k6 distinguishes these *closed* models from *open* arrival-rate models.
• Open-loop arrival model: schedule requests independently of the preceding response duration, e.g., k6 constant-arrival-rate and ramping-arrival-rate. Use it to find capacity under a stated target RPS. Preallocate sufficient test VUs and monitor dropped_iterations so the generator is not mistaken for the service bottleneck.
• Concurrency stress: separately hold 10, 50, 100, 250, 500 and 1,000 in-flight requests using bounded requests and controlled durations. This tests socket, memory, admission and per-request resource limits, not necessarily realistic arrival rate.
• Step-load: increase actual offered load in plateaus (illustratively 5→15→25→35→45 requests/sec), holding long enough to reach an approximate steady state or observe persistent queue growth. Restore baseline between comparative experiments when appropriate.
• Spike/soak: test short bursts, gradual long-running load, and recovery after load stops. Soak testing catches memory leaks, stale connections, growing caches and accumulating retry/dead-letter queues that a 2-minute burst misses.
• Cold versus warm: run an index/query-cache cold test and a warm repeat test; preserve both in reports. A 100% cache-hit benchmark should not be extrapolated to production's unique user questions.
• Loader checks: distribute the load generator across hosts or workers when needed (Locust supports master/worker distributed execution), and report its CPU, memory, open sockets, network egress, VU saturation and scheduled-versus-actually-sent requests.
• Ground-truth safety: use a fixed authorized golden question subset and compare citation validity, answer completeness and decision accuracy under load; HTTP 200 without a usable answer is not success.

3. Instrument one end-to-end trace and each critical queue

• Propagate one request trace ID across gateway, query rewrite, embedding API, vector and lexical retrieval, source filters, reranking, packing, generation, token streaming and response formatting. Keep tenant_id pseudonymized and avoid logging document bodies by default.
• For each stage record arrival, queue-enter, work-start, work-end and completion timestamps if possible; separate wait time from service time. A 4-second vector “duration” may contain 3.5 seconds of pool waiting and 0.5 seconds of actual ANN search.
• Capture stage p50, p95 and p99; throughput; in-flight tasks; concurrency permits; worker/thread event-loop utilization; saturation; queue length and age; timeout/cancellation; retries; and error codes. Do not add stage p95 values to claim an exact end-to-end p95.
• At the gateway watch active connections, CPU, event-loop lag, request backlog, memory, load balancer queue, connection resets and tail latency. Detect whether a client timeout is occurring while the backend continues working.
• At embedding/provider layers watch in-flight calls, total/input/output tokens, quota counters, 429, retry-after, retries per user request, latency and billable tokens. Batch/caching strategy changes cost and throughput; measure both.
• At vector search watch connection-pool occupancy and wait, ANN execution latency, shard/replica CPU, I/O, disk/page cache, index memory, filter selectivity, result completeness, network hops and shard queue depth.
• At reranking and context assembly watch candidate counts, batch sizes, GPU/CPU saturation, tokenization time, memory and context window packing. Heavy reranking can overtake vector search even when raw search stays fast.
• At LLM generation watch provider request latency, TTFT, output tokens/sec, input tokens, output tokens, concurrency slots, quota errors, and whether a long-streaming request continues to occupy an admission permit.
• Use OpenTelemetry-style tracing and versioned semantic conventions where useful; correlate per-stage timing with token usage and resource saturation. Some GenAI metric names evolve; validate against deployed instrumentation library/version rather than assuming a static schema.
• Record offered vs accepted vs completed RPS, errors by component, dropped tests, and outstanding backlog. An increase in 200 responses per minute does not prove healthy throughput if offered demand and error queues are growing.

4. Root-cause debugging: find the first bottleneck

• A. Establish first unstable plateau: graph offered RPS, completed RPS, p95/p99, 429/503 and queue depth at each step. The knee occurs where throughput stops rising roughly with offered load while queues and latency start growing.
• B. Inspect earliest growing queue: compare gateway, embedding semaphore, vector connection pool, reranker slots and model-provider admission. The *first* stage with sustained backlog and high utilization is often the immediate constraint; a later downstream queue may be a symptom.
• C. Examine work-start timestamps: a vector API span that looks slow may mostly wait for a free DB connection. Separate queue wait from vector execution before tuning HNSW, k or index parameters.
• D. Check upstream amplification: retries after 429/503, speculative parallel retrieval, duplicate queries, user cancellation not propagated, and client reconnect can multiply requested work. Measure attempt rate per original request and bound amplification.
• E. Use component isolation: replay queries with a stub embedding result, fixed vector candidates, bypassed reranker, or deterministic mock generator only in an isolated test. Observe which intervention moves the throughput knee. A mock LLM might hide its token-rate bottleneck; interpret each test narrowly.
• F. Inspect server-side pressure: API autoscaling based only on CPU can miss exhausted DB pools or external quota limits. Scaling from 4 to 20 API pods can multiply connection demand and worsen vector-database contention.
• G. Verify external saturation: if embedding/LLM providers show high queue/429 rates, adding local workers will not raise sustainable provider capacity. Measure requests/minute and tokens/minute against the actual assigned limits.
• H. Separate slow dependencies: an extra retrieval source may introduce network fan-out or head-of-line blocking; Q278 will cover multi-source critical-path optimization specifically. Q279 will cover slow external API deadlines; Q277 will cover spike-driven embedding 429 and retry budgets.
• I. Validate quality under load: partial retrieval due to deadlines or throttles can cause wrong citations, stale cache fallback or omitted exception clauses. Correlate answer-quality failures with load stage rather than treating performance and correctness as independent.
• J. Identify instability type: queue saturation, throughput collapse, timeout, crash/OOM, container restart, provider quota, unhealthy replica or client-generator overload each needs different remediation. Do not infer one bottleneck from aggregate CPU.

5. Beginner-friendly capacity and queueing numerical example

• User math: imagine 1,000 active users who, on average, spend 20 seconds thinking and each request takes 4 seconds end-to-end. A simple closed-interaction estimate yields 1000/(20+4) ≈ 41.7 requests/sec; 1,000 users is not 1,000 RPS. Real arrivals are bursty, so test more than this mean.
• Stage capacity (illustrative): embedding 100 RPS; vector search has 16 effective concurrent connection slots, with mean active service time 0.5 sec, giving an idealized upper bound 16/0.5 = 32 searches/sec; reranking 60 RPS; generation provider supports 50 RPS at this specific input/output token mix. Thus vector connection admission is the lowest approximate ceiling: 32 RPS.
• Step observations (hypothetical): 5 RPS → p95 1.8s, 0% errors; 15 RPS → p95 2.0s, 0%; 25 RPS → p95 2.6s, 0%; 35 RPS → p95 8.2s, 2% errors; 45 RPS → p95 14.5s, 8% errors. At 35 RPS, completion throughput hovers around 31–32 RPS while connection wait and outstanding queue keep growing.
• Queue buildup: if 35 requests/sec arrive and vector admission can sustain 32/sec, the idealized undrained backlog increases by 35-32=3 requests/sec or 180 waiting requests after 60 sec before timeouts, shedding and service-time variation. This approximation explains why latency can deteriorate before CPU looks fully used.
• Observed trace: at 35 RPS, vector span p95 is 4.4 sec, including 3.8 sec pool wait and only 0.6 sec search; embedding remains p95 90 ms, reranking p95 140 ms, generator p95 1.1 sec. High *vector waiting*, not raw vector compute, is the immediate first saturation.
• Fix test: raise safe vector capacity by optimizing pool/admission, query execution, shard capacity or adding appropriately provisioned replicas; benchmark a revised sustainable vector service ceiling of 48 RPS under the same mix and with measured accuracy. Then re-run all plateaus: if stable p95 at 35 RPS becomes 2.7s with <1% errors, the bottleneck has moved or been reduced. Do not claim that merely opening more connections always raises capacity; extra connections can overwhelm the database.
• New ceiling: generation at 50 RPS may become the next limiting service, and total demand around 45 RPS could still approach its token/minute or concurrency ceiling when prompts get longer. Capacity planning must consider *token throughput* as well as request throughput.
• Little's law check: at a stable throughput of 25 RPS and mean time in system 2 seconds, average in-flight work is roughly L = λW = 50. This average relation assumes a stable system; it is not valid to extrapolate indefinitely from a queue that is growing without bound.
• Quality check: suppose grounded correct answers are 97/100 at 10 users but 91/100 at peak due to failed source calls and incomplete fallback contexts. Performance release criteria must include grounded-answer quality, not just p95 or success HTTP codes.
• All figures are illustrative rather than measured MachinoAI telemetry. The service capacity calculation assumes stable average service time; measured queueing behavior, tail latencies, shard contention and provider quotas can lower real capacity.

6. Engineering solutions in recommended order

• First improve measurement and isolation: add queue-time spans and dashboards; validate load-generator capacity, realistic token mix and thresholded tests before changing infrastructure.
• Admission and concurrency control: place bounded per-stage semaphores/queues in front of scarce vector connections, reranker GPUs and LLM calls. Reject or defer excess work predictably instead of permitting unbounded memory, queue age and timeouts.
• Capacity scaling: right-size vector pool relative to shard/server limits, scale retrieval replicas where justified, optimize slow filters and index placement, tune rerank batch/concurrency and allocate worker CPU/memory after identifying the actual constraint.
• Provider-aware quotas: cap upstream embedding and generation request/tokens per minute, distinguish 429 from 5xx, use bounded retry-after-aware backoff and per-original-request budgets. Q277 explores embedding-spike backpressure and retries in detail.
• Cancellation propagation: abort or stop expensive downstream work when user request has expired/cancelled, where the provider and client stack support it. Avoid doing retrieval or generation that cannot be returned.
• Cache responsibly: cache authorized embedding results for repeated normalized queries, shared public facts where permitted, and version-scoped document metadata. Include tenant, ACL, source version and effective date in appropriate cache keys, and never use cross-tenant caches that leak results.
• Fairness: per-tenant quotas and weighted scheduling prevent one large tenant or expensive long-context query from starving other users. Separate interactive and batch queues.
• Cost-aware token controls: limit irrelevant context and enforce measured output budgets, but preserve critical evidence; token trimming that drops policy exceptions may reduce latency while making answers incorrect.
• Graceful overload: choose bounded queuing and clear 429/503 responses where necessary; do not return fake “success” responses with unsupported or incomplete claims.
• Scale in stages: after each controlled improvement, rerun the same load mix, then identify the *next* bottleneck. Scaling one stage moves the knee rather than eliminating physical limits.

7. Trade-offs, test pitfalls and failure modes

• Closed-loop only: underestimates high offered traffic when latency rises because users wait; add open-loop tests to avoid coordinated omission.
• Open-loop only: may not model users' natural think times and cancellations; compare both modes with explicit definitions.
• One thousand VUs: does not imply 1,000 simultaneously active API requests or 1,000 RPS. Publish actual scheduled/started/completed rates.
• Short stress-only tests: miss queue buildup, connection leaks and sustained provider quota depletion; include steady-state and soak durations.
• Too few load-generator VUs: k6 dropped_iterations can result from inadequate VU preallocation; report generator utilization and errors separately from server overload.
• Aggressive retries: can turn a manageable transient fault into positive feedback and worsen the overloaded stage. Limit attempts and consider load shedding.
• More application pods: increases upstream concurrency and connection pressure; provision constrained downstream dependencies in tandem.
• Larger pools: can reduce waiting only if the backend has spare capacity; otherwise they can raise contention and tail latency.
• Caching: improves hot-query throughput but can hide cold-query bottlenecks and leak restricted data if cache authorization is wrong.
• Batching: improves throughput on embeddings/reranking but can worsen latency for a single urgent request; measure both.
• Provider tokens/minute: a 50-RPS quota does not guarantee 50-RPS effective throughput for long prompts; track tokens/minute and streaming concurrency.
• Latency percentiles: summing stage p95 values is not mathematically equal to end-to-end p95. Use per-request critical-path traces.
• SLO versus correctness: response-time wins achieved by incomplete source fetching, truncated citations or silent fallback may fail the user. Keep retrieval coverage/groundedness gates.
• Load testing in production: can exhaust third-party quotas or disrupt unrelated users. Use isolation, budget caps, rate safety rails, phased windows and explicit shutdown triggers.

8. Prevention and operational release gates

• Maintain a versioned load-test workload covering short and long prompts, retrieval-only and fully generated responses, cold/warm caches, streaming, tenant filters, concurrent sessions and high-token queries.
• Run ramp/step/spike/soak tests at every material RAG pipeline release; require p50/p95/p99, time-to-first-token, full response latency, accepted/completed RPS, offered load, error/429/503 rates, queue age, dropped iterations and cost.
• Define a maximum sustainable throughput as the highest tested arrival rate where queues remain bounded, output quality meets minimums, the error/latency SLO passes and load can be held without accumulating debt. One fast burst does not establish a stable capacity limit.
• Add synthetic bottleneck scenarios: embedding provider 429, vector-pool exhaustion, slow rerank batch, low model token quota, dropped shard replica, downstream cancellation and load generator CPU saturation. The stage-local telemetry should identify each.
• Track pool occupancy/waits, concurrency permits, memory, event-loop lag, provider request/token quotas, per-tenant fairness and missing authoritative citations at overload.
• Periodically recalculate capacity when embedding/generation models, token budgets, document sizes, index configuration, user mix or provider limits change; a new model snapshot can move the bottleneck.
• Use canary and feature flags for concurrency, cache, replica and batching changes; retain a safety rollback path that respects tenant ACLs and current source validity.
• Set an operational alert for rising queue wait while service time remains flat—an early sign of a saturated admission pool—and a separate alert for service-time inflation indicating CPU, index or provider degradation.
• Keep a short incident runbook: identify first affected request cohort → compare offered/completed load → find earliest growing queue → verify actual backend service capacity → isolate/replay → fix one constraint → rerun exact workload → confirm quality, cost and SLOs.

Interview takeaway: A RAG API that works for 10 concurrent users but fails at 1,000 needs a controlled open/closed-loop load test with per-stage queues and traces, not immediate autoscaling. Convert real user behavior to offered RPS, ramp load to the first throughput knee, locate the earliest queue whose wait grows, then test one bottleneck fix at a time. Verify that improvements preserve grounded answers and do not merely hide failures behind caches or dropped requests.

Research references

• Grafana k6, *Open and Closed Models*: https://grafana.com/docs/k6/latest/using-k6/scenarios/concepts/open-vs-closed/
• Grafana k6, *Constant Arrival Rate*: https://grafana.com/docs/k6/latest/using-k6/scenarios/executors/constant-arrival-rate/
• Grafana k6, *Dropped Iterations*: https://grafana.com/docs/k6/latest/using-k6/scenarios/concepts/dropped-iterations/
• Grafana k6, *API Load Testing and Thresholds*: https://grafana.com/docs/k6/latest/testing-guides/api-load-testing/
• Locust, *Distributed Load Generation*: https://docs.locust.io/en/stable/running-distributed.html
• OpenTelemetry, *GenAI Semantic Conventions and Metrics*: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/model/gen-ai/metrics.yaml