How do you perform load, stress, spike, and soak testing to identify production LLM inference capacity limits?

Load, stress, spike, and soak tests exercise different failure modes of an LLM-serving system. Load testing checks ordinary and expected peak demand; stress testing pushes beyond that operating range to find the saturation knee and safe failure behavior; spike testing checks sudden arrival bursts and recovery; soak testing checks long-running stability, memory growth, and performance drift. Together they identify sustainable SLO-compliant capacity, not merely the highest requests per second reported by a synthetic benchmark.

1. The four tests and what each proves

• Load test: Hold normal and forecast-peak traffic long enough to measure stable p95/p99 TTFT, TPOT/ITL, end-to-end latency, error rate, GPU/KV memory use, and SLO-compliant goodput. This tests whether planned traffic can be served reliably.
• Stress test: Increase *offered* request rate (and token work) beyond expected peaks in steps until latency targets break, errors or rejections rise, or throughput stops improving. Locate the capacity knee and confirm graceful load shedding rather than OOM or crashes.
• Spike test: Jump rapidly from baseline to high demand and back. Check admission control, queue limits, autoscaler lag, retries, model cold starts, p99 queue delay, and recovery without request storms.
• Soak test: Sustain a realistic traffic mix for hours or a workload-appropriate long period. Watch for memory/KV allocation leaks, growing queues, stale prefix-cache entries, fragmentation, slow increases in TTFT/ITL, unhealthy workers, and cost drift.
• Bonus — breakpoint test: A controlled ramp to the failure limit is a subtype of stress testing. Do not run destructive breakpoint experiments against shared production capacity without isolation and guardrails.

2. Why requests/second is not enough for LLMs

• A 200-token chat request generating 50 tokens and a 20,000-token document prompt generating 2,000 tokens impose very different prefill, decode, and KV-cache costs.
• Measure uncached input tokens/s, output tokens/s, queued prompt tokens, active sequences, KV occupancy, preemptions, prefix reuse, and the share of work by tenant/workload.
• Preserve prompt/output length correlation, streaming, rate burstiness, cache-hit ratio, deadlines and client disconnect behavior.
• An independent/open-loop arrival schedule exposes backlog growth; fixed-concurrency/closed-loop clients can accidentally slow their offered load when server latency rises.
• Test both steady-state demand and transient workload changes; an autoscaler that handles sustained load may still miss a 30-second spike.

3. Beginner-friendly numerical capacity example

Assume a 2-GPU LLM serving deployment with these *illustrative measured results* under the same realistic request mix and a joint TTFT ≤ 1 second and TPOT ≤ 50 ms target:

• Load at 8 offered requests/s: 8 successful completions/s with 95% meeting both SLOs → 7.6 SLO-compliant requests/s; p95 TTFT is 0.7 s.
• Higher load at 12 requests/s: 12 successful completions/s with 90% SLO compliance → 10.8 goodput/s; p95 TTFT is 0.95 s.
• Stress at 16 requests/s: 15 successful completions/s with only 55% compliance → 8.25 goodput/s; p95 TTFT rises to 2.3 s.
• Stress at 20 requests/s: 16 completions/s with 30% compliance → 4.8 goodput/s; queues grow or overload protection rejects requests.

Interpretation: Peak observed raw throughput may reach 16 completions/s, but the best tested SLO-compliant goodput is 10.8 requests/s at the 12-RPS load point. A conservative provisioned operating point should be below the saturation knee with headroom for burst variation and worker loss. These figures are illustrative, not a specific product's performance claims.

Queue-growth check: If sustained offered demand is 20 requests/s but the system completes only 16 requests/s, an unbounded queue can grow at about 4 requests/s. After 60 seconds it would accumulate roughly 240 requests if no requests time out, cancel or get rejected. Real systems should enforce bounded admission and record all shedding.

4. Design a testing sequence

1. Freeze the environment: model weights, tokenizer, precision, GPU SKU/count, inference engine, routing, model/context limits, and reproducible traffic dataset.
2. Validate the generator: smoke-test the endpoint; verify offered versus actual dispatched RPS, GPU-side metrics, stream parsing and output-token counts. NVIDIA warns that high-concurrency benchmark clients can bottleneck before the inference server.
3. Baseline and load test: use typical and forecast-peak distributions for sufficiently long stable windows, after explicitly recording warmup and cache state.
4. Stress/ramp test: increase offered RPS in manageable steps while preserving request length mix. Measure the first sustained SLO breaches, queue growth, admission rejections, and OOM/preemptions.
5. Spike/recovery test: rapidly increase rate (for example, 8 → 24 → 8 RPS) and measure p99 latency, retry amplification, maximum queue, autoscaler reaction, and time to stable recovery.
6. Soak test: sustain a normal or moderately high but safe load for an appropriate extended period; look for statistically meaningful memory and latency slopes, cache-fragmentation effects, and long-term availability changes.
7. Failure-injection extension: repeat selected tests with a replica restart, degraded network, or full prefill/decode worker pool to validate resilience without causing customer impact.
8. Compare runs using matched traffic seeds, warm/cold cache assumptions and confidence intervals; preserve raw traces and configuration.

5. Example capacity sweep with GuideLLM

GuideLLM supports sweep profiles that begin with a synchronous baseline, discover a throughput ceiling, then test intermediate rates. Its documented example can be adapted to a compatible OpenAI-style server:

BASH
guidellm run \
  --backend kind=openai_http,target=http://localhost:8000 \
  --data kind=synthetic_text,prompt_tokens=512,output_tokens=256 \
  --profile kind=sweep,sweep_size=10 \
  --constraint kind=max_duration,seconds=120 \
  --seed kind=static,value=42 \
  --output kind=json,path=sweep_results.json

This is a controlled synthetic sweep, not a complete production traffic replay. Use representative trace-based or heterogeneous traffic for production conclusions, and check installed tool/version options.

NVIDIA AIPerf offers configurable request rates, concurrency, Poisson/gamma arrival patterns, synthetic token distributions and trace replay. For general API/application traffic, Grafana k6 documents smoke, normal load, stress, spike and soak profiles. Select the tool and observation window appropriate to the LLM endpoint and full user journey.

6. Establish pass/fail criteria before testing

• Normal load: p95/p99 TTFT/TPOT and end-to-end latency within required SLOs; acceptable errors and rejections; stable backlog and GPU/KV pressure.
• Stress: bounded queues and explicit, controlled overload responses instead of OOM, deadlock, cascading retries or runaway tail latency. Determine *highest stable SLO-compliant operating rate*, not just peak raw tokens/s.
• Spike: quantify maximum queue delay, response-rejection rate, autoscaling lag and recovery time; protect interactive users from batch traffic.
• Soak: no unexplained sustained rise in allocated GPU/host memory, KV-cache occupancy at fixed workload, latency, error rates, or worker restarts.
• Per-tenant fairness: ensure aggregate goodput improvements do not hide starvation of one request class or customer tier.

A service may be “up” but still fail the performance test if nearly all requests violate streaming latency SLOs.

7. Production incident: short tests pass, overnight traffic fails

Incident: The team ran a five-minute fixed-concurrency benchmark and concluded the LLM endpoint could handle 20 RPS. During a long daytime workload, TTFT worsens hour by hour; an evening surge causes queues to explode and users see many timeouts.

Root-cause debugging:

1. The fixed-concurrency generator throttled itself as the server slowed, so its claimed 20-RPS capacity was not a sustained offered load.
2. Reused short prompts created unusually favorable prefix-cache hits versus production.
3. The test ended before a sustained memory/queue growth pattern emerged.
4. The gateway retried 503 responses aggressively, amplifying the spike.

Engineering solution: run open-loop load and stress sweeps, reproduce the real short/long prompt mix and prefix-reuse rate, introduce a bounded token-aware queue and jittered retries, and run a soak test with GPU memory/KV/queue trend alerts. Separate a confirmed leak from legitimate growth caused by different context lengths.

Prevention: make distinct load, stress, spike and soak suites release gates, include peak-traffic and replica-failure scenarios, and record a repeatable capacity envelope for each model/hardware configuration.

8. Critical observations and cost trade-offs

• Saturation point varies with workload: do not state a single “max RPS” without token distributions, model version, GPU topology and latency SLOs.
• Client bottlenecks: check generators' CPU/network limits, dropped scheduled arrivals and client queueing before attributing tail latency to GPUs.
• Scale-up cost: more replicas improve headroom but may incur GPU idle costs and inter-replica KV-transfer/network overhead.
• Aggressive batching: can improve output tokens/s while reducing SLO goodput by increasing TTFT/ITL.
• Early rejection: protects active requests but is not free—record availability, 429/503 and final user success separately to prevent gaming goodput.
• Percentile reliability: collect enough samples within each workload bucket; rare p99 delays are hard to estimate from a tiny run.

9. Interview-ready answer

“I would first validate a realistic open-loop load generator and baseline the deployment. Then I'd run a normal-load test for SLO stability, a stepped stress test for the capacity knee, a sudden spike test for queues and recovery, and a long soak test for memory or latency drift. I'd track p95/p99 TTFT and TPOT, completion/error rates, SLO goodput, KV occupancy and retries, including failures and rejections. The production capacity is the highest repeatably stable *SLO-compliant* demand with safety headroom, not the largest observed raw token rate.”

Research references:

• NVIDIA AIPerf — Benchmarking LLM Inference at Scale
• NVIDIA AIPerf — Metrics Reference
• GuideLLM — Finding Optimal Concurrency
• Grafana k6 — Types of Load Testing