How do you design a production LLM inference benchmark that reproduces realistic traffic and measures tail latency?

A production-realistic LLM inference benchmark recreates how actual users submit prompts—not just a fixed batch of identical requests—and measures how well the system behaves at the tail (p95/p99), where queues, long prefills, slow decodes, and bursts can cause SLO breaches. The main challenge is workload fidelity and measurement validity. It is different from basic inference profiling (Question 189), cache benchmarking (Question 240), and calculating goodput (Question 257).

1. Why uniform synthetic benchmarks mislead

• Real traffic mixes 100-token chat messages, 8k-token RAG queries, long document prompts, and responses from one token to thousands of tokens.
• The workload is often bursty and contains changing numbers of users; averages hide short high-load periods.
• Prefix reuse, prompt templates, tools, streaming, and output-length distribution affect prefill, KV-cache occupancy, and decode load.
• A fixed concurrency test may slow down its request source when the server is slow, hiding the backlog seen under an independent open-loop arrival process.
• A server can post high average tokens/s while some customers see multi-second TTFT or unacceptably slow streaming.

2. Build a representative workload

1. Collect a privacy-safe, approved traffic trace with relative request arrival timestamps, tokenized prompt lengths, observed/generated output lengths, model/adapter, region, streaming setting, request class, tenant tier, and cache-hit eligibility. Strip secrets, personal information, and authentication credentials.
2. Preserve useful correlations, such as long input prompts with particular output lengths, bursty demand from specific workflow types, and repeated system prefixes. Randomly shuffling input/output lengths independently can create unrealistically easy combinations.
3. Stratify by workload: interactive chat, RAG, agentic/tool calls, code generation, and asynchronous summarization. Give each a documented share of the requests or tokens, grounded in measured production traffic.
4. Replay arrival times at their original relative spacing, or generate a trace-calibrated pattern (Poisson or gamma/bursty arrivals). Include ordinary load, realistic spikes, and a peak or failure scenario.
5. If a real trace cannot be used, generate synthetic prompts and token-count distributions, mark the benchmark synthetic, and explain which production behaviors were not reproduced.

Security note: Logs can retain tenant IDs, prompt text, tools and retrieval content. Use least-privilege trace access, redact/tokenize appropriately and avoid sending production prompts to unapproved providers.

3. Open-loop versus closed-loop testing

• Open-loop: requests are submitted according to an external schedule (for example 20 arrivals/s), *whether or not previous requests have finished*. This reveals queues and overload under independently arriving user traffic.
• Closed-loop: a fixed number of clients issue their next request only after the previous one completes. This models some interactive user journeys but can self-throttle when latency increases.
• Use both when meaningful. For independently arriving API requests, open-loop is the primary capacity test; closed-loop helps understand per-session behavior.
• Measure client-side queueing if the load generator caps concurrency. Otherwise a nominal 20-RPS open-loop run might actually apply less load to the backend during a stall.
• Track offered RPS, actual dispatched RPS, completed RPS, rejected RPS and retries separately. Do not assume these are interchangeable.

4. Beginner-friendly numerical example: how the tail can be hidden

Suppose 100 completed requests have the following TTFT measurements:

• 95 requests: 400 ms each.
• 5 requests: 4,000 ms each.

Calculate:

1. Mean TTFT = (95 × 400 + 5 × 4000) / 100 = 580 ms.
2. Median (p50) TTFT = 400 ms.
3. With the nearest-rank percentile convention, p95 is the 95th sorted value = 400 ms.
4. p99 is the 99th sorted value = 4,000 ms.

Interpretation: A 580-ms average sounds good, yet the slowest 5% wait 4 seconds for the first token. The result also demonstrates why a p95 alone can conceal a small group of severe outliers: show p99, histograms, request-length slices, and the fraction violating each explicit SLO. Percentile interpolation conventions vary between tools.

If a 1-second TTFT SLO applies, 95 of 100 completed requests pass TTFT, but this does not establish *overall goodput* without checking TPOT, full completion, errors and the observation window.

5. A practical two-stage benchmark plan

Stage A — Controlled baseline

• Fix model, weights, tokenizer, GPU, inference engine/version, precision, scheduler settings, prefix-cache condition, and request lengths.
• Run repeated low-load and concurrency-sweep tests to learn basic capacity and eliminate client-side bottlenecks.

Stage B — Production-shaped load

• Replay anonymized request traces or set approximate production token-length distributions and arrival timing.
• Use *warmup*, several steady-state windows, peaks and recovery; keep measurement windows and failure counting explicit.
• Test a traffic mix: short chat, long-document prefill, long generation, shared prefix reuse, cache misses, plus intermittent disconnects and deadlines where relevant.
• Sweep offered rates across the saturation knee, not just concurrency, and repeat runs with documented random seeds.
• Run from a load generator with enough CPU, networking and file-descriptor capacity; benchmark the same network path users take, or separately report in-cluster baseline measurements.

6. Example AIPerf command

NVIDIA AIPerf supports OpenAI-style chat endpoints, synthetic input/output length distributions and arrival patterns. The following is an illustrative command; verify flags against the installed version:

BASH
aiperf profile \
  --model Qwen/Qwen3-0.6B \
  --endpoint-type chat \
  --streaming \
  --url localhost:8000 \
  --request-rate 10 \
  --arrival-pattern poisson \
  --synthetic-input-tokens-mean 512 \
  --synthetic-input-tokens-stddev 128 \
  --output-tokens-mean 128 \
  --output-tokens-stddev 32 \
  --random-seed 42 \
  --request-count 200

This example comes from NVIDIA's AIPerf benchmarking guide and gives a Poisson request arrival process with variable input/output lengths. It is a starting point; to reproduce real production traffic, replace or extend the synthetic inputs with approved trace replay.

Alternative tools include GuideLLM for configurable real/synthetic data, rate/concurrency sweeps and SLO analysis, and vLLM benchmark serve for configurable request rate, burstiness, concurrency and request-level percentile metrics. Check current CLI definitions because option names may evolve.

7. Metrics that must be reported

• Per-request latency: p50/p90/p95/p99 TTFT, inter-token latency (ITL), total request latency, and queue waiting time if observable.
• Serving capacity: offered/accepted/completed RPS, generated tokens/s, input tokens/s, concurrency, success/rejection/timeout/error rates.
• SLO compliance: percentage of original requests meeting all required objectives and joint SLO-compliant goodput. Do not count an error or timeout as a fast success.
• Workload fidelity: token-length percentiles, prefix-hit mix, arrival-rate time series, burstiness, request class and cache warming state.
• GPU/system: memory/KV occupancy, preemptions, CPU/load-generator saturation, network, per-worker queue depths and utilization.
• Slices: metrics by prompt-length bucket, output-length bucket, tenant priority, model/adapter, and endpoint; aggregates can conceal disadvantaged groups.

8. Avoid coordinated omission and other measurement mistakes

• A closed-loop client that sends fewer requests during a slowdown can under-sample the worst congestion. Compare scheduled arrival time with actual submission and completion times.
• If a benchmark client has a concurrency limit, record client queue time separately. The vLLM benchmark tooling exposes client-queue-related metrics with the appropriate configuration.
• Do not silently omit timeouts, rejections, cancellations or unfinished requests from user-facing failure/SLO rates.
• A load generator constrained by CPU/network can make a model look faster or slower than it is; cross-check achieved request rate and producer resource use.
• Warm caches can make repeated prompts unrealistically easy. Test cold/warm phases or reproduce the real prefix-reuse ratio, with safe per-tenant isolation.
• Tiny samples provide unstable p99 estimates. At 100 requests, the nearest-rank p99 depends on almost the last observed request; collect enough observations per important traffic slice and repeat runs.
• Compare runs with the same request mix, network path, model and dataset seed; otherwise a changed output distribution can explain apparent speed improvements.

9. Production incident: the benchmark said “fast”

Incident: A chatbot passes a laboratory test with a mean TTFT of 450 ms at 100 concurrent users, but production p99 TTFT exceeds 4 seconds during morning traffic spikes.

Diagnosis: The test sent fixed 512-token prompts to a warmed prefix cache using closed-loop clients, while the live system had bursty open-loop arrivals, RAG prompts from 200 to 20,000 tokens and mixed decode lengths. The generator also throttled new requests during stalls.

Root cause: Unrepresentative request lengths, arrival schedule and cache behavior concealed the long-prefill queue and KV contention.

Fix: Replay approved traffic traces with arrival timestamps and correlated token lengths; report p95/p99 TTFT/ITL and success/error outcomes by workload slice; vary offered RPS around the goodput knee; confirm the generator is not overloaded.

Prevention: Make a trace-based mixed-workload benchmark a release gate after model, routing, scheduler or cache changes. Store benchmark configuration and outputs for repeatability.

10. Interview-ready answer

“I would design a two-stage benchmark: a controlled model/hardware baseline, followed by privacy-safe production trace replay with realistic arrival times, prompt/output length correlations, cache reuse and streaming. I'd use open-loop offered rates for capacity evaluation, measure p95/p99 TTFT/ITL and per-request SLO outcomes, and include errors, timeouts and client-side queueing. I would validate the load generator and choose operating points by sustainable SLO-compliant goodput, not peak tokens/s.”

References:

• NVIDIA — Benchmarking LLM Inference at Scale with AIPerf (September 2026)
• NVIDIA AIPerf — Comprehensive LLM Benchmarking
• Red Hat — GuideLLM for Real-World LLM Inference
• vLLM Benchmark CLI