How do you benchmark and profile LLM inference performance?

Benchmarking LLM inference means measuring how quickly a model-serving system handles a controlled, representative workload. Profiling goes one step further: it identifies *where* the time and GPU resources are spent, so you can explain slow results and choose an optimization.

A useful benchmark answers "How fast and for how many users?" A profiler answers "What is making it slow?"

1. Choose the metrics

• Time to First Token (TTFT): Time from sending a streaming request until the first output token is received. It includes effects such as queuing and prompt prefill.
• Time Per Output Token (TPOT) / Inter-Token Latency (ITL): Time between output tokens after the first token. Tool definitions vary, so keep the same measurement method when comparing systems.
• End-to-end latency: Total time from sending the request until its response finishes.
• Throughput: Successful requests per second (RPS) and generated output tokens per second (tokens/s).
• GPU memory and utilization: Peak allocated/reserved memory, KV-cache occupancy, compute utilization, and memory bandwidth where measurable.
• Tail latency and reliability: Report p50, p95, and p99, errors, timeouts, and rejected requests—not just the mean.

2. Design a fair workload

Fix and record the model version, quantization, GPU, number of GPUs, inference engine, precision, batching settings, and maximum context. Use prompts representative of production and record their input and output token lengths.

Test several cases, for example:

• Short prompt / short response: Chat responsiveness.
• Long prompt / short response: Prefill and TTFT pressure.
• Short prompt / long response: Decode and KV-cache pressure.
• Mixed prompts with concurrent users: Realistic serving behavior.

Run a warm-up before collecting measurements, repeat tests, and ensure the benchmark client and network do not become the bottleneck. Compare systems using the same dataset, tokenizer-aware token counts, decoding settings, and load pattern.

3. Run a load benchmark

A vLLM server can be benchmarked with the `vllm bench serve` command. For example, after starting a matching vLLM API server:

BASH
vllm bench serve \
  --model facebook/opt-350m \
  --dataset-name random \
  --random-input-len 512 \
  --random-output-len 128 \
  --num-prompts 200 \
  --request-rate 10

This is an illustrative load-test configuration, not a performance claim. The model name and endpoint must match the running service, and the server must have sufficient resources.

For more realistic traffic, replay anonymized production request lengths and arrival patterns. NVIDIA AIPerf also supports concurrency sweeps, request-rate tests, and latency/throughput reporting.

4. Interpret the results

Imagine a *hypothetical* 20-second test completes 120 requests, each producing exactly 128 output tokens:

RPS        = 120 / 20 = 6 requests/s
Output TPS = (120 × 128) / 20 = 768 tokens/s

Suppose the same test shows p95 TTFT = 1.8 s and p95 TPOT = 75 ms. Throughput alone might look acceptable, but the tail delays may be poor for interactive chat. Report both metrics, not just tokens/s.

Increase request rate or concurrency gradually and observe where throughput stops improving while queuing, TTFT, or TPOT worsens. The best production capacity is often the highest throughput that still meets latency SLOs (sometimes called *goodput*).

5. Profile the bottleneck

Once a benchmark reveals a regression, collect a small representative trace:

• High TTFT: Inspect request queues, tokenization, long-prompt prefill, and prefill scheduling.
• High TPOT / ITL: Inspect decode kernels, KV-cache reads, memory bandwidth, batching, and synchronization.
• Low throughput with idle GPU: Check scheduling gaps, small batches, CPU overhead, and data transfers.
• Out-of-memory or falling concurrency: Inspect model-weight memory, KV-cache capacity, fragmentation, and allocation policies.

Use PyTorch Profiler for operator and GPU-kernel traces, and hardware profilers or GPU telemetry for detailed device behavior. Be careful: profiling itself adds overhead; use unprofiled runs for final benchmark numbers.

Common benchmarking mistakes

• Measuring one request and claiming production throughput.
• Mixing input and output tokens in one tokens/s figure without labeling it.
• Comparing average latency while ignoring p95/p99.
• Comparing models with different prompt lengths, output limits, or hardware.
• Benchmarking the service with an overloaded load generator.

In short: Benchmark across realistic input lengths and concurrency levels, report TTFT, TPOT/ITL, latency percentiles, throughput and memory, then profile the slowest stage before optimizing.

References: vLLM benchmarking CLI, NVIDIA AIPerf CLI, NVIDIA inference-metrics definitions, and PyTorch Profiler.