How do you calculate and optimize SLO-compliant goodput for production LLM inference?

SLO-compliant goodput measures how many LLM requests finish successfully and meet every required service-level objective (SLO) per second. Unlike raw request throughput, it excludes requests that finish too slowly to be useful. It therefore captures a central production trade-off: adding concurrency or larger batches can increase raw token throughput while decreasing user-visible success within latency limits.

1. Throughput, latency and goodput

• Request throughput: completed requests per second, regardless of whether they met latency targets.
• Output-token throughput: generated output tokens per second. This metric can increase even when users experience more waiting.
• TTFT (time to first token): elapsed time from request submission to the first generated token, including any network/queueing delay included by the benchmark.
• TPOT (time per output token) or mean inter-token latency: average time between generated tokens after the first. Benchmark tools differ in edge-case treatment, so compare like with like.
• Goodput: number of successfully completed SLO-conforming requests per second. A typical request is conforming only if both TTFT and TPOT limits hold; an optional end-to-end deadline can also be enforced.

Formula: goodput (requests/s) = count(successful completed requests meeting every configured SLO) / observation duration (seconds)

A request with TTFT under its target but TPOT above its target does not count, even though the response was technically completed. Incomplete/errored requests should not be counted as successful goodput; explicitly document the benchmark's population, window and missing-metric handling.

2. Beginner-friendly numerical example

Assume a load test runs for 60 seconds with these SLOs:

• TTFT must be ≤ 1,000 ms.
• TPOT must be ≤ 50 ms/token.
• Both conditions must hold for a successful request to count.

Configuration A — moderate concurrency

1. 600 requests complete successfully in 60 seconds.
2. 540 meet both TTFT and TPOT SLOs.
3. Raw throughput = 600 / 60 = 10 requests/s.
4. Goodput = 540 / 60 = 9 requests/s.
5. SLO attainment among completed requests = 540 / 600 × 100 = 90%.

Configuration B — aggressive batching / concurrency

1. 900 requests complete in the same 60 seconds.
2. Only 360 meet both targets.
3. Raw throughput = 900 / 60 = 15 requests/s — a 50% raw throughput increase.
4. Goodput = 360 / 60 = 6 requests/s — a 33.3% goodput decrease versus A.
5. SLO attainment = 360 / 900 × 100 = 40%.

Decision: If serving interactive users subject to these SLOs, A is preferable: it delivers 9 useful requests/s versus 6, despite its lower raw throughput. These are invented educational measurements, not benchmark results from a particular model.

3. The throughput–goodput knee

When offered load is low, most requests satisfy their latency objectives. Increasing load can raise both throughput and goodput.

As load approaches the GPU's saturation knee:

• Queueing increases TTFT.
• Excessive active decode sequences and KV-cache pressure can worsen TPOT.
• More requests cross one or both SLO thresholds.
• Raw throughput may plateau or even rise, while goodput peaks and then falls.

The operational sweet spot is often before maximum raw throughput. Do not size the production service solely at the largest batch size that fits in GPU memory.

4. Build a repeatable benchmark

1. Define the SLOs: e.g., interactive chat requires TTFT ≤ 1 s, TPOT ≤ 50 ms; a long-document job may have different targets. Specify whether success, deadline, and correctness conditions also apply.
2. Choose realistic traffic: fixed model/tokenizer/hardware, representative prompt and output length distributions, prefix-cache hit rates, tenants and streaming settings.
3. Sweep offered load: test increasing arrival rates and concurrency, including bursts. Use a sufficiently long steady-state observation window with warmup handled explicitly.
4. Record each request: arrival, first-token and last-token times; output-token count; queue wait; success/error/cancellation; and any SLO violations.
5. Compute joint conformity per request: count a completed successful request only when every configured objective passes. Do not infer joint goodput by multiplying separate TTFT and TPOT pass percentages without evidence that they are independent.
6. Select the operating point: choose the highest sustainable goodput with sufficient safety margin, acceptable tail latencies, and stable error/rejection rates.
7. Repeat and validate: account for load-generator bottlenecks, variance across runs and differences between open-loop and closed-loop testing.

5. Practical code-style calculation

For each completed request r, using measured TTFT and TPOT in milliseconds:

passes(r) = r.success && r.TTFT_ms <= 1000 && r.TPOT_ms <= 50

goodput = sum(passes(r) for r in completed_requests) / elapsed_seconds

If TPOT is undefined for a one-token response, define a deliberate policy (e.g. evaluate only TTFT or exclude from the TPOT-constrained population) rather than silently treating an unavailable measurement as a pass. GuideLLM defines an undetermined SLO result where a required measurement is missing.

6. How to optimize goodput

• Tune concurrency and batching: sweep max_num_seqs and batched-token budgets; stop when SLO misses increase faster than throughput gains.
• Separate prefill/decode contention: chunked prefill, fair scheduling, or prefill/decode disaggregation may protect both latency phases, subject to hardware and KV-transfer costs.
• Improve routing: choose compatible replicas using queue length, estimated prefill work, KV occupancy, prefix reuse and predicted TTFT/TPOT headroom.
• Reuse computation: exact prefix caching can reduce repeated prefill work, but cache hit patterns must be representative and tenant-safe.
• Enforce admission/backpressure: when saturation is unavoidable, reject or defer excess work promptly rather than allowing all requests to time out. Rejections reduce service availability, so count them in overall user-facing success-rate SLOs even if a completed-request-only goodput metric omits them.
• Cancel abandoned requests: release GPU/KV resources for requests whose client disconnected or deadline expired.
• Autoscale the bottleneck: scale the right worker pool, considering GPU memory, KV-transfer bandwidth and cost; more replicas alone may not fix uneven routing.

7. Production incident

Incident: An engineer doubles the LLM batch/concurrency limit to improve token throughput. GPU utilization rises and output-token throughput grows 25%, but interactive p95 TTFT jumps from 0.7 s to 2.2 s and user timeouts increase.

Root cause: More queued prefills and active decode sequences create contention. Raw tokens/s hides the percentage of requests meeting latency objectives.

Diagnosis: Plot offered RPS against raw throughput, joint TTFT+TPOT goodput, p95/p99 TTFT, TPOT, KV occupancy, scheduler preemptions and rejection rate.

Fix: Roll back to the goodput-maximizing concurrency range; tune prefill token budgets; protect decode scheduling, introduce bounded admission, and route requests based on SLO headroom.

Prevention: Gate deployment changes on SLO-compliant goodput at representative traffic, not on GPU utilization or peak tokens/s alone.

8. Accuracy, fairness and cost caveats

• Pure latency goodput does not measure answer correctness. If the product requires valid JSON, reliable tool calls or factual accuracy, treat these as additional eligibility checks or independent quality metrics.
• A system can artificially raise a benchmark's completed-request-only goodput by refusing difficult requests. Monitor acceptance, successful completion, user-facing availability and rejected workload alongside goodput.
• Report results by tenant, request-length bucket and workload class; an aggregate gain can mask a severe regression for long prompts or lower-priority customers.
• Cost efficiency: SLO-compliant requests / GPU-hour or cost / SLO-compliant request can guide economic optimization, but compare equivalent response quality and workload mix.

9. Research and tooling

• The DistServe paper formalized serving-capacity optimization under separate TTFT and TPOT objectives, motivating phase-specific resource allocation.
• GuideLLM explicitly defines goodput with per-request TTFT, TPOT and optional end-to-end constraints; all configured objectives must pass.
• NVIDIA's GenAI-Perf documents latency and throughput metrics and configurable goodput constraints; its documentation now directs new benchmarking work to AIPerf, so confirm current tool support and metric definitions before choosing a benchmark.

Interview-ready answer: “Goodput is the number of successful requests per second that satisfy all configured latency SLOs, typically TTFT and TPOT. I would benchmark representative traffic across concurrency levels, mark each request conforming only if both limits pass, and find the load that maximizes useful throughput. I would watch rejection and error rates so the metric cannot be gamed by shedding hard requests, and tune batching, routing, caching and capacity based on SLO-compliant goodput rather than raw tokens/s.”

References:

• GuideLLM — Goodput SLO definition
• GuideLLM — Metrics
• DistServe: Goodput-optimized LLM Serving
• NVIDIA — LLM Inference Benchmarking Fundamentals
• NVIDIA — GenAI-Perf documentation