How do you estimate GPU capacity and cost per SLO-compliant LLM inference request?

Estimate inference capacity from measured SLO-compliant goodput, then divide the fully loaded cost of the required GPU fleet by successful SLO-conforming requests. A GPU that produces more raw tokens is not necessarily the cheaper deployment if its TTFT, inter-token latency, or error rates violate application objectives. This lesson builds on Question 257 (defining goodput), Question 258 (realistic benchmarking), and Question 259 (finding sustainable capacity); it focuses on converting those measurements into GPU counts, headroom, and unit economics.

1. First check that the model fits

• Approximate model-weight memory = parameter count × bytes per stored parameter, plus quantization metadata and runtime overhead.
• Add memory for KV cache at expected context lengths and concurrency, CUDA/workspace buffers, and fragmentation headroom.
• A serving replica may need multiple GPUs via tensor or pipeline parallelism; replicas are not GPUs.
• Tensor/pipeline parallelism can make one copy fit across GPUs, but communication can reduce efficiency. Data parallelism duplicates a model and raises throughput only after each replica fits.
• NVIDIA Dynamo's deployment documentation expresses a starting check as weights + KV cache + overhead ≤ aggregate VRAM available to one model replica. This is a fit check, not a promise of latency or throughput.

2. Benchmark the correct unit of capacity

For a *specific deployment unit* (for example, one model replica using two GPUs):

1. Replay the expected input/output token distribution, burst pattern, prefix-cache behavior and streaming mix.
2. Sweep offered arrival rate and concurrency.
3. Record completed and rejected requests, TTFT, TPOT/ITL, p95/p99 end-to-end latency and error rates.
4. Compute SLO-compliant goodput: requests completing successfully with all required latency objectives satisfied per second.
5. Choose a sustainable operating goodput with safety margin for worker loss and bursts, not the absolute peak seen in a short experiment.

Goodput is workload- and SLO-specific: a short-chat 8-RPS configuration does not imply eight 32k-document requests per second.

3. Beginner-friendly numerical sizing example

Assume an illustrative two-GPU replica, using an approved model and fixed traffic mix:

• Measured sustainable SLO-compliant goodput: 8 requests/second per replica while meeting defined TTFT and TPOT targets.
• Forecast peak demand: 30 incoming requests/second.
• Choose a conservative 75% target operating utilization relative to that measured capacity.

Sizing:

1. Target capacity per replica = 8 × 0.75 = 6 SLO-compliant requests/s.
2. Required replicas = ceil(30 / 6) = 5 replicas.
3. Each replica uses two GPUs → 5 × 2 = 10 GPUs.
4. Provisioned nominal benchmark capacity = 5 × 8 = 40 requests/s. At 30 RPS, nominal loading is 75%.

Failure-domain check: if one of five replicas fails, remaining benchmark capacity = 4 × 8 = 32 requests/s, but at the chosen 75% target its planned load-safe capacity is only 24 RPS. Thus five replicas do not satisfy the 75%-utilization policy after N−1 failure. For this requirement, solve (replicas − 1) × 6 ≥ 30: six replicas / 12 GPUs are needed. This distinction prevents falsely claiming availability from nominal peak capacity alone.

These figures are illustrative, not measured hardware benchmarks. Scale for token work and the relevant SLO, not generic RPS.

4. Unit-cost calculation

Suppose each GPU costs $3 per GPU-hour (illustrative, not a current market price), and we provision five two-GPU replicas = 10 GPUs for a whole hour.

• GPU cost per hour: 10 × $3 = $30/hour.
• Suppose actual observed successful SLO-conforming production traffic is 20 requests/second sustained.
• Useful requests per hour: 20 × 3,600 = 72,000.
• GPU-only cost per SLO-compliant request: $30 / 72,000 ≈ $0.000417, or approximately $0.417 per 1,000 good requests.

Do not use capacity of 40 requests/s as the denominator when only 20 requests/s actually arrive and pass SLOs. The provisioned GPUs are billed even when idle, so lower demand raises cost per useful request.

For a fully loaded cost, add CPU nodes, networking, shared storage, KV transfer, control plane, monitoring, model-provider/license fees, cross-zone traffic, and engineering/operations charges as appropriate. Normalize to the same region, billing term and utilization.

5. Cost per token versus cost per SLO request

• GPU cost per output token: GPU-hours cost divided by *successfully generated output tokens*. Input prompt tokens require distinct treatment.
• Cost per million tokens: useful for comparing similar request mixes, but can be misleading when one fleet processes very long prompts and another short prompts.
• Cost per good request: total serving cost divided by successful, SLO-conforming requests. This aligns more closely with the interactive user's experience.
• Cost per good request at a quality target: even better when model variants differ in accuracy or format compliance; a cheap wrong answer is not a successful service.
• Different models can output different token counts for the same task. Benchmark *equivalent workload and quality* before declaring a model cheaper.

6. Compare two possible deployments

Assume the same traffic mix and SLO at 30 peak RPS:

• Plan A: two-GPU replicas at 8 good requests/s each and $3/GPU-hour. Five replicas → 10 GPUs, $30/h before shared overhead.
• Plan B: one-GPU replicas at 5 good requests/s each and $4/GPU-hour. At 75% target loading, a replica contributes 5 × .75 = 3.75 RPS, so ceil(30/3.75)=8 replicas → 8 GPUs, $32/h.

Plan B uses fewer GPUs but is more expensive per hour at its illustrative price. Conversely, an option with expensive GPUs may be cheaper *per good request* if it offers substantially higher SLO-conforming throughput. Re-evaluate after N−1, zoning and minimum-instance requirements.

7. Production incident: expensive underutilized fleet

Incident: A team buys 16 GPUs because a synthetic test reported a very high tokens/s figure. At real workloads, p95 TTFT breaches 1-second SLO whenever long prompts arrive, and normal overnight traffic leaves much of the fleet idle.

Debug: Compare actual request/token distributions with benchmark inputs, tail TTFT/TPOT, prefix-hit ratio, each replica's KV occupancy, accepted versus successful requests, and GPU billing hours.

Root cause: Capacity planning used peak raw output tokens/s, assumed every GPU was independently serving one complete replica, and divided cost by theoretical throughput instead of actual SLO-conforming traffic.

Engineering solution: Run representative goodput sweeps; identify the appropriate parallelism configuration and memory fit; use peak-plus-failure headroom, autoscaling where practical, and compute cost per observed good request. Consider separating online and offline workload pools.

Prevention: Review capacity with production traces, repeat benchmarks after model/hardware changes, budget for deployment startup, check N−1 and multi-zone constraints, and alert on goodput-per-GPU and cost-per-good-request trends.

8. Production trade-offs and monitoring

• More headroom: better resilience and tail latency, higher idle-GPU cost.
• More batching/concurrency: may raise raw throughput but reduce latency-compliant goodput as KV or decode contention increases.
• Quantization: can reduce model/KV memory and permit additional concurrency, but confirm answer quality and kernel performance.
• Prefix reuse: may lower prefill costs; do not generalize warm-cache synthetic results to cold, privacy-isolated traffic.
• Disaggregated prefill/decode: size each worker class independently and include KV-transfer and networking costs; not all GPUs have the same role.
• Autoscaling: account for model-loading delay, reserved capacity, minimum GPU commitments, and short bursts that cannot wait for new replicas.
• Metrics: cost/GPU-hour, GPUs/model replica, queue depth, KV use, average and p95/p99 TTFT/ITL, successful SLO requests/hour, GPU-hours per 1,000 successful requests, fallback and rejection rates.

9. Practical decision recipe

1. Verify weights+KV+overhead fit in the planned replica topology.
2. Find sustainable SLO-goodput under a representative production load profile.
3. Divide peak forecast by measured goodput × planned utilization/headroom; round up and account for N−1 and failure domains.
4. Multiply replica count by GPUs per replica and their actual billed hourly price.
5. Divide total cost by observed or realistic forecast successfully completed SLO-compliant requests, not nominal request capacity.
6. Run sensitivity scenarios (traffic ×1.5; long-prompt share doubles; one replica fails; GPU rate changes).
7. Revisit periodically as model versions, latency policies, or observed traffic shifts.

Interview-ready answer: “I'd verify the model and KV cache fit, benchmark sustainable TTFT-and-TPOT-compliant goodput per multi-GPU replica, and size replicas for peak demand with headroom and N−1. Then I'd compute fully loaded dollars per hour divided by *actual* SLO-passing requests per hour. I wouldn't divide by theoretical tokens/s, because it ignores tail latency, unused paid capacity, and workload mix.”

References:

• NVIDIA — LLM Inference Benchmarking: How Much Does Your LLM Inference Cost?
• NVIDIA Dynamo — Model Deployment and Sizing
• NVIDIA NIM — Inference Benchmarking
• NVIDIA Dynamo — Advanced Performance Tuning