How do you design SLO-aware autoscaling policies for production LLM inference GPU fleets?

SLO-aware autoscaling adjusts the number of *ready, compatible GPU serving replicas* so real workloads meet latency and reliability objectives at an acceptable cost. Unlike CPU-based autoscaling, LLM scaling must account for request/token queueing, TTFT, inter-token latency (ITL/TPOT), KV-cache memory, prefix reuse, model-loading delay, and whole GPU or tensor-parallel groups. The controller should add capacity *before* long queues violate SLOs, remove capacity cautiously, and maintain sufficient headroom for bursts and failures.

1. Why ordinary CPU/GPU-utilization autoscaling is insufficient

• A GPU near 100% utilization can still meet latency targets with effective batching; a GPU at lower utilization might have a long queue or poor cache locality.
• One 32k-token prompt can consume far more prefill work and KV capacity than many 100-token prompts.
• Request-count queue depth is an early signal but not a complete measure of queued tokens or predicted work.
• TTFT rises with queueing and long prefill; ITL/TPOT rises when decoding becomes congested. They need separate attention.
• Loading a large model or provisioning GPUs may take minutes; a controller that reacts only after p95 TTFT breaks may be too late.
• Some model replicas need 2/4/8 GPUs; scale in whole compatible replica groups, not fractional GPUs.

2. Four metrics and the control policy

Leading indicators: offered incoming prompt/output token-work rate, waiting prompt tokens, active sequences, queue age, KV-block utilization and available ready replicas.

Lagging or validation indicators: observed p95/p99 TTFT, ITL/TPOT, joint SLO-compliant goodput, deadline misses and rejection/error rates.

A useful policy combines:

1. Capacity planning baseline: benchmark goodput per replica for the *actual* workload mix.
2. Queue/KV pressure triggers: add replicas when sustained queued work or KV occupancy predicts imminent TTFT/ITL violations.
3. SLO guardrails: increase capacity or enable overload protection when p95 latency or deadline errors cross thresholds. Measure under enough request volume to make percentiles meaningful.
4. Scale-down safety: keep a lower threshold and a stabilization/cooldown window to avoid oscillation, then drain idle-capable replicas without interrupting active streams.

KServe documents Prometheus/KEDA autoscaling on vLLM waiting-request and KV-cache metrics; vLLM Production Stack offers a KEDA/Prometheus integration. NVIDIA Dynamo Planner describes SLO-driven scaling with latency and KV telemetry. Treat these as distinct supported implementation options, not universally equivalent features.

3. Beginner-friendly numerical example

Suppose one ready GPU model replica sustains 6 successful SLO-conforming requests/second for the measured production workload at the chosen target operating level, with TTFT ≤ 1 second and TPOT ≤ 50 ms/token.

• Normal incoming load = 12 requests/s, so ceil(12/6)=2 replicas are required.
• An upcoming traffic burst is forecast to reach 25 requests/s.
• Minimum needed at the same safe capacity = ceil(25/6)=5 replicas.
• Add 3 ready replicas before the burst if model warmup and GPU provisioning time are significant.
• If one-replica failure tolerance must also be preserved during the burst, provision 6 replicas, because losing one still leaves 5 × 6 = 30 requests/s safe modeled capacity.
• At an illustrative $2/GPU-hour and one GPU per replica, cost rises from $4/h for 2 replicas to $10/h for 5, or $12/h for 6 with failure headroom. These prices are hypothetical, not live cloud rates.

Critical nuance: ceil(load / per-replica goodput) is a planning approximation, not a guarantee. Traffic variance, queueing, token mix, cache warmness, uneven routing and actual acceptance/error policies determine achieved SLOs.

4. A concrete hybrid scaling rule

Start with a measured safe per-replica capacity c, observed/forecast offered work rate λ, minimum replicas r_min and configured headroom:

• r_demand = ceil(λ / c).
• r_queue = ceil(total_queued_token_work / desired_queue_work_per_replica) or another calibrated backlog estimator.
• r_target = clamp(max(r_demand, r_queue, r_kv, r_slo), r_min, r_max), where extra terms reflect calibrated KV and SLO pressure.
• Apply fast, rate-limited scale-up, slower scale-down with hysteresis, and a configurable maximum number of GPUs provisioned per decision.

This is a conceptual controller design—not a direct KEDA or Kubernetes configuration. Do not add uncalibrated metrics with incompatible units. A queue cap suitable for 512-token messages may be ineffective for 32k-token document prompts.

Scale-down example: if traffic falls from 25 to 12 RPS, do not immediately remove 3 replicas. Keep the desired count until an adequate low-load window passes, then stop routing new requests to selected replicas, complete/cancel existing work according to deadlines, release KV state and terminate them safely.

5. Startup, model warmup and bursts

• Track GPU node provisioning + image pull + model weight loading + readiness + warmup as one time-to-ready distribution.
• Predictive prewarming helps known daily/weekly surges; reactive scaling may be too slow for a sudden 30-second burst.
• A replica should receive new requests only after model/version/tokenizer/adapter compatibility, KV allocation and readiness are confirmed.
• While new capacity is unavailable, use bounded queues, admission control and appropriate 429/503 responses; unbounded queueing does not preserve the SLO.
• Reserve some ready capacity or use pool-level scheduling for predictable SLOs; trade increased idle GPU cost against user latency and reliability.

6. Multi-GPU replicas and disaggregated models

• A tensor-parallel serving group using four GPUs is one replica for admission purposes, not four independent request-serving replicas.
• A disaggregated deployment has distinct prefill, decode, KV transfer and sometimes router bottlenecks.
• Tune prefill and decode counts using their *own* bottleneck telemetry; scale one role without overproducing KV transfers for a saturated downstream role.
• Some infrastructure controllers scale complete tensor-parallel groups as atomic units. Protect the network and model-store limits that might delay simultaneous launches.
• Question 244 covers detailed sizing of individual disaggregated prefill/decode pools; this lesson focuses on fleet-wide SLO-based scaling controls, warmup, hysteresis and economics.

7. Production incident: autoscaler reacts too late

Incident: A chat service runs two one-GPU replicas, both at high decode occupancy. A burst increases demand from 12 to 25 RPS. The controller scales only when aggregate GPU utilization stays above 90% for several minutes. New model pods take 3 minutes to become ready; meanwhile p95 TTFT climbs from 700 ms to 5 seconds, queued requests grow and retries amplify overload.

Root cause: GPU utilization was a lagging/insufficient signal, queue tokens were unmeasured, and the controller did not consider model startup lag.

Fix: Establish safe per-replica goodput, alert/scale on early queue-token/KV pressure, prewarm replicas for forecast peaks, use fast scale-up but bounded GPU provisioning, propagate cancellation and enforce token-aware admission during the warmup gap.

Prevention: Replay realistic bursts and replica failures; verify both the *time to ready* and the recovered p95 TTFT/ITL. Test downscale/draining so live generations are not interrupted.

8. Testing and operational guardrails

• Run closed-loop and open-loop tests, emphasizing offered RPS rather than only completions.
• Record p50/p95/p99 TTFT, ITL/TPOT, joint SLO-compliant goodput, deadline misses, 429/503, pending token work and KV occupancy.
• Measure autoscaler recommendation vs actual ready replica count, pending GPU nodes, cold-start time and resource quotas.
• Compare time-to-scale, queue overshoot, stability/flapping, cost per SLO-successful request and N−1 recovery.
• Break down metrics by tenant/model/context length so one busy fleet does not conceal another fleet's failures.
• Apply rollout safeguards and version checks: a KEDA trigger expecting particular vLLM metric names may not work after incompatible engine/adapter upgrades.
• Rollback rule: if p95/99 SLO breaches, queue growth or cost escalation exceeds the preset threshold, revert to a safe minimum pool size/policy and inspect metric accuracy.

9. Interview-ready answer

“I would benchmark safe SLO-goodput per compatible GPU replica, forecast or observe offered token work, then scale on early queued-token and KV pressure with TTFT/ITL guardrails. I'd account for model startup time, keep N−1 headroom where required, scale whole multi-GPU replicas, and downscale slowly only after draining. I'd evaluate p95 TTFT, TPOT, goodput, errors, time-to-ready and cost per good request, rather than CPU/GPU utilization alone.”

References:

• KServe — Autoscale InferenceService with LLM metrics
• vLLM Production Stack — KEDA Autoscaling
• NVIDIA Dynamo — GPU inference autoscaling
• Google Cloud — Best practices for LLM inference autoscaling