How do you design autoscaling and capacity planning for production ML inference services?

Production inference autoscaling adjusts serving capacity as demand changes, but a safe design must account for model warmup, request concurrency, shared feature dependencies, minimum ready capacity and the actual latency or freshness SLO. Scaling on CPU alone can fail when inference is GPU-bound, blocked on external features or sitting behind a queue.

Curriculum boundary: Question 38 introduced CPU, GPU and accelerator scheduling; Question 39 introduced general resource estimates; Question 119 contrasted serving operating models; Question 120 covered packaging and startup requirements; Question 121 covered staging load testing; Question 122 covered rollout capacity; and Question 125 linked these into CI/CD. This lesson concentrates on runtime capacity control, autoscaler signals and production failure modes after deployment—not general hardware selection or release orchestration.

Beginner-friendly scenario

A delivery-risk endpoint normally receives 80 requests/second but spikes to 300 requests/second at lunchtime. Each *warm* inference replica safely handles 40 RPS while meeting the p95 latency requirement. If the company keeps only two replicas and a new replica takes 60 seconds to download and warm its model, autoscaling that reacts only *after* CPU increases will not protect the first minute of the lunch spike.

The engineer must decide:

• Capacity target: How much tested throughput can each ready replica deliver under the actual request mix?
• Headroom: How much capacity should remain available for sudden demand, node failures and model rollouts?
• Scaling signal: CPU, RPS/ready replica, active requests, queue age, GPU occupancy, event lag or another metric that correlates with service health.
• Reaction time: How long do metric collection, scaling decisions, cluster scheduling, image pulls, model loading and readiness take?
• Safety limits: When to admit, queue, degrade or reject work because scaling will not catch up.
• Cost budget: Minimum warm capacity, maximum replicas, GPU quota, network, storage, feature-store load and idle instances.

Numerical example 1: replicas needed for peak traffic

Assume a load test shows a warm replica can sustain 40 RPS under the specified payload mix while meeting p95 ≤ 250 ms. To retain 25% planning headroom, use a 75% utilization target:

• Sustainable planning rate per replica = 40 × 0.75 = 30 RPS.
• For 300 RPS, required ready replicas ≈ ceil(300/30) = 10.
• With only two currently ready replicas, planned capacity is 2 × 30 = 60 RPS; the gap is 240 RPS.
• If all ten ready replicas are running, planned capacity = 10 × 30 = 300 RPS.

This is a *sizing approximation*, not a guarantee. It assumes traffic is load-balanced reasonably, dependencies scale, and 40 RPS was tested under realistic concurrency. GPU memory limits, uneven request sizes and tail latency can dominate before average RPS saturates.

Numerical example 2: cold-start gap and overload

Suppose incoming traffic jumps from 80 to 300 RPS. Existing warm replicas can handle 100 RPS without violating the tested hard capacity ceiling, and the first additional capacity does not become ready for 60 seconds.

• Excess arrivals during the gap ≈ (300 − 100) × 60 = 12,000 requests.
• If the application queues all 12,000 requests without a sufficient drain advantage, some will exceed an interactive latency SLO by many seconds.
• A bounded queue of 1,000 requests would reach capacity in roughly 1,000/200 = 5 seconds under this simplified constant-rate gap.
• Therefore use minimum warm replicas, predictable scheduled pre-scaling for known peaks, short scale-up feedback loops, admission control, fair queues and documented overload responses.
• A separate queue-capable asynchronous service can buffer tolerable work; ordinary HTTP clients demanding 250 ms responses cannot safely depend on a minute-long cold start.

Do not infer that autoscaler CPU or HPA replica targets guarantee a latency SLO. The observable goal is ready capacity before or during demand, not merely an increased desired-replica count.

Numerical example 3: cost versus protection

Illustrative worker price only: $0.60 per instance-hour, 30-day month.

• Keeping 10 replicas all day costs 10 × 24 × 30 × $0.60 = $4,320 in worker-hours.
• Keeping 3 warm replicas for 20 hours/day and 10 for 4 hours/day costs [(3×20)+(10×4)] × 30 × $0.60 = $1,800.
• Nominal compute difference = $2,520 per 30-day period.
• But this ignores network, feature-store scale, load balancers, storage, autoscaling headroom, cold-start charges and any business cost of missed SLOs. Schedule-based minimums help only when the peak is predictable.

Engineering decision: choose a minimum warm footprint based on the earliest plausible burst and its recovery time; then autoscale above it using tested service signals. Savings that create severe customer-visible timeouts are false economies.

Choosing an autoscaling signal

• CPU utilization: Straightforward for CPU-bound processing, but a sleeping worker waiting on an external feature service may show low CPU while request queues and latency explode.
• GPU utilization or memory: Relevant for accelerator workloads, but utilization alone may be an unreliable request-capacity predictor; monitor memory, batching/concurrency and accelerator queues.
• Requests per ready replica: Often intuitive for online serving if workloads are reasonably homogeneous. Report a denominator based on ready and actually serving replicas, not desired Pods.
• In-flight requests / concurrency: Helpful when request times vary; can react to saturation before CPU rises. Ensure gauges represent work truly underway rather than only accepted traffic.
• Queue length / oldest-message age: Strong indicators for asynchronous or batch workers, provided work is durable and retry-safe. Queue age often captures user-visible delay better than count alone.
• Streaming consumer lag / processing delay: Useful for stream inference, but partitions, hot keys, keyed state and downstream sinks may cap useful parallelism.
• Tail latency and timeout rate: Essential health signals and emergency guardrails. Latency alone may be too delayed, noisy or unstable as the sole replica control input.

Kubernetes HPA v2 supports CPU/resource, custom and external metrics; with multiple metrics it selects the largest replica recommendation, subject to configuration and limits. Custom/external metric adapters must actually be installed, authorized and monitored. KEDA adds event-driven scaling integrations and is commonly used for queue-oriented processing. Neither automatically understands the model's semantic quality.

A practical autoscaling policy (illustrative Kubernetes YAML)

YAML
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: risk-inference
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: risk-inference
  minReplicas: 3
  maxReplicas: 12
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 65
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
    scaleDown:
      stabilizationWindowSeconds: 300

This is a *valid structural example*, not a complete inference autoscaling solution. CPU requests must be configured appropriately, the metrics pipeline must be healthy, and the cluster must have space to schedule replicas. Use a relevant custom metric such as RPS-per-ready-replica when CPU does not reflect workload pressure. Kubernetes documents stabilization windows and scaling policies; the example's min/max values are business-specific. On Kubernetes versions supporting the relevant feature, scaling to zero can be configured for eligible external/object metrics; an HTTP Service does not buffer requests while zero Pods are ready, so it is generally unsuitable for strict interactive latency without an additional buffering/activation layer.

Implementation steps and acceptance checks

1. Benchmark a realistic model release. Measure p95/p99 request latency and maximum sustainable RPS at different concurrency levels, including feature store calls, serialization, payload mix, cold cache and actual hardware.
2. Set a service contract. Declare peak and normal RPS, latency SLO, minimum ready replicas, headroom, recovery time, resource limits, budget and max concurrent work per instance.
3. Measure startup end-to-end. Track metric sample delay, controller calculation, node scheduling, image/model pull, deserialization, warmup and readiness. A newly created Pod is not usable until it is actually ready.
4. Configure proportional scaling. Choose relevant metrics and adapter; cap changes to avoid violent oscillation, use separate scale-up/down behaviors, and consider scheduled baseline increases where peaks are predictable.
5. Protect the request path. Bounded queue, clear timeouts, sane retry budgets and rate limiting. Unbounded retries may amplify demand and make overload worse. Different tenants may require fair admission limits.
6. Check shared bottlenecks. More replicas can increase QPS against a feature store, database, model registry, GPU pool, network egress or external inference endpoint. Agree on dependency quotas and backpressure.
7. Check rollout interactions. Progressive delivery may temporarily run stable and candidate fleets together. Ensure spare GPU/CPU quota and stable fallback capacity, especially during model warmup.
8. Fault-inject scale events. Test traffic spikes, failed image pulls, unavailable metrics, exhausted GPU nodes, unhealthy replicas and dependencies slowing under fan-out.
9. Review cost and SLO impact together. Track idle capacity, overprovisioning, instance-hours per useful prediction, errors, dropped work and p95/p99 in the same dashboard.

Batch and streaming nuances

• Online: Keep minimum ready replicas and a load-aware signal. Scaling is not a substitute for an overload policy when response times are measured in milliseconds.
• Asynchronous/batch: Queue length, age and deadline slack can drive workers. Scale-to-zero may be economical when a durable queue can hold work until workers initialize. Pin a model version per logical scoring job; autoscaling must not change it mid-run.
• Streaming: Scale by sustained arrivals, consumer lag and event-to-output freshness, but preserve event ordering and state redistribution requirements. Increasing consumers beyond available partitions may offer no throughput gain. Stateful redistribution can temporarily increase lag and checkpoint pressure.
• GPU inference: Admission limits and per-replica batching often matter as much as replica count. Scaling a fleet whose memory allocation already fails on startup cannot solve the bottleneck.

These are capacity-specific extensions of the architecture overview in Question 119.

Common production failure scenarios

Incident A: CPU looks normal but p99 climbs

• Evidence: queue depth, in-flight concurrency and feature-store latency rise; CPU remains low because inference threads block on remote calls.
• Root cause: autoscaler observes CPU rather than the limiting dependency.
• Mitigation: instrument true load and feature timeouts; scale only if more replicas help, otherwise limit concurrency and fix upstream capacity.

Incident B: desired replicas increase but none become ready

• Evidence: pending Pods, image-pull failures, GPU quota errors, out-of-memory, model initialization delay.
• Root cause: resource or artifact bottleneck; autoscaler cannot conjure capacity.
• Mitigation: maintain node/accelerator headroom, pre-pull models where supported, reduce model load overhead, use correct startup probes and explicit fallback.

Incident C: repeated scale-up/down thrashing

• Evidence: replica counts oscillate and each new instance warms expensively while latency becomes unstable.
• Root cause: noisy metric, insufficient stabilization and poor model warmup accounting.
• Mitigation: appropriate scale-down stabilization window, controlled scaling policies and minimum warm footprint.

Incident D: scaling makes a database incident worse

• Evidence: model replicas double, feature-store connections and timeouts explode, request success falls.
• Root cause: shared service saturates; extra replicas amplify demand.
• Mitigation: enforce feature-store concurrency budgets, cache where appropriate, shed load and coordinate dependency scaling.

Incident E: missing metrics prevent scale down

• Evidence: HPA custom metric becomes unknown; replicas stay elevated.
• Root cause: metrics adapter stopped exporting the series, or zero was encoded as no data.
• Mitigation: keep zero-valued gauges available; alarm on metrics freshness and diagnose autoscaler conditions. Kubernetes intentionally treats some missing-metric conditions conservatively.

Monitoring dashboard and alerting

• Traffic: incoming/accepted/rejected RPS, request sizes, in-flight calls, queue length/oldest age.
• Experience: p50/p95/p99 end-to-end latency, timeouts, retry rates, error budget burn, user-level outcomes.
• Scaling: desired/ready/available Pods, time-to-ready, scaling recommendation versus effective replicas, stabilization status, pending reasons, GPU quota and node supply.
• Resource: CPU, memory, GPU utilization/memory, batch size, model-load duration, network and feature-store calls.
• Cost: active instance-hours, idle buffer, spend per million successful predictions, broker/feature-store dependency costs.
• Release: model/image digest by replica and autoscaled instance; autoscaling must not accidentally launch a different mutable release.
• Quality: predictive feature freshness and result distribution, which are not guaranteed by healthy capacity.

Alert on failed customer SLO and inadequate available capacity, rather than merely “CPU > 80%.” Also alert on stalled autoscaling, degraded metric coverage, repeated OOMKills and dependency capacity exhaustion.

Trade-offs and limitations

• Aggressive minimum capacity: Better burst protection and simpler recovery; higher idle cost.
• Reactive HPA: Lower steady-state compute cost; can be late for sudden peaks and slow model starts.
• Predictive/scheduled pre-scaling: Useful for known time-of-day patterns; wrong forecasts waste capacity or miss unplanned spikes.
• Scale-to-zero: Strong for sporadic queue-tolerant work; dangerous for latency-sensitive synchronous endpoints without a safe buffering layer.
• RPS metrics: Meaningful for stable request sizes; may misrepresent mixed light/heavy predictions.
• GPU utilization metrics: Important but potentially misleading when large model weights, memory pressure or batching govern throughput.
• More replicas: Can increase total throughput only when storage, state, upstream services, networking, model loading and quotas also support the growth.

Common mistakes

• Choosing min/max replicas from an average traffic graph without peak, cold-start or failure analysis.
• Assuming HPA target replicas are immediately available or p95 latency compliant.
• Using CPU-only scaling on GPU-bound or I/O-bound inference.
• Keeping readiness true before model load completes.
• Sending unlimited retries while scaling catches up.
• Ignoring the quota and cost impact of candidate + stable fleets during a rollout.
• Scaling consumers beyond broker partitions or without state migration tests.
• Using a scaling signal that silently disappears during low traffic.
• Interpreting scale-to-zero as instant HTTP capacity.
• Scaling horizontally when a hot key, remote feature dependency or single sink is the actual bottleneck.
• Optimizing instance cost but never measuring cost per successful, on-time prediction.

Takeaway: Autoscaling is an operational feedback loop, not a capacity guarantee. Benchmark the deployed release, select signals tied to the real bottleneck, plan around warmup and shared dependencies, and preserve sufficient ready capacity to keep latency/freshness commitments during spikes and releases.