How do GPU cold starts, model loading, and prewarming affect LLM inference autoscaling?

GPU cold starts are the delays between deciding to add an LLM inference replica and having that replica *ready to serve compliant requests*. They can dominate autoscaling response time because the system must obtain GPU capacity, fetch or mount checkpoint shards, load weights into accelerator memory, initialize the runtime, compile/capture kernels, and confirm serving readiness. Prewarming moves some or all of this work ahead of the burst, trading extra memory and idle-GPU cost for lower time-to-ready and fewer missed latency SLOs.

Question 261 designed the overall autoscaling policy. This lesson narrows in on startup-path bottlenecks, model-loading optimizations, readiness, prewarming, and the economics of reserve capacity, rather than restating general scaling triggers.

1. Break down the end-to-end cold-start path

1. Detect demand: monitoring/forecasting interval plus autoscaler decision delay.
2. Acquire capacity: Kubernetes scheduling, GPU node provisioning or allocation, driver/runtime readiness, and placement of the correct multi-GPU topology.
3. Prepare the container: image pull, startup, secrets/config and network/storage mounts.
4. Stage model weights: download from object storage or read from local/shared filesystem; fetch tokenizer and model/adapter metadata.
5. Initialize the model: deserialize/checkpoint-map shards, allocate and load parameters into GPU HBM, configure tensor parallelism/quantization and reserve KV/cache/workspaces.
6. Optimize engine startup: kernel compilation, torch.compile, CUDA graph capture and any runtime-specific preparation.
7. Warm up and become ready: exercise supported shapes, verify a simple inference, advertise version/adapter compatibility, then allow the router to send traffic.

These phases sometimes overlap. Critical path duration is not necessarily the sum of all observed stage timers when work executes concurrently.

2. Beginner-friendly numerical example

A hypothetical service experiences the following serial cold-start times for *one two-GPU model replica*:

• GPU placement and node readiness: 80 s
• Container and configuration: 20 s
• Checkpoint transfer/load: 70 s
• Engine compilation and capture: 40 s
• Warmup/readiness: 10 s

Total = 80 + 20 + 70 + 40 + 10 = 220 seconds, approximately 3 minutes 40 seconds before the new replica can serve.

Now suppose a burst arrives in 30 seconds:

• Reactive scale-out cannot make this cold replica ready by the burst.
• If the existing fleet has 12 SLO-safe requests/s and burst demand reaches 24 requests/s, the instantaneous shortfall is 12 requests/s until capacity becomes available.
• Ignoring cancellations, early rejection and transient demand changes, a completely unbounded queue could add approximately 12 × (220 − 30) = 2,280 requests across the remaining 190 seconds if demand stayed constant and the new replica resolved the deficit upon readiness.
• A realistic production system should bound queued work and shed/defer requests, not accumulate thousands beyond their deadlines.

3. Compare warm-start strategies

A. Keep a fully ready spare

• A GPU replica already loaded, warm and in the router's compatible capacity pool can accept eligible work almost immediately.
• Best latency protection, but incurs ongoing reserved GPU/HBM cost and potential wasted idle capacity.

B. Keep a preloaded checkpoint on node-local NVMe/shared PVC

• Avoid repeated remote downloads and registry/object-storage bottlenecks.
• Still pay GPU scheduling, model construction, disk-to-HBM transfer, compilation and readiness costs.

C. Cache compilation/engine artifacts

• Reuse compatible compiled kernels or model metadata to reduce repeated setup. Cache validity depends on GPU architecture, model settings, parallel topology, dtype and runtime version.
• A warm start with cached artifacts is different from an already-running, fully warm replica.

D. Weight preloading and persistent GPU residency

• vLLM's preload feature uses a daemon holding per-GPU weights and an engine ipc_cache loader, subject to fingerprint and GPU-UUID compatibility checks; a cache mismatch falls back to normal disk loading unless configured otherwise.
• This is an implementation-specific optimization and does not mean every vLLM deployment or GPU tenancy model can use it.

E. Preinitialized shadow engine / standby

• NVIDIA Dynamo documents shadow engine recovery as a specialized, preview failover design that keeps initialized standby state near the active engine. A reported B200 case recovered in 7.3 s versus 283 s for a cold restart.
• This is engine recovery after failure, not necessarily provisioning an entirely new GPU replica for demand scaling. Do not equate its failover speed with general autoscale startup time.

4. Memory and bandwidth fundamentals

For a checkpoint containing 70 billion parameters at 2 bytes/parameter, weight storage alone is approximately: 70,000,000,000 × 2 = 140,000,000,000 bytes ≈ 130.4 GiB.

At an ideal aggregate 5 GiB/s effective transfer rate, moving 130.4 GiB takes at least 26.1 seconds. Real initialization can be much longer due to per-file I/O, decompression, CPU construction, data layout, network contention, checkpoint sharding and GPU allocations. Parallel tensor-sharded loading may spread bytes and work across several GPUs but adds communication/coordination complexity.

Beware shared storage stampedes: if 10 replicas simultaneously read a 130.4 GiB checkpoint from the same storage tier, the fleet may demand about 1.3 TiB of reads if no effective cache or deduplication exists. This can saturate network/storage and cause scale-out to become *slower as more replicas are requested*. Stage weights, limit concurrent launches or use a distributed model cache.

5. Prewarm based on demand and startup lead time

1. Measure p50/p95 time-to-ready by model, GPU group, node pool and cache state.
2. Forecast token-work demand and spike timing; issue scale-up sufficiently before peak when demand is predictable.
3. Maintain a minimum warm replica pool for high-priority, latency-sensitive workloads.
4. Keep startup in progress visible separately from ready capacity; Kubernetes desired/created replicas are not ready serving replicas.
5. Use safe scale-up waves when model storage or GPU provisioning is rate-limited.
6. Scale down cautiously after sustained low load, drain existing streams and release KV state; remove the correct number of GPU groups rather than killing workers mid-generation.
7. Review idle warm-pool cost against reduced SLO violations and avoided rejected requests.

Example: if measured p95 end-to-end startup is 240 s, a forecast peak at 12:00 suggests starting scale-up no later than roughly 11:56, plus a small operational buffer. For an unpredictable 15-second spike, only already-ready capacity or bounded overload behavior can protect immediate traffic; cold GPU provisioning is too slow.

6. Distinguish readiness from liveness

• Liveness: the process/container is running; not evidence that the correct model weights are loaded.
• Readiness: model/version/tokenizer/adapter compatible, engine initialized, GPU/KV memory allocated, and supported inference path usable.
• Avoid routing production traffic merely because the pod is running or the TCP port is open.
• Probe readiness after full model initialization; ideally verify a small inference under a controlled check without creating excessive load.
• Prevent premature autoscaler scale-down of newly launched but not-yet-ready workers due to misleading utilization or idle metrics.

7. Production incident: scaling creates a model-download storm

Incident: A chat service has two ready GPU replicas and forecasts a high-load period poorly. At noon, demand doubles. Autoscaler requests six additional replicas, all downloading the same very large checkpoint from one shared storage backend. Network throughput collapses, startup jumps from 3 minutes to 9 minutes, queues grow and user p95 TTFT exceeds its target.

Debug:

• Trace autoscaler recommendation → pod scheduled → image ready → checkpoint available → HBM loaded → engine compiled → first successful inference.
• Inspect object-storage request latency, disk throughput, shared PVC IOPS, GPU allocatable capacity, weight-download retries, compilation cache hits and readiness probe transitions.
• Compare time-to-ready for one instance versus many launching concurrently.

Root cause: The policy scaled on desired replica count but did not account for weight distribution contention or distinguish cold/ready capacity. Excessive parallel starts caused a storage stampede.

Fix:

• Stage checkpoint shards in a shared or node-local cache, use compatible parallel-loading configuration, and control startup concurrency.
• Prewarm enough fully ready replicas ahead of expected peaks.
• Drive admission decisions from ready GPU capacity, not replicas requested.
• Maintain bounded queues/deadlines and cancel abandoned work while new capacity becomes available.

Prevention: Benchmark cold versus warm startup with multiple simultaneous replicas, test node replacement and cache misses, track p95 readiness latency, and enforce storage/GPU quota headroom.

8. What to measure and how to choose a strategy

• Cold and warm startup p50/p95/p99, total decision-to-ready time and breakdown per stage.
• Weight transfer throughput (GB/s), storage IOPS, cache hit rate and concurrent model-load saturation.
• Warm capacity cost: paid GPU-hours, standby memory and useful SLO-compliant requests.
• SLO outcomes: tail TTFT/ITL during bursts, admission rejections, queue-token depth and time to recover.
• Reliability: image/model version mismatches, failed GPU allocations, compilation errors, readiness probe failures and failed rollbacks.
• Economics: balance cost of maintaining spare ready GPUs against the measured probability and cost of cold-start SLO violations.

9. Edge cases and limitations

• A scale-to-zero design may be economical for asynchronous or noninteractive traffic but is usually incompatible with strict sub-second response objectives unless an appropriate warm layer or fallback exists.
• Quantized or mixture-of-experts model loading may involve specialized metadata, packing, expert placement or kernel preparation; raw checkpoint size does not capture all latency.
• Some model revisions invalidate compilation caches or weight fingerprints.
• GPUs on the wrong node topology or in another region cannot always be assembled into the required tensor-parallel replica.
• Persistent warm weights/standby engines consume memory or limit GPU sharing; validate isolation/security requirements.
• Avoid blindly reusing shared KV or model caches across incompatible adapters/models or tenant security boundaries.

Interview-ready answer: “I'd measure every stage from autoscaling decision to an inference-ready model, not merely pod creation. Checkpoint download, HBM load and compilation can take minutes, and starting many replicas together can saturate shared storage. I'd maintain a warm minimum pool, preload/copy weights where supported, cache compiled artifacts, prewarm based on p95 startup lead time, and gate routing on readiness. I'd compare SLO violations during bursts against the GPU-hour cost of spare capacity.”

Research references:

• IBM Research / MLSys 2026 — Breaking the Ice: Analyzing Cold Start Latency in vLLM
• vLLM — Preload
• vLLM — Cold/Warm Startup Benchmark
• NVIDIA Dynamo — Model Caching
• NVIDIA Dynamo — Fast Model Weight Loading
• NVIDIA — Shadow Engine Recovery in Dynamo (August 2026)