How do you safely drain, replace, and roll out LLM inference replicas without interrupting streaming requests?

Graceful draining lets existing LLM generations finish while preventing new requests from entering a replica scheduled for replacement. A safe rollout couples routing exclusion, bounded in-flight completion, sufficient replacement capacity, and observable rollback. Inference requests are special because a streamed response may stay open for many seconds or minutes and hold scarce GPU/KV-cache state. Merely removing a pod from a load balancer does not guarantee its existing streams survive.

This is different from Question 262 (making cold replicas ready) and Question 253 (cancelling unwanted work). Here the objective is to preserve useful active work during deliberate rolling upgrades or scale-down.

1. Why standard rolling updates can interrupt LLM streams

• Long prompts and lengthy decode sequences create request lifetimes longer than ordinary HTTP transactions.
• Once an SSE or gRPC streaming connection is established, it is typically attached to a specific engine and KV-cache state.
• Killing the old replica during a token stream can produce a truncated answer, a client-visible disconnect and discarded KV blocks.
• A new replica cannot generally reconstruct and continue the same generation seamlessly unless explicit state migration/resume is supported and validated.
• New replicas may have different model weights, tokenizers, adapters, cache layouts or safety policies; blindly replaying a partial response can duplicate tokens or tool side effects.

2. Safe deployment sequence

1. Provision replacement first: start enough new replicas, load model checkpoints and adapters, warm up the attention/runtime paths, then confirm true inference readiness—not just container liveness.
2. Check version compatibility: route requests only to replicas with the correct model revision, context window, LoRA adapters, region and security requirements.
3. Mark the old replica draining: make it ineligible for *new* sessions at the gateway/router. Validate that Endpoint Picker, load balancer and all ingress paths have observed that state.
4. Keep existing streams alive: allow running requests to decode on the old engine, retaining their KV blocks and connection paths.
5. Wait for an in-flight counter or deadline: track active/queued requests and outstanding SSE streams; avoid creating new queues on the draining replica.
6. Apply a termination ceiling: after a configured grace period, explicitly cancel/close the remaining requests with clear error semantics rather than allowing uncontrolled process termination; longer grace protects long jobs but consumes billed GPU time.
7. Terminate only after drain completion or timeout, then release GPU/KV allocations, unload checkpoint and record success/failure metrics.
8. Roll out gradually: monitor the canary and proceed from a small traffic fraction to larger fractions only when quality, p95/p99 TTFT and streaming completion are healthy.

3. Beginner-friendly numerical example: choosing a drain budget

Assume an old inference replica has 10 active streaming requests when the operator initiates scale-down:

• 6 requests are estimated to complete within 15 seconds.
• 3 requests need 40 seconds.
• 1 unusually long request needs 120 seconds.

Suppose the application uses a 60-second Kubernetes termination grace period, with 15 seconds reserved for traffic-propagation/preStop handling and 45 seconds for engine draining.

Illustrative result:

• The six 15-second requests and three 40-second requests may finish normally if their estimated durations hold and the termination signal/transport are handled correctly.
• The remaining 120-second generation will not finish in 45 seconds and needs an explicit cancellation/error path, a longer grace policy or a workload-specific route to a non-draining pool.
• 9 of 10 active requests finish within the 45-second drain window in this simplified example (90%), but the result is not guaranteed by the timeout setting alone.
• Setting a 120-second grace *may* allow the last request to finish, but raises the period during which the draining GPU stays unavailable for new traffic and still incurs cost.

KServe v0.19 example: its release documentation describes a preStop sleep of 15 seconds to let routing detect endpoint removal, and a vLLM --shutdown-timeout derived from the remaining grace period (45 seconds for a 60-second example). These are implementation-specific settings, not Kubernetes-wide defaults. <https://kserve.github.io/website/blog/kserve-0.19-release>

4. Why preStop and grace-period accounting matter

• Kubernetes invokes a preStop hook before TERM, and that hook's runtime counts against the pod's overall termination grace period.
• With terminationGracePeriodSeconds: 60 and a 15-second preStop, the application has roughly 45 seconds left—not another full 60 seconds—before forced termination.
• Terminating EndpointSlice endpoints have ready=false; a separate serving condition may indicate whether existing connections can still be served.
• A router/load balancer must actively respect the terminating/draining state and its own connection-drain behavior; do not rely on readiness toggles alone for every external gateway.
• SIGTERM handlers should stop new admission, finish or cancel pending work, and exit within the remaining budget. A hanging preStop or overly short grace triggers SIGKILL.
• Do not confuse liveness and readiness: removing a drained worker from routing is a controlled action, not a crash requiring immediate restart.

5. Rolling-update capacity and availability calculations

Assume a deployment with 4 replicas that can each deliver 5 SLO-compliant requests/s on the current traffic mix:

• Total benchmark-safe serving capacity = 20 requests/s.
• Live demand = 16 requests/s.
• If one replica is removed before a replacement becomes ready, capacity falls to 3 × 5 = 15 requests/s, below demand. The queue grows and new users may miss TTFT SLOs.
• A surge replica made ready before terminating the old worker preserves at least 20 requests/s during the transition, assuming routing is effective.
• If long-lived streams keep the old replica busy, count its GPU usage but do not assume it is available to accept new traffic. Capacity accounting must distinguish active/draining and ready-for-new-requests.

Capacity planning should consider Kubernetes maxSurge / maxUnavailable, PodDisruptionBudget or equivalent protections, GPU quota and topology availability. These settings are not a substitute for actual inference readiness and SLO checks.

6. Canary and progressive rollout

• Start new model/server versions alongside stable replicas; validate startup, health and one real inference path before routing substantial traffic.
• Send a small, explicit traffic fraction to the candidate while keeping stable capacity ready for rollback.
• Observe stream truncation/disconnect rate, p95/p99 TTFT/ITL, queue delays, KV pressure, error rate, structured-output quality and tool/schema correctness.
• Increase the candidate weight gradually only if it passes the predeclared gates.
• If quality or serving health regresses, stop new requests to the candidate and drain its existing streams when safe; route new requests back to the stable group.
• KServe's canary rollout documentation describes weighted routing groups for LLMInferenceService, with weights representing relative share, not a literal percentage field. Verify the feature in the deployed KServe version.
• A canary is not a proof that old open streams can migrate. Treat each established streaming connection as bound to its serving engine unless an explicit resume feature exists.

7. Production incident: rollout cuts off answers

Incident: During a vLLM image update, 20% of long-running streamed answers end early with network disconnects. A new pod becomes ready, but the old pod is killed after a short default shutdown period.

Root-cause debugging:

1. Correlate rollout timestamps with streaming disconnects and request completion codes.
2. Inspect pod events, preStop duration, SIGTERM/SIGKILL timing and actual terminationGracePeriodSeconds.
3. Check router logs: did it stop assigning new requests to terminating pods?
4. Check engine active requests, decode durations, KV occupancy and the distribution of stream lifetimes.
5. Confirm the new replica had fully loaded the right model/adapter and sufficient SLO capacity.

Fix: make drain state visible to the router; allow routing propagation before TERM; use a shutdown timeout sized to real active-stream durations; ensure replacement capacity is ready; preserve model/version compatibility; introduce canary traffic and rollback thresholds.

Prevention: regularly test deployment and scale-down during long SSE/gRPC streams. Run forced-termination chaos tests to confirm graceful errors and no duplicate tool side effects.

8. Trade-offs and operational guardrails

• Longer drain grace improves completion of lengthy requests but delays freeing GPUs and can slow rollouts.
• Shorter grace frees capacity quickly but increases stream termination and client retries.
• Drain versus retry: retrying a partially streamed response may produce different text. A generic client retry is not a continuation protocol.
• Stateless ingress versus stateful KV cache: frontend load balancing cannot usually transfer the active decoder's KV memory to another arbitrary worker.
• Disaggregated prefill/decode: drain coordinated worker groups or guarantee that existing decode workers still have the necessary transferred KV state; don't tear down a shared dependency prematurely.
• Storage and model-startup contention: starting several surge replicas at once can create a model-download storm; coordinate with prewarming from Question 262.
• Security: maintain authenticated model routing and avoid crossing tenant/region boundaries during rollback or failover.

9. What to monitor

• Draining replicas, active sessions at drain start/end, stream completion and forced cancellation counts.
• Time from draining announcement to last new request arriving on the old replica.
• Time-to-drain p50/p95/p99 by output-length bucket, plus grace-period expirations.
• Unexpected SSE/gRPC disconnects, partial output, duplicate tool effects and client retries.
• Ready-for-new-traffic capacity, GPU quota, queue tokens, KV occupancy and p95 TTFT/ITL.
• Canary-versus-stable failure, latency and quality metrics; rollback frequency.

Interview-ready answer: “I'd bring up and warm a compatible replacement before draining the old LLM replica. Then I'd remove the old endpoint from new-request routing, keep its current streaming generations alive, and give the engine a bounded drain period covered by Kubernetes termination grace. I'd watch active requests, stream-truncation rate, p95 latency and ready capacity while rolling out gradually. A live stream cannot normally be moved to a different replica just by changing the load balancer, so expiry needs an explicit cancellation/recovery policy.”

Research references:

• KServe v0.19: graceful shutdown with vLLM drain
• Kubernetes: Pods and Endpoint termination flow
• Kubernetes: container lifecycle hooks
• KServe LLMInferenceService canary rollout
• Kubernetes: Pod lifecycle