How do you design health-aware failover and graceful degradation for production LLM inference?

Health-aware failover sends an LLM request to a different, compatible serving endpoint when the intended endpoint is unhealthy or cannot meet its latency objective. Graceful degradation deliberately offers a reduced but safe service—such as a smaller approved model, shorter output, or delayed batch processing—when full-quality inference is unavailable. Neither automatically improves reliability: fallbacks can overload the remaining capacity, change answer quality, violate tenant boundaries, or re-execute tool actions.

1. Failover versus graceful degradation

• Failover: move from replica A to compatible replica B running the same model or equivalent deployment. This should preserve the intended task as closely as possible; cache locality and latency may change.
• Graceful degradation: use a different, explicitly approved level of service, e.g. a simpler summarization model or a queued asynchronous workflow. This is a product-level trade-off, not an invisible one-to-one replacement.
• Circuit breakers (Question 254) can exclude persistently unhealthy endpoints; health-aware routing decides where an eligible request should go next.
• A healthy endpoint must be checked for actual spare capacity, not just a responsive health-check URL.

2. A robust request-routing path

1. Classify the request: tenant permissions, region/data residency, model/adapter, tools, context length, modality, safety policy and end-to-end deadline.
2. Discover live endpoints and exclude incompatible or unhealthy instances (failed readiness probes, recent high error rates, unhealthy circuit state).
3. Estimate capacity: active sequences, queued prompt-token work, KV occupancy, decode backlog and observed TTFT/ITL.
4. Prefer compatible, loaded-but-healthy capacity while considering prefix cache locality. Do not route onto a replica that cannot fit the new request.
5. Attempt a bounded failover only if the request is safe to replay, the original attempt is cancelled or accounted for, and enough deadline remains.
6. If no suitable replica exists, apply an explicitly approved degradation policy or return a prompt, honest 429/503-style error instead of silently changing semantics.

llm-d's router provides an endpoint picker with health/load/cache-aware filters and scoring. Its precise deployed behavior depends on router version and configuration.

3. Beginner-friendly capacity example

A service normally receives 120 requests/second across three replicas, each able to sustain 50 requests/second at its SLO.

• Total capacity = 3 × 50 = 150 requests/s; planned headroom = 30 requests/s.
• Replica A fails, leaving 2 × 50 = 100 requests/s capacity.
• Offered load is still 120 requests/s, so attempting to fail over every request creates a 20 requests/s shortfall (20% above surviving capacity).
• A safe controller can admit up to about 100 requests/s under these simplified assumptions and shed/defer at least 20 requests/s, or use preprovisioned capacity.
• Suppose 30 requests/s are noncritical summarizations and the business explicitly allows delayed batch handling. Deferring 20 of them leaves 100 requests/s for the surviving replicas without changing interactive request semantics.

Important: These rates are illustrative, not measured GPU benchmarks. Real LLM workload capacity depends on input/output token lengths, concurrency, KV memory, prefix hits, and tail-latency targets; admission should be token/work-aware.

4. Choose a safe degradation ladder

For each workload, declare in advance which changes are allowed:

• Tier 0 — equivalent failover: same approved model/version and required capabilities on another healthy replica, region or provider within policy.
• Tier 1 — operational simplification: reduce maximum output length, unnecessary tool passes or optional context *only when product requirements allow it*. Never silently truncate safety-critical inputs.
• Tier 2 — model downgrade: route an eligible noncritical task to a smaller/cheaper model with separately tested quality thresholds, schema support, tool compatibility and disclosure where important.
• Tier 3 — delayed processing: queue background jobs with durable status and idempotency, when the task does not need an immediate answer.
• Tier 4 — fail closed: for prohibited cross-region processing, regulated or safety-critical workflows, unavailable required tools, or unverified correctness, return a clear temporary failure.

Never interpret “graceful” as permission to fabricate answers or silently reuse stale, personalized cache entries.

5. Timeouts, streaming and side effects

• Share the original request deadline. A failover attempt must use the time left; it must not reset the user's timeout.
• Avoid retry amplification. Set one coordinated attempt budget across gateway, SDK and router.
• Streaming partial output: once a client has received tokens, restarting with another model may duplicate or contradict earlier text. Prefer an explicit interruption/error unless the application has a designed resume protocol.
• Tools and agents: a failed response does not prove a previous email, payment, database write or tool action failed. Use operation IDs, idempotency and reconciliation before retrying.
• KV state: in-flight generation cannot simply move between arbitrary replicas; transfer or resume requires compatible infrastructure. Otherwise a safe new attempt recomputes the prompt.

6. Health signal design

• Liveness: is the process running? This is not sufficient for traffic admission.
• Readiness: can it accept requests with the correct model/version loaded?
• Performance health: rolling error and timeout rates, queue delay, TTFT, KV/cache pressure, preemptions and concurrency.
• Isolation: classify by model, replica, region or provider so one outage does not open a global breaker against healthy endpoints.
• Use smoothed metrics, hysteresis and a bounded recovery ramp to avoid oscillation and flapping.

7. Production incident

Incident: A four-replica endpoint loses one worker. The router moves every failed request to the other three, but they were already near their safe token-work budget. Queueing drives p95 TTFT from 700 ms to 5 seconds; clients retry, creating a cascading outage.

Root cause: failover assumed endpoint availability meant spare serving capacity; it ignored request work, KV pressure and retry budgets.

Fix: isolate the failed endpoint, route only within compatible healthy capacity, shed/slow excess work, cancel abandoned attempts, and degrade only explicitly eligible background tasks. Restore capacity gradually.

Prevent recurrence: chaos-test replica and regional failures during peak traffic, including mixed prompt lengths, streaming disconnects, and client retries.

8. What to monitor

• Original requests versus backend attempts; failover frequency, chosen endpoint and remaining deadline.
• p95/p99 TTFT, ITL, end-to-end latency and successful requests meeting SLOs.
• Per-replica prompt/output tokens, waiting work, KV occupancy, preemptions and capacity headroom.
• Degradation tier distribution, quality failures, incorrect-model routing and tenant/region policy violations.
• Recovery ramp, flapping endpoints, cancellation effectiveness and duplicate side effects.

Interview-ready answer: “I would filter endpoints by model and security compatibility, then use live readiness, error rate, queues and KV capacity to choose a healthy target. Failover must share the original deadline and retry budget, and only use spare capacity. If capacity is insufficient, I would apply an explicitly approved degradation ladder—batch deferral or a tested simpler model for eligible tasks—or fail fast. I would chaos-test the rarely used paths because untested fallbacks can make outages worse.”

References:

• llm-d Router Architecture
• vLLM llm-d Integration
• AWS Builders' Library — Avoiding Fallback in Distributed Systems
• Hugging Face Chat UI — LLM Router fallback routing