How do retry storms and circuit breakers affect the reliability of production LLM inference?

Retry storms happen when many clients retry failed or slow LLM requests simultaneously, making an already-overloaded inference backend even busier. Circuit breakers prevent repeated calls to an unhealthy dependency until there is evidence it has recovered. Together with bounded retries, jitter, deadline propagation, and admission control, they limit cascading failures and wasted GPU work.

1. Why retry storms are especially expensive for LLMs

• A retried request may repeat costly prompt prefill, allocate KV-cache blocks, and generate a long response. Even requests whose original callers have timed out can continue consuming memory and compute unless cancellation propagates.
• Retries may multiply across API client, gateway, agent orchestrator, and model-provider SDK. A seemingly small failure rate can become much larger downstream load.
• The correct response to an overloaded model endpoint is often to shed work or route to genuinely available capacity, not immediately generate more traffic.

2. Beginner-friendly numerical example: retry amplification

Assume 100 incoming requests/second reach a model fleet that can safely complete 100 requests/second.

1. A temporary problem causes 20 requests/second to fail or time out.
2. If each failed request is retried twice, these failures may add up to 40 extra attempts/second, yielding 140 attempts/second total in a simplified sustained snapshot.
3. The added traffic exceeds the fleet's capacity by 40% and can create longer queues, more timeouts and more retries, producing positive feedback.
4. Now imagine three independent layers each permit *up to two retries*: one logical request could generate 3 × 3 × 3 = 27 model attempts in a pathological nested sequence. This is a worst-case amplification, not an expected average.
5. If each attempt consumes a hypothetical 3,000 input tokens, those 27 attempts may repeatedly process up to 81,000 input-token equivalents before any valid answer is returned; prefix reuse may mitigate some prefill but not all processing or output work.

Engineering lesson: coordinate a single retry owner and budget rather than letting every layer retry independently.

3. Retry safely only when the failure may recover

• Potentially retryable: transient network resets, selected 502/503/504 responses, or a 429 where the service explicitly offers a valid retry delay and the request can still meet its deadline.
• Usually not retryable without input changes: authentication failures, malformed requests, unsupported models, excessive context length, or invalid tool arguments.
• Not automatically safe: a read-like LLM completion may be repeated, but it can produce a different stochastic answer; tool-execution agents can duplicate emails, payments or database writes. Use idempotency keys and operation-status checks for side effects.
• Retries must share the original end-to-end deadline (Question 253), be bounded in count, and stop if no meaningful time remains.

4. Exponential backoff with jitter

A retry strategy may use a capped exponential delay:

cap_k = min(max_delay, base_delay × 2^k)

Then draw a random wait uniformly between 0 and cap_k (full jitter), unless a valid server retry hint dictates a longer wait.

Numerical illustration: with base_delay = 0.5 s and max_delay = 8 s, consecutive retry delay caps are 0.5, 1, 2, 4, 8 seconds. Full jitter spreads each retry across its interval, rather than synchronizing thousands of callers at a common boundary.

• Set a global retry budget (for example, a small fraction of the original request rate) and a maximum number of attempts.
• Prefer retrying at one well-defined application layer while downstream SDK retries are disabled or accounted for.
• Respect provider quotas and Retry-After where applicable.
• On sustained overload, backoff is not a substitute for admission control.

5. Circuit breakers: closed, open, half-open

A circuit breaker tracks failures for a specific dependency or endpoint.

1. Closed: normal requests flow through. Record eligible failures and latency.
2. Open: a rolling failure-rate or overload threshold is reached. Fail fast or use a safe, authorized, capacity-available alternative, without continuing to hammer the unhealthy endpoint.
3. Half-open: after a cooldown, allow a *small bounded number* of probe requests; if service health is restored, close the breaker, otherwise reopen it.

Hypothetical policy: open if at least 20 attempts occur in a recent evaluation window and more than 50% encounter eligible failures. After 30 seconds, permit two probes. These are teaching values; select thresholds from measured traffic, avoid low-sample false positives, and account for startup/recovery behavior.

Important distinctions:

• A circuit breaker detects repeated dependency failure and interrupts calls to that dependency.
• A rate limiter regulates how much work is allowed.
• A bulkhead isolates pools/tenants so a failing workload cannot exhaust every worker.
• Backpressure reduces demand when resources are saturated.

These mechanisms complement rather than replace one another.

6. LLM-specific routing and fallback pitfalls

• Breakers should generally be per model deployment, region or endpoint, not one global switch that disables all healthy replicas.
• Failover must consider compatible model, prompt, tokenizer/context window, policy and tenant restrictions. A cheaper or smaller fallback model may not meet answer-quality requirements.
• Never silently route sensitive workloads to a provider or region without appropriate permissions.
• A secondary endpoint needs spare capacity. Blind failover can overload the backup and trigger another cascading failure.
• If an output stream has already delivered tokens, transparently restarting with a different model can duplicate or contradict partial output; treat partially completed streams and tool side effects explicitly.
• When a circuit is open, return a clear temporary error or safe cached answer only if validity, privacy and freshness rules permit it.

7. Production incident: overload amplified by retries

Incident: An inference replica becomes slow during a peak traffic burst. Client timeouts increase, and automatic SDK plus gateway retries triple attempt volume. The model fleet's p95 TTFT rises sharply, queues grow, and KV-cache occupancy remains high even after some callers disconnect.

Root-cause investigation:

1. Compare original logical requests with actual model attempts; compute retry amplification.
2. Trace which layer initiated each retry and whether its previous attempt was cancelled.
3. Inspect 429/503/timeout rates, queue depth, KV occupancy, worker health and latency by endpoint.
4. Verify whether a secondary route is also overloaded.

Fix:

• Centralize retry policy with bounded attempts, full jitter and strict deadline propagation.
• Open a per-endpoint circuit when failures indicate sustained unavailability.
• Reject early rather than queue indefinitely; cancel orphaned generations.
• Route to healthy capacity *only within supported compatibility and capacity limits*.
• Restore gradually via controlled half-open probes and admission ramps.

Prevention: Run chaos/load tests combining replica failures, high-concurrency requests, partial network partitions and retrying clients. Verify no excessive downstream amplification and that recovery is stable.

8. Production metrics and guardrails

• Retry amplification = backend model attempts / original logical requests.
• Retry success and wasted attempts: attempts that recover a request versus those that only increase load.
• Circuit state and transitions: open/half-open time, probe results, false trips and recovery time.
• Latency and service outcomes: p95/p99 TTFT, ITL, final success rate, deadline misses, error rates and SLO-compliant goodput.
• Capacity: active/queued requests, KV-cache pressure, preemptions, GPU utilization and spare fallback capacity.
• Safety: duplicate tool side effects, wrong-model fallbacks, tenant isolation and unnecessary provider charges.

Interview-ready answer: “Retry storms amplify an overloaded LLM endpoint by repeating expensive prefill and decode work. I'd allow only bounded, deadline-aware retries for transient failures, use exponential backoff with jitter, and coordinate retries at one layer. A per-endpoint circuit breaker fails fast when a deployment is unhealthy, then tests recovery with limited half-open probes. I'd measure model-attempt amplification, GPU/KV pressure, tail latency and final user success, while ensuring failover and tool actions remain safe.”

References:

• Google SRE — Addressing Cascading Failures
• AWS — Timeouts, Retries, and Backoff with Jitter
• Microsoft Azure — Circuit Breaker Pattern
• AWS Well-Architected — Control and Limit Retry Calls