How do you protect production ML inference services from overload using admission control and backpressure?

An inference service needs explicit overload protection even when it autoscales. Admission control rejects or defers work that cannot finish within its deadline; bounded queues absorb only tolerable bursts; concurrency caps protect model compute and downstream features; backpressure tells callers to slow down; bounded retries avoid multiplying an outage. The objective is not to accept every request—it is to preserve *successful, on-time* predictions and prevent cascading failure.

Curriculum boundary: Question 126 explained *how much capacity to provision and how autoscaling changes it*. Question 127 explains what to do immediately when incoming work exceeds available capacity, especially during autoscaling delay or feature-service degradation. Question 57 addressed retry safety in *training pipelines*, Question 117 introduced asynchronous APIs, and Question 119 compared serving modes. This lesson concentrates on inference-specific request admission, concurrency, deadlines, fairness, backpressure and incident recovery, without repeating the previous sizing or architecture tutorials.

Beginner-friendly production incident

A delivery-risk service uses a warm model with a checkout response-time SLO of p95 ≤ 250 ms. During a promotion, a feature-store slowdown doubles inference time. Clients retry timeouts, more work arrives than replicas can finish, request queues accumulate, and even healthy traffic starts timing out. Adding Pods takes a minute to warm the model. The immediate solution is to stop accepting more in-flight work than can be completed safely, preserve critical checkout requests, and communicate temporary overload to callers.

A practical request path is:

1. Gateway: Authenticate cheaply, validate payload limits and apply tenant/request-rate limits.
2. Admission controller: Apply a global/tenant concurrency budget and request deadline; reject or divert work beyond the limit.
3. Very small bounded waiting queue: Admit only work that can plausibly meet the remaining time budget. Expire stale requests rather than performing useless inference.
4. Inference worker: Cap parallel model executions/GPU batches and monitor ready capacity, batching delay and memory.
5. Feature/dependency bulkhead: Separately cap concurrent feature-store calls, maintain timeouts and circuit-break failing dependencies.
6. Response: Return the authoritative prediction on time; otherwise send an explicit, bounded overload signal or a *business-approved* fallback. Never silently return a low-quality default score.

Numerical example 1: Little's law and concurrency

Suppose each successful inference takes 0.10 seconds end-to-end on average when uncongested and arrives at 120 requests/second.

• By Little's law, average in-flight work is approximately L = λW = 120 × 0.10 = 12 requests.
• If feature latency doubles end-to-end time to 0.20 seconds, average in-flight demand becomes 120 × 0.20 = 24, even with identical RPS.
• A configured global concurrency ceiling of 20 means the service cannot indefinitely admit all 120 RPS at 200 ms work duration. The idealized concurrency-limited completion rate is at most 20/0.20 = 100 RPS.
• At 120 arrivals/sec and 100 completions/sec, backlog grows at 20 requests/sec if nothing is rejected.
• A waiting queue of 100 requests fills in roughly 100/20 = 5 seconds under these simplified rates. For a 250 ms interactive SLO, that queue is already far too large; many accepted requests become stale.

Lesson: A high request-rate allowance is not enough when each request becomes slower. Tune both in-flight concurrency and waiting-time/deadline limits from load tests. Little's law describes averages under appropriate stability conditions; this calculation is an illustrative planning diagnostic, not a promise that a loaded system remains stable.

Numerical example 2: compare bounded admission with an unbounded queue

At an incident peak, 300 requests/sec arrive, but the currently warm fleet can finish 200 requests/sec within the SLO.

• Without shedding, backlog grows by (300 − 200) × 30 = 3,000 requests over 30 seconds.
• At 200 completions/sec, an item behind 3,000 pending requests would have roughly 15 seconds of work ahead of it in a simple FIFO model—nowhere near a 250 ms checkout target.
• If a limiter admits only 200/sec and rejects or safely defers the excess 100/sec for 30 seconds, about 3,000 requests receive an immediate explicit overload response instead of creating a long time-out tail.
• This is not “better availability” by request-success percentage: successful capacity remains 200/sec. It is better containment: accepted checkout requests remain useful, and callers can decide whether to wait, change behavior or retry later.

For user-critical requests, reserve capacity or use fair scheduling rather than allowing the noisiest tenant to consume the entire 200/sec. Where 429s are counted as unsuccessful user requests, report overload rejection rate separately and do not hide it from the end-to-end availability SLO.

Numerical example 3: retry amplification

Suppose 1,000 original inference requests encounter a saturated feature service. If each of three layers can independently make two retries, the worst-case attempts reaching the deepest service can be 1,000 × 3 × 3 × 3 = 27,000 (the initial call plus two retries at each of three layers), ignoring early success and cancellation.

Even one retry layer with 2 extra attempts can increase attempted traffic by up to 3× during failure. Configure one carefully chosen retry boundary with a total deadline, capped attempts, exponential backoff with jitter, and a retry budget. For instance, allow at most 50 retry attempts for every 1,000 original requests over a short control window: this caps added attempts at 5% (subject to measured budget implementation). Never retry invalid schemas, authorization failures or known non-retryable effects. For external writes, require idempotency safeguards.

AWS's guidance specifically warns that retries can magnify overload and recommends backoff, jitter and disciplined retry placement; it is not a mandate that every request must retry.

Admission control: rate, concurrency, cost and fairness

• Rate limiting: Token buckets suit brief bursts with a steady average refill. A 100-token burst bucket with a 200-token/sec refill can allow short excess demand, but it does not constrain long-running requests or GPU memory. Fixed-window counters can allow boundary bursts; select algorithms intentionally.
• Concurrency limiting: Semaphores or bulkheads bound current in-flight work. Estimate concurrency from measured service time and utilization, then validate p95/p99 rather than relying on simple averages.
• Work-cost admission: A large-image or 100-page document inference is not equivalent to a short text classification. Consider token count, pixels, sequence length, model version, GPU memory and predicted compute cost; reject oversized work or route it to a long-running async path.
• Tenant fairness: Separate pools, per-tenant limits or weighted-fair queues prevent a single client from monopolizing scarce accelerator or feature-store capacity. Preserve an emergency/reserved pool for priority work, without starving other essential traffic.
• Deadline-aware admission: Check the remaining end-to-end deadline at entry and before expensive steps. If a queued request cannot plausibly finish on time, fail early rather than consuming compute for a useless answer.
• Dependency bulkheads: Cap feature-store reads independently of inference workers; scaling model replicas must not overwhelm a smaller shared feature service.
• Circuit breakers: Open temporarily after evidence of dependency failure; allow bounded recovery probes, reset deliberately, and provide a documented failure or fallback path.

A distributed service must choose whether limits apply per replica or globally. Ten replicas each admitting 100 concurrent calls is 1,000 concurrent calls, not a global 100-call safety budget. Global limits need coordinated counters/leases or enforced quotas at the correct boundary, with measured fail-open/fail-closed behavior.

Response and retry contract

• Return HTTP 429 when a caller exceeds a defined usage/rate quota; 503 can represent service-wide temporary unavailability or overload. Include a truthful Retry-After when a retry time can reasonably be estimated, and use a stable machine-readable error reason.
• Propagate deadline and cancellation from gateway to inference to feature calls where supported, so a cancelled checkout does not continue performing expensive work.
• Do not have every layer retry independently; set a global deadline, limited attempt count and retry budget with backoff plus jitter.
• Never make the client retry an idempotency-unsafe operation merely because the server timed out after committing a side effect.
• If a business-approved fallback exists (cached score or simpler model), log the fallback variant, freshness, reason and impact. A “successful” HTTP response from a fallback still requires a correctness policy; do not mislabel it as the primary model's prediction.
• For authenticated tenants, classify limits after a safe identity check but before costly body parsing and model execution. Guard against excessive, untrusted payload sizes even on rejected requests.

Practical policy example (illustrative, not vendor-specific YAML)

YAML
service: delivery-risk
slo:
  p95_end_to_end_ms: 250
admission:
  global_max_inflight: 200
  per_tenant_max_inflight: 20
  per_request_deadline_ms: 250
  max_waiting_requests: 20
  max_queue_wait_ms: 25
  reject_when_deadline_unachievable: true
feature_dependency:
  max_inflight: 80
  timeout_ms: 60
  circuit_breaker: enabled
retry:
  max_total_attempts: 2
  shared_deadline: true
  retry_budget_percent: 5
  exponential_backoff_with_jitter: true
overload:
  reject_rate_limit_status: 429
  reject_service_overload_status: 503
  fallback: approved_only
telemetry:
  label_by_release_and_tenant_tier: true

These thresholds are *illustrative* and must be measured per actual serving topology. A max-inflight count of 200 does not guarantee a 250 ms SLO; 60 ms dependency timeout and 25 ms queue wait must fit with serialization, inference and network time. Aggregate budgeted times should leave tail-latency headroom, and dependency requests may occur in parallel rather than sequentially.

Mode-specific operational behavior

Online inference: Reject early when there is no realistic path to a sub-second response. Avoid turning a synchronous SLA into a long queue. Prefer admission signals, bounded in-flight work and a documented fallback or clear overload response.

Asynchronous inference: Durable queues can absorb more demand, but enforce queue age, per-job quotas, deadlines, consumer concurrency and dead-letter/poison-message handling. A queued job is not automatically successful; monitor time-to-completion and customer-visible backlog.

Batch inference: Prioritize critical deadlines, partition work fairly, set worker quotas and backfill concurrency, and isolate batch jobs from online feature-store capacity. Do not let retries or speculative tasks cause duplicate output commits.

Streaming inference: Control consumer rate, in-flight inference and downstream sink pressure; monitor event lag and watermark freshness. Pausing consumers may protect a dependency but can expand event lag, retention and recovery requirements. Stateful partition/hot-key limits do not disappear when replicas increase.

These are overload-specific extensions of Questions 116–119.

Failure scenarios and runbooks

Incident: p99 rises while CPU remains low

• Inspect feature-store latency, in-flight requests and blocked inference workers. The model is waiting on network I/O rather than consuming CPU.
• Cap dependency concurrency, enforce timeouts, open a circuit if failure thresholds are met, and verify the primary SLO recovers. Autoscaling may worsen the upstream load.

Incident: retry storm after transient 503

• Inspect retry attempt count by layer, client timing and gateway response metadata.
• Apply a shared attempt budget and jitter; temporarily throttle retries and verify *total attempts* fall, not just original request rate. Restore slowly when the dependency recovers.

Incident: one tenant starves every other tenant

• Inspect tenant-level admitted, rejected, in-flight and queued work. Enforce per-tenant bulkheads or fair scheduling, and reserve critical service capacity.
• Avoid a global queue that lets the largest tenant monopolize requests; assess fairness using observed success/latency per tenant.

Incident: queue is small but expired requests consume GPU

• Track deadline-expired requests *during processing*, not just in the queue. Propagate cancellation and check remaining deadline before model execution.
• Record avoidable GPU-seconds spent on work that could not return within its SLO.

Incident: overload rejection causes client retry avalanche

• Use clear 429/503 reasons and bounded retry advice. Coordinate client/server retry behavior; do not promise retry-after a fixed time if recovery cannot be predicted.
• Protect essential traffic with capacity reservations; ensure monitoring counts immediate failures as user-visible unavailability.

Incident: fallback silently changes business decisions

• Halt fallback if its policy or freshness constraints are violated. Trace source model/version and actual decision outcome.
• Treat fallback as its own measurable release/behavior rather than masking it under the primary model's healthy-request metric.

Monitoring, alerting and verification

• Inbound demand: original request RPS, attempted RPS including retries, request cost distribution, payload sizes, per-tenant mix and burst intensity.
• Admission: accepted/rejected/deferred by reason, global and tenant inflight count, queue age, queue length, deadline expirations and remaining deadline at dispatch.
• Serving: p50/p95/p99 *accepted and end-user* latency, rate of on-time successful predictions, GPU/CPU/memory pressure, effective concurrency and throughput by immutable release.
• Dependencies: feature-store p95/p99, inflight, 429/503, circuit state, cache misses, retry amplification and shared quotas.
• Fairness: tenant success fraction, saturation, rejection distribution and age of oldest pending jobs.
• Business quality: fallback usage, invalid scores, missing/stale features and key decision-rate changes.
• Recovery: time to stop overload growth, backlog drain time, time until accepted traffic and overall end-to-end service return within agreed SLOs.

Operational tests: Ramp traffic above measured capacity, inject feature-store slowness and 503s, add skewed large requests, simulate multiple tenants, kill capacity during a model rollout and measure that queues remain bounded, critical requests are protected, rejected requests fail quickly and healthy throughput recovers. Ensure actual user-visible failures remain represented honestly in SLO reports.

Engineering trade-offs and limitations

• Aggressive shedding: preserves useful accepted-request latency but raises rejection rates; appropriate policy depends on user criticality.
• Large queues: smooth longer bursts for asynchronous workloads; disastrous for strict interactive deadlines when queue wait consumes the entire latency budget.
• Static limits: predictable and simpler to audit; require retuning as model cost, feature behavior and capacity change.
• Adaptive concurrency: responds to latency and saturation shifts but can oscillate or misread noisy data; keep hard ceilings and test reactions to metric outages.
• Per-replica quotas: cheap and fast but do not guarantee a service-wide limit after scale-out; centralized coordination adds latency and availability dependencies.
• Priority traffic: protects critical actions but requires trustworthy classification and starvation safeguards.
• Caching/fallback: reduces cost or response time in some cases but introduces freshness and correctness risks; neither replaces a hard overload ceiling.

Common production mistakes

• Believing autoscaling reacts fast enough to every burst.
• Using unlimited FIFO queues behind a low-latency endpoint.
• Enforcing only RPS limits while long inference or feature calls saturate concurrency.
• Treating an HPA desired-replica count as already-ready capacity.
• Allowing each microservice to retry independently and multiply downstream work.
• Reporting only latency for accepted requests while hiding customer-facing rejections.
• Returning unapproved default predictions to make error metrics look green.
• Sharing a single concurrency pool across tenants without fairness.
• Scaling inference replicas into an already exhausted feature-store quota.
• Never testing the overload *rejection path*, cancellation or queue expiry.
• Assuming token bucket rate limiting replaces GPU memory/concurrency control.
• Letting “retry-after” clients synchronize into a new thundering herd.

Takeaway: Autoscaling increases future capacity; admission control and backpressure protect current capacity. A robust inference service bounds concurrent work, waiting time, retries and downstream demand, rejects impossible deadlines early, and measures *on-time useful predictions plus explicit overload failures* rather than raw accepted requests.