How do you design admission control and backpressure to prevent overload in production LLM inference?

Admission control decides whether a new LLM request can enter the inference system now. Backpressure slows or rejects incoming work when the model, KV cache, queue, or downstream workers are near saturation. The objective is not maximum raw requests accepted; it is high SLO-compliant goodput without runaway queueing, out-of-memory errors, or unfair tenant starvation.

1. Why LLM inference needs more than a request-per-second limit

• A 100-token prompt generating 20 tokens and a 20,000-token prompt generating 2,000 tokens cost very different amounts of GPU work and KV memory.
• Prefill requires prompt-processing work, while decode repeatedly reads model/KV state; long outputs occupy running slots and memory longer.
• If every request is allowed into an unbounded queue, time-to-first-token (TTFT) increases even before GPUs run out of memory.
• Admission control sets limits at the gateway, queue and model scheduler; backpressure propagates saturation to upstream clients through bounded waiting, explicit rejection, or client-side concurrency controls.

2. Beginner-friendly numerical example

Assume a hypothetical deployment can safely run 12 concurrent LLM requests and retain 8 additional waiting requests while meeting its latency objectives.

1. Total accepted/in-flight budget = 12 + 8 = 20 requests.
2. A burst of 35 simultaneous requests arrives while all slots are initially available.
3. Admit 12 to running, queue 8, and reject or shed the remaining 15 promptly.
4. Without limits, all 35 might remain in flight and 23 wait; that long queue can miss deadlines and amplify retries.
5. Rejected clients can receive a bounded retry hint; accepted requests should still respect their deadlines. This example is deliberately simplified: a real scheduler also budgets prompt tokens, output tokens and KV memory, not request counts alone.

For a second, token-aware example, suppose a prefill pool sustains 6,000 input tokens/s under its SLO. A 300-ms queue-work budget corresponds to roughly 6,000 × 0.3 = 1,800 queued prompt tokens. If the next eligible request adds 2,500 prompt tokens to a nearly full budget, the gateway may defer, reroute or reject it instead of hiding the overload in a long queue. This is a capacity-planning approximation, not an exact TTFT guarantee; prefix-cache hits, mixed prompt sizes, batching and queueing variability matter.

3. Where to enforce limits

1. Edge/API gateway: authenticate, limit per-tenant request rate and in-flight requests, constrain prompt and maximum output token budgets, and reject obviously invalid or abusive workloads early.
2. Shared inference router: monitor worker health and estimated workload; route to a compatible worker with available queue/KV capacity rather than flooding a busy replica.
3. Model scheduler: cap active sequences, batched tokens, pending queue size, KV allocation, and long-prefill concurrency.
4. Disaggregated prefill/decode pools: maintain separate limits for prefill backlog, KV-transfer bandwidth and decode occupancy; otherwise faster prefill can overload the decode side.

Current vLLM serving controls include --max-num-seqs (model-runner sequence capacity), --max-num-active-seqs (RUNNING admission), --max-num-queued-reqs (an API-server in-flight/queue valve), --max-num-queued-tokens (a prefill token-backlog cap), and --watermark for KV allocation headroom. The documented queued-request and queued-token caps reject with HTTP 503 when reached. Supported flags and precise defaults must be checked against the deployed release.

4. Choose sensible rejection and retry behavior

• 429 Too Many Requests: useful when the client or tenant exceeds a policy quota or rate limit.
• 503 Service Unavailable: useful when healthy infrastructure is temporarily saturated; vLLM's documented queue caps use 503.
• Clients should use exponential backoff with jitter, a strict retry budget and an end-to-end deadline, preferably honoring a valid Retry-After indication.
• Avoid automatic, synchronized retries that magnify overload. Do not indiscriminately retry a side-effecting or partially executed agent/tool workflow.
• Cancel abandoned generation when the client disconnects or its deadline expires, where the serving backend supports it. Continuing an unused long decode wastes KV capacity and GPU time.
• For overload, a fast explicit rejection can be better than allowing requests to time out after seconds in a queue.

5. Fairness and priority

• Protect latency-sensitive user traffic from large batch or low-priority jobs with separate queue budgets, priority bands or weighted fair scheduling.
• Prevent one tenant with long prompts from monopolizing KV blocks and blocking others; bound requests *and* estimated token-work budgets per tenant.
• Priorities should not imply infinite waiting for low-priority users. Apply admission and aging/fairness rules, plus clear shedding policies.
• llm-d documents queueing and backpressure combined with multi-tenant flow fairness and priority. Distinguish priority policy from physical worker capacity.

6. Production incident and debugging

Incident: A chatbot's p95 TTFT rises from 0.6 s to 5 s after batch summarization jobs start. GPU usage is near 100%, pending requests rise, and timeouts trigger aggressive client retries.

Root cause: No bounded queue or prompt-token admission, so long prefills and accumulating generations exceed service capacity. Retried requests create an overload feedback loop.

Engineering response: Separate batch traffic, impose running and queued-token limits, reserve KV headroom, return 503 quickly on saturation, apply retry jitter, and cancel expired requests. Increase pool capacity only after measuring which stage is limiting.

Prevention: Exercise load spikes, mixed prompt lengths, tenant traffic and worker outages in tests. Alert on backlog growth *before* OOM or mass timeout.

7. Measure success

• Goodput: responses meeting correctness and TTFT/ITL/deadline objectives per second, not just raw token/s.
• Admission: accepted, queued, rejected, timeout and cancelled requests, sliced by tenant/priority.
• Queues: p50/p95 wait, waiting requests, queued prompt tokens and age of oldest waiting work.
• GPU and scheduler: running sequences, KV-cache occupancy and preemptions.
• User experience: p50/p95/p99 TTFT, inter-token latency, end-to-end latency and retry amplification.
• Capacity: compare guarded/unbounded behavior under bursts; ensure fast rejections occur as configured instead of hidden queueing.

Interview-ready answer: “I’d enforce bounded in-flight requests, token-aware prefill limits, decode/KV capacity controls and tenant fairness. When capacity is exhausted I’d reject quickly with an appropriate status and retry guidance, rather than queue indefinitely. I’d use cancellation, jittered retries and load-aware routing to prevent amplification, and tune limits against p95 TTFT, ITL, error rates and SLO-compliant goodput.”

References:

• vLLM serve CLI — active sequences, queued requests/tokens and KV watermark
• llm-d — Queue and Prioritize Requests (Flow Control)
• NVIDIA Triton — Request Cancellation
• NVIDIA Triton — Metrics