Large Language Models

Large language models are deep learning models, usually transformer-based, trained to predict tokens from large corpora. After pretraining, they are often instruction-tuned and aligned so they can follow prompts and assist users.

How do you implement priority scheduling and fairness across tenants in production LLM inference?

Priority scheduling decides which classes of LLM requests should run first, while tenant fairness prevents one customer from monopolizing shared inference capacity. A production scheduler must balance both with real GPU/KV memory limits, time to first token (TTFT), inter token latency (ITL), and service level objectives (SLOs). Simple first come, first served (FCFS) or global request rate limits are often insufficient because LLM requests differ enormously in prompt length and generated output length. 1. Why priority and fairness are different Priority: interactive user requests might deserve shorter waiting time than background summarization. A higher priority can favor online service during contention. Fairness: among requests at the same priority, Tenant A sending thousands of prompts should not indefinitely block Tenant B. Admission/backpressure: controls whether work enters the system at all. It protects resources; scheduling decides which accepted request runs next . This is separate from Question 250. Cost awareness: a request with 20,000 input tokens and 2,000 output tokens consumes more prefill work, decode time, and KV capacity than a 200 token FAQ. Fairness by request count does not necessarily mean fairness by GPU work. 2. Three stage scheduling 1. Authenticate and classify: a trusted gateway derives tenant ID, subscription or workload class, request deadline and authorization; never trust client supplied premium labels. 2. Pick a priority band: for example, high priority interactive chat, ordinary interactive chat, and best effort offline summarization. 3. Pick a tenant within that band: round robin, weighted fair queuing, deficit round robin, or a token work aware policy. Keep bounded queues per tenant. 4. Pick a request within that tenant: FCFS or earliest deadline first (EDF) when deadline estimates are meaningful. 5. Dispatch only with available capacity: check GPU scheduling slots, prefill token budgets, KV occupancy and downstream worker saturation. llm d Flow Control follows this general three tier model: priority band → tenant fairness flow → within flow ordering . Its EPP router can hold requests while model workers are saturated; it currently requires enabling the flowControl feature gate in the documented architecture. Default configuration can behave as global FCFS, so enabling Flow Control alone does not automatically establish round robin tenant fairness. 3. Beginner friendly numerical example Two tenants, A and B , share a model pool with 6 available dispatch opportunities during one short scheduling window: Tenant A has 12 queued requests . Tenant B has 3 queued requests . Both are in the same priority band. With a simple global FCFS queue, assume all 12 A requests arrived first: the next six dispatches could all go to A, leaving B waiting. With round robin fairness between active tenants, the next six dispatches can be A, B, A, B, A, B : A receives 3 slots and B 3 , preventing B from being starved. Now suppose a policy assigns A:B weights of 2:1 . A basic weighted cycle illustration yields A, A, B, A, A, B : A receives 4 slots and B 2 . Important: These examples illustrate dispatch opportunity fairness , not equal GPU seconds, equal tokens or a guarantee of fixed shares every six requests. If A's requests are 10× longer, request count weighting can still give A most of the compute. Real weighted schedulers account for work cost, deficits and availability across longer windows. 4. How token work aware fairness improves things Estimate input prefill cost from uncached prompt tokens and model/hardware characteristics. Estimate decode demand from maximum or predicted output tokens; actual decoding duration is uncertain. Use per tenant work budgets, token quotas or deficit round robin so short requests are not treated as equal cost to very long jobs. A conceptual deficit scheduler adds credits each round and deducts estimated work when scheduling a request. If the request exceeds remaining credit, it waits for more credit, subject to carefully designed latency and maximum request size policies. A shared KV cache budget and concurrency cap protect against a tenant with long contexts consuming all active memory. Avoid allocating resources based solely on estimated output length: estimates are imperfect and must be reconciled with real work. 5. Priority inversion and starvation Problem: Strict priority can continually admit interactive traffic while leaving background jobs waiting forever. Solutions and trade offs: Bounded queues and deadlines: reject or expire work that can no longer meet its objective. Reserve or budget some capacity for lower priority work, where policy requires progress guarantees. Aging: gradually raise the effective priority of old requests, where supported. This protects progress but may soften strict latency guarantees for premium users. Traffic isolation: separate interactive and batch worker pools if their demands are hard to reconcile. Safe preemption: do not assume an already running LLM generation can be interrupted without cost; cancellation or recomputation may be necessary. Priority at the gateway does not itself mean GPU preemption. Fairness versus cache locality: a strict round robin schedule can reduce prefix reuse or batching efficiency. Benchmark the throughput/latency trade off. 6. How this maps to current serving systems llm d: Flow Control identifies a flow using fairness ID + priority , queues flows separately, selects a priority band, then a fairness policy and an ordering policy. Documented options include round robin fairness, FCFS ordering, EDF ordering, saturation detection and configurable priority band limits. A trusted ingress should strip client supplied fairness/priority headers and inject verified values; otherwise a tenant may impersonate another or self upgrade priority. Strict priority between bands and fairness within a band should not be mistaken for weighted fair queuing across all bands. vLLM: The documented scheduling policy supports fcfs and priority; lower numeric priority means earlier scheduling , with arrival time breaking ties. That priority option by itself is not a complete multi tenant fairness or quota system . Apply tenant fairness, budgets, and admission logic at the trusted gateway or orchestration layer when required. Check the deployed version's feature support before choosing any specific router or engine configuration. 7. Production incident: a noisy neighbor Incident: A shared chatbot API's p95 TTFT rises from 600 ms to 4 seconds after Tenant A starts a large document processing job. Tenant B has a modest request rate but its requests wait behind A's long prefills. Debug: Measure queue wait by tenant and band, uncached prompt tokens, running sequences, KV occupancy, per tenant token usage and p95 TTFT. Check if gateway traffic classification is trusted and whether the policy actually uses tenant aware queues. Root cause: Global FIFO and unlimited tenant level work let A monopolize the queue and consume disproportionate GPU capacity. Fix: Put batch traffic in a lower priority band, enforce per tenant token/concurrency budgets, enable same band fairness, cap queues with backpressure and reserve enough resource headroom for interactive SLOs. Prevent recurrence: Reproduce with skewed tenant loads; assert minimum progress for ordinary tenants, p95/p99 TTFT targets for interactive users, bounded rejection rates and no cross tenant priority spoofing. 8. What to measure Per tenant: accepted/rejected requests, queue depth, queue age, request and token throughput, prefill/decode work, and p95/p99 TTFT/ITL. Per priority band: deadline misses, goodput, time spent waiting and share of actual GPU work. Pool wide: KV cache occupancy, preemptions, model throughput, cache locality, GPU efficiency and cost per SLO compliant request. Fairness: distribution of served work compared with allocated weights; track starvation and long tail waiting, not just aggregate request/s. Security: verify that untrusted clients cannot modify tenant or premium priority and that routing/caching respects isolation. Interview ready answer: “I would classify requests at a trusted gateway, queue them by priority band and tenant, enforce per tenant token/concurrency budgets, then use a fairness policy within each band. The engine only receives work when KV/compute capacity exists. Strict priority protects interactive SLOs but risks starving batch work; fairness and aging or reserved capacity address that. I'd measure per tenant wait and GPU work shares, p95 TTFT and SLO compliant goodput rather than assuming request count round robin is fair.” References: llm d Flow Control architecture llm d Queue and Prioritize Requests vLLM scheduling policy reference vLLM serving CLI scheduling controls

How do you prevent head-of-line blocking when LLM requests have widely different prompt and output lengths?

Head of line (HOL) blocking occurs when a long or resource hungry LLM request at the front of a queue prevents shorter requests behind it from making progress, even if the shorter work could fit into available compute or scheduling opportunities. In production inference this can happen with long prompt prefills , long decodes occupying scarce KV/concurrency slots, and requests pinned to a saturated worker. The goal is not simply to prefer short requests always . Use bounded per step work, workload aware routing, fairness, admission control, and real service level objectives (SLOs) so short interactive requests remain responsive without starving long jobs. 1. Why LLM scheduling differs from ordinary FIFO queues A 500 token FAQ prompt and a 40,000 token document prompt are not equal cost tasks. Prefill processes input tokens; decode emits output tokens step by step. A long prefill can monopolize a scheduler's token budget for multiple steps. A long output generation might occupy an active sequence slot and GPU KV memory for seconds or minutes; a short response behind it may have to wait when concurrency or KV capacity is exhausted. Basic first come, first served (FCFS) ordering considers arrival time, not predicted work or deadline. 2. Beginner friendly numerical example Assume one engine step can schedule 1,024 prefill tokens and a queue contains: Request A: a long 8,192 token prompt, arriving first. Request B: a short 128 token prompt, arriving slightly later. FIFO with unrestricted long prefill chunks: 1. A may take the entire 1,024 token budget per step. 2. A needs approximately 8 such steps before B can obtain prefill capacity. 3. B waits through those steps even though its own prompt takes just 128 tokens. With a 512 token cap on A per step: 1. A receives 512 tokens , leaving 512 of the step budget. 2. B's 128 tokens can now be scheduled in the same step, subject to KV availability and actual scheduler rules. 3. The remaining 384 tokens may be used for other eligible work. 4. B potentially begins decoding much sooner, while A now needs at least 16 steps to finish its prefill if it is limited to 512 tokens each step. This is a conceptual scheduling budget illustration, not a prediction of real elapsed time . Kernel launches, existing decode tokens, batching, KV memory and scheduling policy affect actual progress. Long requests can suffer higher TTFT as a result. 3. Chunked prefill is necessary but not always sufficient Question 184 introduced chunked prefill : split a long prompt over multiple scheduling iterations. Here is the production nuance: Chunking alone doesn't necessarily prevent a long request from taking the full budget every step when it sits ahead of other work. vLLM describes long prefill token threshold to limit how many tokens a long request may consume per step , making it possible for short prefills to enter. Its long prefill token threshold adaptive option can floor the effective cap at a fair share token budget under contention in supported versions. max num batched tokens controls the scheduler's step level token budget. It must be tuned jointly with prefill caps, live decoding and available KV cache. The specific flag behavior and support depend on the installed vLLM version . Test against that version rather than assuming any threshold is universally optimal. 4. Other ways to avoid HOL blocking A. Short workload aware queues Separate interactive short request traffic from long document processing or offline batch jobs. Apply fair scheduling, deadline aware ordering, and carefully bounded priority so one class cannot dominate indefinitely. Use predicted uncached prompt and output token work as imperfect cost estimates , not guarantees. B. Workload aware routing Avoid assigning short requests to a worker with a heavy long prefill queue when another compatible replica has capacity. Combine queue depth, expected prefill work, decode/KV occupancy and prefix cache locality. Keep routing decisions within tenant/model compatibility boundaries. C. Decode protection Preserve decode scheduling opportunities when newly admitted prefills are heavy. vLLM documents a configurable prefill schedule interval for concentrating prefill work on some steps while leaving others to decode; availability and measured effect vary by version and workload. Prefill/decode disaggregation can reduce interference between phases, but adds KV transfer cost and does not guarantee better throughput. D. Capacity and backpressure Bound waiting requests and queued prompt tokens; reject promptly when a deadline cannot be met. Avoid letting long generations consume all active sequences or KV cache capacity. Where supported, protect space for short interactive work while allowing long requests to make forward progress. 5. An important fairness trade off Shortest job first can improve average response time when work estimates are accurate, but continuously arriving short prompts can starve long prompts. Prefer a policy that also considers: Aging or maximum waiting time limits for long work. Weighted tenant fairness and workload priority agreements. Deadline feasibility instead of blindly serving whichever prompt is shortest. Completion and goodput , not only short query TTFT. A cap that is too small may fragment long prefill work, increase scheduler overhead and worsen long request TTFT. A cap that is too large can preserve HOL blocking. Find the threshold using representative mixed traffic. 6. Production incident: long documents block chat Incident: An AI assistant normally serves 128–512 token prompts, but after enabling a 32k token document analysis workflow its p95 TTFT climbs from 400 ms to 2.4 s , even as GPU usage remains high. Root cause investigation: 1. Break down TTFT into queueing, prefill and first decode time by prompt length bucket . 2. Inspect per step token budgets: are a few long prefills consuming most scheduled tokens? 3. Track waiting short requests, per tenant queues, decode interruptions, KV occupancy and preemptions. 4. Verify router placement: are long prompts concentrated on one model worker? Fix: Cap long prefill tokens per step at a measured value. Preserve some scheduling opportunities for shorter prompts and ongoing decodes. Route or isolate document jobs when contention remains high. Keep admission control and tenant fairness in place; roll out gradually. Prevention: Load test short prompt bursts concurrent with long prefill and long output requests. Alert on p95 TTFT by prompt length bucket and oldest waiting request, not just fleet wide average latency. 7. How to measure success Compare FCFS , chunked prefill without per request caps , capped long prefill , and workload aware routing/queues with the same model, GPU count and offered traffic: p50/p95/p99 TTFT for short and long prompts separately. Long request completion time and starvation rate . p95 inter token latency for active decodes. Queue wait distributions, scheduled tokens per request/step, KV occupancy and preemptions. Total token throughput and SLO compliant goodput . Fairness and total GPU cost under sustained mixed load. A successful design reduces short request tail latency without unbounded delays for long work . Interview ready answer: “HOL blocking occurs when a long prefill or generation monopolizes a scheduling token budget, queue, or KV capacity while short requests wait. I'd use chunked prefill with a per step cap , workload aware routing, tenant/deadline aware queues and decoder headroom. I'd avoid pure shortest job first starvation, and validate p95 TTFT by request length, decode ITL, long job completion and SLO compliant goodput.” References: vLLM x AgentX: Optimizing for Real World Agentic Serving (2026) vLLM scheduling / run batch CLI reference llm d flow control architecture llm d request queuing and prioritization

How do request deadlines, cancellation, and timeout propagation reduce wasted GPU work in production LLM inference?

Request deadlines, cancellation, and timeout propagation stop an LLM system from spending GPU resources on results that users can no longer receive or use. A deadline is the latest acceptable completion time; a timeout is a configured wait limit for an operation; cancellation is a signal to cease work. Correct systems propagate the remaining time budget and cancellation to routers, inference queues, model workers, and any retrieval or tool calls. 1. Why this matters in LLM serving A long prompt prefill and hundreds of output tokens can continue consuming GPU compute and KV cache memory even after a browser closes or the API gateway has timed out. Unbounded queued requests waste scheduling capacity when they are already too late to meet the client's service level objective (SLO). Timeouts at the edge do not automatically guarantee GPU work stops . The application and inference backend must observe cancellation and release scheduled work. On a streaming response, client disconnects and request cancellations can occur after some output has been sent ; cancellation then stops further work rather than undoing tokens or external actions already produced. 2. Beginner friendly numerical example A chatbot request has a 5 second total deadline : 1. Gateway/authentication takes 0.2 seconds . 2. Retrieval takes 0.8 seconds . 3. Router/queue takes 1.5 seconds . 4. Remaining model time budget = 5 − 0.2 − 0.8 − 1.5 = 2.5 seconds . If the gateway gives the model worker a fresh 5 second timeout instead of 2.5 seconds , it can run past the original user deadline. Propagating the absolute deadline or remaining budget prevents this. Wasted token illustration: Assume 30 users abandon streaming requests and, absent cancellation, each worker would generate 200 unnecessary tokens at a measured 50 tokens/second . Wasted work ≈ 6,000 generated tokens and 120 seconds of aggregate sequence generation time . This is not necessarily 120 seconds of wall clock GPU time: inference batches run concurrently and costs vary. Effective cancellation can avoid much of that work, though some kernels or in flight operations may complete before the signal is observed. 3. End to end lifecycle 1. At ingress: set a trusted end to end deadline based on the product SLO and enforce per tenant policies. In distributed systems, propagate an absolute timestamp with clock skew care or a monotonic remaining budget. 2. Before queueing: reject or terminate requests whose remaining budget cannot plausibly cover expected waiting, prefill, transfer and decode. 3. At every hop: pass a cancellation context/token to retrieval, prompt construction, router, model endpoint and tools. The next hop must not reset the original deadline. 4. While queued: remove expired/cancelled requests promptly, without letting them consume prefill or KV allocations. 5. While running: signal the serving engine to stop future decode iterations and free active KV blocks and scheduling slots when safe. 6. On disconnect: stop the specific generation unless it is deliberately serving other legitimate waiters; prevent a cancelled subscriber from terminating a shared underlying computation still needed by others. 7. On completion/error: clean up request state exactly once and record the terminal reason (success, timeout, cancellation, rejection, error). 4. Cancellation is cooperative—not instantaneous gRPC documents that cancellation should propagate through the computation tree, but some server handlers must periodically check for cancellation . A notification does not forcibly interrupt arbitrary application code. NVIDIA Triton checks cancellation at selected scheduler and ensemble points; whether already running work ends early depends on the model backend . Its documentation lists cancellation support for TensorRT LLM and vLLM backends, while some cases need backend level logic. For custom async generators, ensure the generator is closed and the worker side request is aborted. Merely closing the frontend HTTP connection is insufficient unless the deployed gateway/engine forwards cancellation. In flight GPU kernels and transfers can be non preemptible at fine granularity; measure cancellation detection and cleanup delay. 5. Production patterns A. Deadline aware admission: Estimate queue + prefill + expected decode against the remaining request budget. Reject rapidly rather than admitting hopeless work. B. Streaming cancellation: For SSE/gRPC streams, detect disconnects and propagate a cancel event; stop generating further tokens and release KV state. C. Timeout budgets: If a request's total budget is 5 seconds , reserve a small response/cleanup margin and give retrieval, routing, and inference only their remaining portions. Avoid nested timeout resets. D. Cancel safe retries: Retries consume the same original deadline; use jitter and a strict retry budget. Never blindly repeat side effecting tool actions whose outcome may already have committed. E. Shared execution: If 10 callers are coalesced onto one generation, cancel the shared generation only when no valid waiters remain or its independent work policy expires. F. Graceful shutdown: During worker draining, stop admitting new work, give active requests bounded time to finish, and cancel or transfer the rest according to serving support. Clean up KV buffers and request metadata. 6. Incident, diagnosis, and fix Incident: A customer dashboard times out after 3 seconds , but p95 GPU queue depth and memory occupancy keep rising. The model engine continues decoding responses long after users have disconnected. Root cause: The gateway has a 3 second HTTP timeout, but the model RPC uses a separate 30 second timeout and never receives cancellation; expired requests retain KV blocks and scheduling slots. Fix: Propagate one end to end deadline, abort queued or running inference when it expires, wire the disconnect signal through the gateway, and verify KV/sequence cleanup. Add deadline aware queue admission and suppress retries after expiration. Prevention: Simulate disconnects during queueing, prefill, KV transfer and decode. Verify work stops within a tested grace period, queued/active counts fall, and unrelated users' p95 TTFT improves. 7. Observability and validation Track: Cancelled/expired requests , by hop and reason; proportion of cancellations that actually reached the model worker. Detection and teardown delay : time from client cancellation to scheduler removal and KV release. Wasted output tokens and estimated compute after client deadline; exact attribution may need tracing. Deadline misses , p50/p95 queue wait, TTFT, ITL and end to end latency. KV occupancy , active/queued sequences, and work still active after the final subscriber cancels. Retry amplification and incorrectly repeated tool actions. Interview ready answer: “I’d use one end to end deadline, pass the remaining time budget through retrieval and routing, reject queued work that cannot finish in time, and propagate client disconnect cancellation into the LLM engine. Cancellation must actually stop later decode steps and release KV slots; a gateway timeout alone does not. I'd measure orphaned generations, wasted tokens, cleanup delay and p95 latency under disconnect load.” Research references: gRPC — Cancellation gRPC — Deadlines NVIDIA Triton — Request Cancellation NVIDIA Triton gRPC client timeout reference

How do retry storms and circuit breakers affect the reliability of production LLM inference?

Retry storms happen when many clients retry failed or slow LLM requests simultaneously, making an already overloaded inference backend even busier. Circuit breakers prevent repeated calls to an unhealthy dependency until there is evidence it has recovered. Together with bounded retries, jitter, deadline propagation, and admission control, they limit cascading failures and wasted GPU work. 1. Why retry storms are especially expensive for LLMs A retried request may repeat costly prompt prefill , allocate KV cache blocks, and generate a long response. Even requests whose original callers have timed out can continue consuming memory and compute unless cancellation propagates. Retries may multiply across API client, gateway, agent orchestrator, and model provider SDK. A seemingly small failure rate can become much larger downstream load. The correct response to an overloaded model endpoint is often to shed work or route to genuinely available capacity , not immediately generate more traffic. 2. Beginner friendly numerical example: retry amplification Assume 100 incoming requests/second reach a model fleet that can safely complete 100 requests/second . 1. A temporary problem causes 20 requests/second to fail or time out. 2. If each failed request is retried twice, these failures may add up to 40 extra attempts/second , yielding 140 attempts/second total in a simplified sustained snapshot. 3. The added traffic exceeds the fleet's capacity by 40% and can create longer queues, more timeouts and more retries , producing positive feedback. 4. Now imagine three independent layers each permit up to two retries : one logical request could generate 3 × 3 × 3 = 27 model attempts in a pathological nested sequence. This is a worst case amplification , not an expected average. 5. If each attempt consumes a hypothetical 3,000 input tokens , those 27 attempts may repeatedly process up to 81,000 input token equivalents before any valid answer is returned; prefix reuse may mitigate some prefill but not all processing or output work. Engineering lesson: coordinate a single retry owner and budget rather than letting every layer retry independently. 3. Retry safely only when the failure may recover Potentially retryable: transient network resets, selected 502/503/504 responses, or a 429 where the service explicitly offers a valid retry delay and the request can still meet its deadline. Usually not retryable without input changes: authentication failures, malformed requests, unsupported models, excessive context length, or invalid tool arguments. Not automatically safe: a read like LLM completion may be repeated, but it can produce a different stochastic answer ; tool execution agents can duplicate emails, payments or database writes. Use idempotency keys and operation status checks for side effects. Retries must share the original end to end deadline (Question 253), be bounded in count, and stop if no meaningful time remains. 4. Exponential backoff with jitter A retry strategy may use a capped exponential delay: cap k = min(max delay, base delay × 2^k) Then draw a random wait uniformly between 0 and cap k ( full jitter ), unless a valid server retry hint dictates a longer wait. Numerical illustration: with base delay = 0.5 s and max delay = 8 s, consecutive retry delay caps are 0.5, 1, 2, 4, 8 seconds . Full jitter spreads each retry across its interval, rather than synchronizing thousands of callers at a common boundary. Set a global retry budget (for example, a small fraction of the original request rate) and a maximum number of attempts. Prefer retrying at one well defined application layer while downstream SDK retries are disabled or accounted for. Respect provider quotas and Retry After where applicable. On sustained overload, backoff is not a substitute for admission control . 5. Circuit breakers: closed, open, half open A circuit breaker tracks failures for a specific dependency or endpoint. 1. Closed: normal requests flow through. Record eligible failures and latency. 2. Open: a rolling failure rate or overload threshold is reached. Fail fast or use a safe, authorized, capacity available alternative , without continuing to hammer the unhealthy endpoint. 3. Half open: after a cooldown, allow a small bounded number of probe requests; if service health is restored, close the breaker, otherwise reopen it. Hypothetical policy: open if at least 20 attempts occur in a recent evaluation window and more than 50% encounter eligible failures. After 30 seconds , permit two probes. These are teaching values; select thresholds from measured traffic, avoid low sample false positives, and account for startup/recovery behavior. Important distinctions: A circuit breaker detects repeated dependency failure and interrupts calls to that dependency. A rate limiter regulates how much work is allowed. A bulkhead isolates pools/tenants so a failing workload cannot exhaust every worker. Backpressure reduces demand when resources are saturated. These mechanisms complement rather than replace one another. 6. LLM specific routing and fallback pitfalls Breakers should generally be per model deployment, region or endpoint , not one global switch that disables all healthy replicas. Failover must consider compatible model, prompt, tokenizer/context window, policy and tenant restrictions . A cheaper or smaller fallback model may not meet answer quality requirements. Never silently route sensitive workloads to a provider or region without appropriate permissions. A secondary endpoint needs spare capacity . Blind failover can overload the backup and trigger another cascading failure. If an output stream has already delivered tokens, transparently restarting with a different model can duplicate or contradict partial output; treat partially completed streams and tool side effects explicitly. When a circuit is open, return a clear temporary error or safe cached answer only if validity, privacy and freshness rules permit it . 7. Production incident: overload amplified by retries Incident: An inference replica becomes slow during a peak traffic burst. Client timeouts increase, and automatic SDK plus gateway retries triple attempt volume. The model fleet's p95 TTFT rises sharply, queues grow, and KV cache occupancy remains high even after some callers disconnect. Root cause investigation: 1. Compare original logical requests with actual model attempts ; compute retry amplification. 2. Trace which layer initiated each retry and whether its previous attempt was cancelled. 3. Inspect 429/503/timeout rates, queue depth, KV occupancy, worker health and latency by endpoint. 4. Verify whether a secondary route is also overloaded. Fix: Centralize retry policy with bounded attempts, full jitter and strict deadline propagation. Open a per endpoint circuit when failures indicate sustained unavailability. Reject early rather than queue indefinitely; cancel orphaned generations. Route to healthy capacity only within supported compatibility and capacity limits . Restore gradually via controlled half open probes and admission ramps. Prevention: Run chaos/load tests combining replica failures, high concurrency requests, partial network partitions and retrying clients. Verify no excessive downstream amplification and that recovery is stable. 8. Production metrics and guardrails Retry amplification = backend model attempts / original logical requests. Retry success and wasted attempts: attempts that recover a request versus those that only increase load. Circuit state and transitions: open/half open time, probe results, false trips and recovery time. Latency and service outcomes: p95/p99 TTFT, ITL, final success rate, deadline misses, error rates and SLO compliant goodput. Capacity: active/queued requests, KV cache pressure, preemptions, GPU utilization and spare fallback capacity. Safety: duplicate tool side effects, wrong model fallbacks, tenant isolation and unnecessary provider charges. Interview ready answer: “Retry storms amplify an overloaded LLM endpoint by repeating expensive prefill and decode work. I'd allow only bounded, deadline aware retries for transient failures, use exponential backoff with jitter, and coordinate retries at one layer. A per endpoint circuit breaker fails fast when a deployment is unhealthy, then tests recovery with limited half open probes. I'd measure model attempt amplification, GPU/KV pressure, tail latency and final user success, while ensuring failover and tool actions remain safe.” References: Google SRE — Addressing Cascading Failures AWS — Timeouts, Retries, and Backoff with Jitter Microsoft Azure — Circuit Breaker Pattern AWS Well Architected — Control and Limit Retry Calls

How do you design health-aware failover and graceful degradation for production LLM inference?

Health aware failover sends an LLM request to a different, compatible serving endpoint when the intended endpoint is unhealthy or cannot meet its latency objective. Graceful degradation deliberately offers a reduced but safe service—such as a smaller approved model, shorter output, or delayed batch processing—when full quality inference is unavailable. Neither automatically improves reliability: fallbacks can overload the remaining capacity, change answer quality, violate tenant boundaries, or re execute tool actions. 1. Failover versus graceful degradation Failover: move from replica A to compatible replica B running the same model or equivalent deployment. This should preserve the intended task as closely as possible; cache locality and latency may change. Graceful degradation: use a different, explicitly approved level of service, e.g. a simpler summarization model or a queued asynchronous workflow. This is a product level trade off, not an invisible one to one replacement. Circuit breakers (Question 254) can exclude persistently unhealthy endpoints; health aware routing decides where an eligible request should go next . A healthy endpoint must be checked for actual spare capacity , not just a responsive health check URL. 2. A robust request routing path 1. Classify the request: tenant permissions, region/data residency, model/adapter, tools, context length, modality, safety policy and end to end deadline. 2. Discover live endpoints and exclude incompatible or unhealthy instances (failed readiness probes, recent high error rates, unhealthy circuit state). 3. Estimate capacity: active sequences, queued prompt token work, KV occupancy, decode backlog and observed TTFT/ITL. 4. Prefer compatible, loaded but healthy capacity while considering prefix cache locality. Do not route onto a replica that cannot fit the new request. 5. Attempt a bounded failover only if the request is safe to replay, the original attempt is cancelled or accounted for, and enough deadline remains. 6. If no suitable replica exists, apply an explicitly approved degradation policy or return a prompt, honest 429/503 style error instead of silently changing semantics. llm d's router provides an endpoint picker with health/load/cache aware filters and scoring. Its precise deployed behavior depends on router version and configuration. 3. Beginner friendly capacity example A service normally receives 120 requests/second across three replicas, each able to sustain 50 requests/second at its SLO. Total capacity = 3 × 50 = 150 requests/s ; planned headroom = 30 requests/s . Replica A fails, leaving 2 × 50 = 100 requests/s capacity. Offered load is still 120 requests/s , so attempting to fail over every request creates a 20 requests/s shortfall (20% above surviving capacity). A safe controller can admit up to about 100 requests/s under these simplified assumptions and shed/defer at least 20 requests/s , or use preprovisioned capacity. Suppose 30 requests/s are noncritical summarizations and the business explicitly allows delayed batch handling. Deferring 20 of them leaves 100 requests/s for the surviving replicas without changing interactive request semantics. Important: These rates are illustrative, not measured GPU benchmarks. Real LLM workload capacity depends on input/output token lengths, concurrency, KV memory, prefix hits, and tail latency targets; admission should be token/work aware. 4. Choose a safe degradation ladder For each workload, declare in advance which changes are allowed: Tier 0 — equivalent failover: same approved model/version and required capabilities on another healthy replica, region or provider within policy. Tier 1 — operational simplification: reduce maximum output length, unnecessary tool passes or optional context only when product requirements allow it . Never silently truncate safety critical inputs. Tier 2 — model downgrade: route an eligible noncritical task to a smaller/cheaper model with separately tested quality thresholds, schema support, tool compatibility and disclosure where important. Tier 3 — delayed processing: queue background jobs with durable status and idempotency, when the task does not need an immediate answer. Tier 4 — fail closed: for prohibited cross region processing, regulated or safety critical workflows, unavailable required tools, or unverified correctness, return a clear temporary failure. Never interpret “graceful” as permission to fabricate answers or silently reuse stale, personalized cache entries. 5. Timeouts, streaming and side effects Share the original request deadline. A failover attempt must use the time left; it must not reset the user's timeout. Avoid retry amplification. Set one coordinated attempt budget across gateway, SDK and router. Streaming partial output: once a client has received tokens, restarting with another model may duplicate or contradict earlier text. Prefer an explicit interruption/error unless the application has a designed resume protocol. Tools and agents: a failed response does not prove a previous email, payment, database write or tool action failed. Use operation IDs, idempotency and reconciliation before retrying. KV state: in flight generation cannot simply move between arbitrary replicas; transfer or resume requires compatible infrastructure. Otherwise a safe new attempt recomputes the prompt. 6. Health signal design Liveness: is the process running? This is not sufficient for traffic admission. Readiness: can it accept requests with the correct model/version loaded? Performance health: rolling error and timeout rates, queue delay, TTFT, KV/cache pressure, preemptions and concurrency. Isolation: classify by model, replica, region or provider so one outage does not open a global breaker against healthy endpoints. Use smoothed metrics, hysteresis and a bounded recovery ramp to avoid oscillation and flapping. 7. Production incident Incident: A four replica endpoint loses one worker. The router moves every failed request to the other three, but they were already near their safe token work budget. Queueing drives p95 TTFT from 700 ms to 5 seconds ; clients retry, creating a cascading outage. Root cause: failover assumed endpoint availability meant spare serving capacity; it ignored request work, KV pressure and retry budgets. Fix: isolate the failed endpoint, route only within compatible healthy capacity, shed/slow excess work, cancel abandoned attempts, and degrade only explicitly eligible background tasks. Restore capacity gradually. Prevent recurrence: chaos test replica and regional failures during peak traffic, including mixed prompt lengths, streaming disconnects, and client retries. 8. What to monitor Original requests versus backend attempts; failover frequency, chosen endpoint and remaining deadline. p95/p99 TTFT, ITL, end to end latency and successful requests meeting SLOs. Per replica prompt/output tokens, waiting work, KV occupancy, preemptions and capacity headroom. Degradation tier distribution, quality failures, incorrect model routing and tenant/region policy violations. Recovery ramp, flapping endpoints, cancellation effectiveness and duplicate side effects. Interview ready answer: “I would filter endpoints by model and security compatibility, then use live readiness, error rate, queues and KV capacity to choose a healthy target. Failover must share the original deadline and retry budget, and only use spare capacity. If capacity is insufficient, I would apply an explicitly approved degradation ladder—batch deferral or a tested simpler model for eligible tasks—or fail fast. I would chaos test the rarely used paths because untested fallbacks can make outages worse.” References: llm d Router Architecture vLLM llm d Integration AWS Builders' Library — Avoiding Fallback in Distributed Systems Hugging Face Chat UI — LLM Router fallback routing

How do you use SLO-aware routing and latency prediction to choose an LLM inference replica?

SLO aware routing selects a healthy, compatible model replica based on whether it is predicted to satisfy the request's time to first token (TTFT) and time per output token (TPOT) objectives—not simply which GPU has the fewest requests. An online latency predictor estimates the delays on each candidate from prompt size, prefix cache reuse, queueing, active decode load, and KV memory pressure. The router favors endpoints with positive SLO headroom and avoids wasting time on a replica likely to miss the deadline. Unlike Question 241 (which introduced cache plus load aware routing) and Question 255 (health aware failover), this lesson focuses on predictive latency models, per request SLO headroom, uncertainty, routing decision overhead and validating predicted versus actual SLO attainment . 1. Why naive balancing falls short Round robin ignores expensive prompts and uneven active generations. Fewest active requests treats a 100 token prompt like a 20,000 token prompt. Prefix affinity alone can send a request to an overloaded replica just because the replica has reusable KV. Current GPU utilization is a lagging and incomplete proxy for TTFT/TPOT; a busy GPU can still meet SLOs, while a queued workload on another may not. A predictor can use richer request/server signals to estimate what the next request will experience on each candidate endpoint , including a hypothetical placement. 2. Inputs to a latency predictor For each eligible replica collect: 1. Input prompt tokens and an estimated output token length or current generated token count where relevant. 2. Prefix cache match fraction and uncached prefill tokens . 3. Queue depth and input tokens in flight , not only queued request count. 4. Running sequences , GPU concurrency, and remaining decode work if estimable. 5. KV cache utilization , evictions/preemptions and memory headroom. 6. Hardware/model/adapter compatibility; network/KV transfer delay for disaggregated deployments. llm d documents an online trained predictor using recent request level observations, with TTFT features including input length, queue depth, active requests, KV use and prefix match. Its TPOT features also include generated tokens. Current implementations can train an XGBoost based model online. The prediction's accuracy depends on telemetry quality, traffic coverage and how recently it was updated. 3. Beginner friendly numerical routing example Suppose the application requires TTFT ≤ 800 ms and TPOT ≤ 50 ms per output token . The predictor evaluates three compatible endpoints: Replica A: predicted TTFT 500 ms , TPOT 65 ms . It has TTFT headroom 300 ms but violates the TPOT target by 15 ms . Replica B: predicted TTFT 650 ms , TPOT 35 ms . Headroom is 150 ms for TTFT and 15 ms for TPOT . Replica C: predicted TTFT 900 ms , TPOT 25 ms . It misses TTFT by 100 ms , despite excellent predicted TPOT. Decision: choose B if the product requires both targets. A greedy minimum TTFT router might choose A and harm token streaming; a greedy minimum TPOT router might choose C and delay the first token. Important: These are illustrative point predictions , not guarantees. A robust production policy should include predictor error margins and consider p95 probabilities or a measured confidence budget. For example, a 150 ms predicted TTFT headroom does not protect the SLO if recent forecast errors frequently exceed 200 ms. 4. One workable routing algorithm 1. Authorize and filter: require matching model/version, adapter, context, tools, region, security boundary, and readiness. 2. Classify SLOs: determine tenant/workload specific TTFT/TPOT objectives from trusted policy, with sensible defaults. 3. Predict both metrics for each compatible candidate. 4. Calculate headroom: H ttft = SLO ttft − predicted TTFT; H tpot = SLO tpot − predicted TPOT. 5. Prefer candidates with nonnegative headroom for both. Rank with headroom, predicted violation probability and resource efficiency rather than solely one latency number. 6. If all candidates violate an SLO , use a documented fallback policy: least bad available candidate when useful, queue/defer within deadline, or reject quickly; do not represent the target as guaranteed. 7. Record actual outcomes and compare predictions against measured TTFT/TPOT to update calibration. llm d's latency scoring and headroom filters follow this broad design. The specific score is implementation dependent; not every router optimizes a simple sum of the two metrics . 5. Worked production incident Incident: Interactive chat p95 TTFT increases from 650 ms to 1.8 seconds after nightly long context summarization starts. Round robin spreads requests evenly by count , but one worker handles a high share of 16k token prefills and another holds many long decode sequences. Diagnosis: Split observed TTFT/TPOT by replica, prompt length, prefix match rate, queue delay and KV occupancy. Compare predicted versus actual latencies to see whether the model captures long prompts and decode saturation. Check that routing did not concentrate requests on a cache affine but already busy replica. Fix: Route based on predicted TTFT+TPOT feasibility, incorporate token workload/KV telemetry, keep short interactive requests within a suitable SLO tier, and admit only when capacity makes the target plausible. Preserve an ordinary load aware fallback while prediction is unavailable or cold. Prevention: Test changing prompt/output distributions, replica restarts, sudden batch bursts, prediction model cold starts and mixed tenants. Recalibrate as hardware, inference engine or attention kernels change. 6. Trade offs and failure modes Cold start model: early predictions may be uninformative until real traffic provides training data; use load aware routing until enough evidence accumulates. Distribution drift: new prompt lengths, model releases, hardware or batch patterns can break learned latency relationships. Prediction overhead: scoring every candidate adds CPU work and network calls. With R requests/s and N eligible replicas, prediction demand can be roughly R × N per request candidate predictions. For 200 RPS and 8 replicas , that is 1,600 candidate predictions/s before batching/caching and other costs. Tail risks: an accurate average model can still underestimate rare queue spikes. Track p95/p99 target attainment and conservative uncertainty margins. Feedback loops: routing changes endpoint load, thereby changing future latency. Monitor oscillation and add hysteresis or bounded score updates. Goodput versus throughput: maximize responses meeting both SLOs , not just generated tokens/s. More replicas may be expensive; route and autoscale together. 7. What to monitor Forecast error: p50/p95 absolute TTFT/TPOT error, calibration by request length, tenant and replica. User SLOs: actual p50/p95/p99 TTFT and TPOT, violation rate for each objective and percentage meeting both. Router overhead: prediction latency, routing decision time, predictor availability and fallback frequency. Worker state: queued tokens, active sequences, KV occupancy, prefix cache hits, preemptions and per replica load. Business efficiency: SLO compliant goodput, cost per successful response and any tenant fairness regression. Interview ready answer: “I would filter endpoints for model and tenant compatibility, predict TTFT and TPOT on every candidate from token counts, prefix reuse, queue state and KV load, and prefer replicas with positive headroom against both SLOs. I would audit predicted versus observed latency, limit prediction overhead and fall back to load aware routing when the predictor is stale. The goal is goodput meeting real user SLOs—not merely balanced request counts.” References: llm d — Latency Predictor llm d — Predicted Latency Based Scheduling for LLMs Google Cloud — Predicted latency based routing with GKE Inference Gateway llm d latency scorer

How do you calculate and optimize SLO-compliant goodput for production LLM inference?

SLO compliant goodput measures how many LLM requests finish successfully and meet every required service level objective (SLO) per second. Unlike raw request throughput, it excludes requests that finish too slowly to be useful. It therefore captures a central production trade off: adding concurrency or larger batches can increase raw token throughput while decreasing user visible success within latency limits . 1. Throughput, latency and goodput Request throughput: completed requests per second, regardless of whether they met latency targets. Output token throughput: generated output tokens per second. This metric can increase even when users experience more waiting. TTFT (time to first token): elapsed time from request submission to the first generated token, including any network/queueing delay included by the benchmark. TPOT (time per output token) or mean inter token latency: average time between generated tokens after the first. Benchmark tools differ in edge case treatment, so compare like with like. Goodput: number of successfully completed SLO conforming requests per second. A typical request is conforming only if both TTFT and TPOT limits hold; an optional end to end deadline can also be enforced. Formula: goodput (requests/s) = count(successful completed requests meeting every configured SLO) / observation duration (seconds) A request with TTFT under its target but TPOT above its target does not count , even though the response was technically completed. Incomplete/errored requests should not be counted as successful goodput; explicitly document the benchmark's population, window and missing metric handling. 2. Beginner friendly numerical example Assume a load test runs for 60 seconds with these SLOs: TTFT must be ≤ 1,000 ms . TPOT must be ≤ 50 ms/token . Both conditions must hold for a successful request to count. Configuration A — moderate concurrency 1. 600 requests complete successfully in 60 seconds. 2. 540 meet both TTFT and TPOT SLOs. 3. Raw throughput = 600 / 60 = 10 requests/s. 4. Goodput = 540 / 60 = 9 requests/s. 5. SLO attainment among completed requests = 540 / 600 × 100 = 90%. Configuration B — aggressive batching / concurrency 1. 900 requests complete in the same 60 seconds. 2. Only 360 meet both targets. 3. Raw throughput = 900 / 60 = 15 requests/s — a 50% raw throughput increase . 4. Goodput = 360 / 60 = 6 requests/s — a 33.3% goodput decrease versus A. 5. SLO attainment = 360 / 900 × 100 = 40%. Decision: If serving interactive users subject to these SLOs, A is preferable : it delivers 9 useful requests/s versus 6 , despite its lower raw throughput. These are invented educational measurements, not benchmark results from a particular model. 3. The throughput–goodput knee When offered load is low, most requests satisfy their latency objectives. Increasing load can raise both throughput and goodput. As load approaches the GPU's saturation knee : Queueing increases TTFT . Excessive active decode sequences and KV cache pressure can worsen TPOT . More requests cross one or both SLO thresholds. Raw throughput may plateau or even rise, while goodput peaks and then falls . The operational sweet spot is often before maximum raw throughput . Do not size the production service solely at the largest batch size that fits in GPU memory. 4. Build a repeatable benchmark 1. Define the SLOs: e.g., interactive chat requires TTFT ≤ 1 s, TPOT ≤ 50 ms; a long document job may have different targets. Specify whether success, deadline, and correctness conditions also apply. 2. Choose realistic traffic: fixed model/tokenizer/hardware, representative prompt and output length distributions, prefix cache hit rates, tenants and streaming settings. 3. Sweep offered load: test increasing arrival rates and concurrency, including bursts. Use a sufficiently long steady state observation window with warmup handled explicitly. 4. Record each request: arrival, first token and last token times; output token count; queue wait; success/error/cancellation; and any SLO violations. 5. Compute joint conformity per request: count a completed successful request only when every configured objective passes. Do not infer joint goodput by multiplying separate TTFT and TPOT pass percentages without evidence that they are independent. 6. Select the operating point: choose the highest sustainable goodput with sufficient safety margin, acceptable tail latencies, and stable error/rejection rates. 7. Repeat and validate: account for load generator bottlenecks, variance across runs and differences between open loop and closed loop testing. 5. Practical code style calculation For each completed request r, using measured TTFT and TPOT in milliseconds: passes(r) = r.success && r.TTFT ms <= 1000 && r.TPOT ms <= 50 goodput = sum(passes(r) for r in completed requests) / elapsed seconds If TPOT is undefined for a one token response, define a deliberate policy (e.g. evaluate only TTFT or exclude from the TPOT constrained population) rather than silently treating an unavailable measurement as a pass. GuideLLM defines an undetermined SLO result where a required measurement is missing. 6. How to optimize goodput Tune concurrency and batching: sweep max num seqs and batched token budgets; stop when SLO misses increase faster than throughput gains. Separate prefill/decode contention: chunked prefill, fair scheduling, or prefill/decode disaggregation may protect both latency phases, subject to hardware and KV transfer costs. Improve routing: choose compatible replicas using queue length, estimated prefill work, KV occupancy, prefix reuse and predicted TTFT/TPOT headroom. Reuse computation: exact prefix caching can reduce repeated prefill work, but cache hit patterns must be representative and tenant safe. Enforce admission/backpressure: when saturation is unavoidable, reject or defer excess work promptly rather than allowing all requests to time out. Rejections reduce service availability , so count them in overall user facing success rate SLOs even if a completed request only goodput metric omits them. Cancel abandoned requests: release GPU/KV resources for requests whose client disconnected or deadline expired. Autoscale the bottleneck: scale the right worker pool, considering GPU memory, KV transfer bandwidth and cost; more replicas alone may not fix uneven routing. 7. Production incident Incident: An engineer doubles the LLM batch/concurrency limit to improve token throughput. GPU utilization rises and output token throughput grows 25% , but interactive p95 TTFT jumps from 0.7 s to 2.2 s and user timeouts increase. Root cause: More queued prefills and active decode sequences create contention. Raw tokens/s hides the percentage of requests meeting latency objectives. Diagnosis: Plot offered RPS against raw throughput, joint TTFT+TPOT goodput , p95/p99 TTFT, TPOT, KV occupancy, scheduler preemptions and rejection rate. Fix: Roll back to the goodput maximizing concurrency range; tune prefill token budgets; protect decode scheduling, introduce bounded admission, and route requests based on SLO headroom. Prevention: Gate deployment changes on SLO compliant goodput at representative traffic , not on GPU utilization or peak tokens/s alone. 8. Accuracy, fairness and cost caveats Pure latency goodput does not measure answer correctness. If the product requires valid JSON, reliable tool calls or factual accuracy, treat these as additional eligibility checks or independent quality metrics. A system can artificially raise a benchmark's completed request only goodput by refusing difficult requests. Monitor acceptance, successful completion, user facing availability and rejected workload alongside goodput. Report results by tenant, request length bucket and workload class; an aggregate gain can mask a severe regression for long prompts or lower priority customers. Cost efficiency: SLO compliant requests / GPU hour or cost / SLO compliant request can guide economic optimization, but compare equivalent response quality and workload mix. 9. Research and tooling The DistServe paper formalized serving capacity optimization under separate TTFT and TPOT objectives, motivating phase specific resource allocation. GuideLLM explicitly defines goodput with per request TTFT, TPOT and optional end to end constraints; all configured objectives must pass. NVIDIA's GenAI Perf documents latency and throughput metrics and configurable goodput constraints; its documentation now directs new benchmarking work to AIPerf , so confirm current tool support and metric definitions before choosing a benchmark. Interview ready answer: “Goodput is the number of successful requests per second that satisfy all configured latency SLOs, typically TTFT and TPOT. I would benchmark representative traffic across concurrency levels, mark each request conforming only if both limits pass, and find the load that maximizes useful throughput. I would watch rejection and error rates so the metric cannot be gamed by shedding hard requests, and tune batching, routing, caching and capacity based on SLO compliant goodput rather than raw tokens/s.” References: GuideLLM — Goodput SLO definition GuideLLM — Metrics DistServe: Goodput optimized LLM Serving NVIDIA — LLM Inference Benchmarking Fundamentals NVIDIA — GenAI Perf documentation

How do you design a production LLM inference benchmark that reproduces realistic traffic and measures tail latency?

A production realistic LLM inference benchmark recreates how actual users submit prompts—not just a fixed batch of identical requests—and measures how well the system behaves at the tail (p95/p99), where queues, long prefills, slow decodes, and bursts can cause SLO breaches. The main challenge is workload fidelity and measurement validity . It is different from basic inference profiling (Question 189), cache benchmarking (Question 240), and calculating goodput (Question 257). 1. Why uniform synthetic benchmarks mislead Real traffic mixes 100 token chat messages, 8k token RAG queries, long document prompts, and responses from one token to thousands of tokens. The workload is often bursty and contains changing numbers of users; averages hide short high load periods. Prefix reuse , prompt templates, tools, streaming, and output length distribution affect prefill, KV cache occupancy, and decode load. A fixed concurrency test may slow down its request source when the server is slow, hiding the backlog seen under an independent open loop arrival process. A server can post high average tokens/s while some customers see multi second TTFT or unacceptably slow streaming. 2. Build a representative workload 1. Collect a privacy safe, approved traffic trace with relative request arrival timestamps, tokenized prompt lengths, observed/generated output lengths, model/adapter, region, streaming setting, request class, tenant tier, and cache hit eligibility. Strip secrets, personal information, and authentication credentials. 2. Preserve useful correlations , such as long input prompts with particular output lengths, bursty demand from specific workflow types, and repeated system prefixes. Randomly shuffling input/output lengths independently can create unrealistically easy combinations. 3. Stratify by workload: interactive chat, RAG, agentic/tool calls, code generation, and asynchronous summarization. Give each a documented share of the requests or tokens, grounded in measured production traffic. 4. Replay arrival times at their original relative spacing, or generate a trace calibrated pattern (Poisson or gamma/bursty arrivals). Include ordinary load, realistic spikes, and a peak or failure scenario. 5. If a real trace cannot be used, generate synthetic prompts and token count distributions, mark the benchmark synthetic, and explain which production behaviors were not reproduced. Security note: Logs can retain tenant IDs, prompt text, tools and retrieval content. Use least privilege trace access, redact/tokenize appropriately and avoid sending production prompts to unapproved providers. 3. Open loop versus closed loop testing Open loop: requests are submitted according to an external schedule (for example 20 arrivals/s), whether or not previous requests have finished . This reveals queues and overload under independently arriving user traffic. Closed loop: a fixed number of clients issue their next request only after the previous one completes. This models some interactive user journeys but can self throttle when latency increases. Use both when meaningful. For independently arriving API requests, open loop is the primary capacity test ; closed loop helps understand per session behavior. Measure client side queueing if the load generator caps concurrency. Otherwise a nominal 20 RPS open loop run might actually apply less load to the backend during a stall. Track offered RPS , actual dispatched RPS, completed RPS, rejected RPS and retries separately. Do not assume these are interchangeable. 4. Beginner friendly numerical example: how the tail can be hidden Suppose 100 completed requests have the following TTFT measurements : 95 requests: 400 ms each. 5 requests: 4,000 ms each. Calculate: 1. Mean TTFT = (95 × 400 + 5 × 4000) / 100 = 580 ms. 2. Median (p50) TTFT = 400 ms . 3. With the nearest rank percentile convention , p95 is the 95th sorted value = 400 ms . 4. p99 is the 99th sorted value = 4,000 ms . Interpretation: A 580 ms average sounds good, yet the slowest 5% wait 4 seconds for the first token. The result also demonstrates why a p95 alone can conceal a small group of severe outliers: show p99, histograms, request length slices, and the fraction violating each explicit SLO . Percentile interpolation conventions vary between tools. If a 1 second TTFT SLO applies, 95 of 100 completed requests pass TTFT, but this does not establish overall goodput without checking TPOT, full completion, errors and the observation window. 5. A practical two stage benchmark plan Stage A — Controlled baseline Fix model, weights, tokenizer, GPU, inference engine/version, precision, scheduler settings, prefix cache condition, and request lengths. Run repeated low load and concurrency sweep tests to learn basic capacity and eliminate client side bottlenecks. Stage B — Production shaped load Replay anonymized request traces or set approximate production token length distributions and arrival timing. Use warmup , several steady state windows, peaks and recovery; keep measurement windows and failure counting explicit. Test a traffic mix: short chat, long document prefill, long generation, shared prefix reuse, cache misses, plus intermittent disconnects and deadlines where relevant. Sweep offered rates across the saturation knee, not just concurrency, and repeat runs with documented random seeds. Run from a load generator with enough CPU, networking and file descriptor capacity; benchmark the same network path users take, or separately report in cluster baseline measurements. 6. Example AIPerf command NVIDIA AIPerf supports OpenAI style chat endpoints, synthetic input/output length distributions and arrival patterns. The following is an illustrative command ; verify flags against the installed version: This example comes from NVIDIA's AIPerf benchmarking guide and gives a Poisson request arrival process with variable input/output lengths. It is a starting point; to reproduce real production traffic, replace or extend the synthetic inputs with approved trace replay . Alternative tools include GuideLLM for configurable real/synthetic data, rate/concurrency sweeps and SLO analysis, and vLLM benchmark serve for configurable request rate, burstiness, concurrency and request level percentile metrics. Check current CLI definitions because option names may evolve. 7. Metrics that must be reported Per request latency: p50/p90/p95/p99 TTFT, inter token latency (ITL) , total request latency, and queue waiting time if observable. Serving capacity: offered/accepted/completed RPS, generated tokens/s, input tokens/s, concurrency, success/rejection/timeout/error rates. SLO compliance: percentage of original requests meeting all required objectives and joint SLO compliant goodput . Do not count an error or timeout as a fast success. Workload fidelity: token length percentiles, prefix hit mix, arrival rate time series, burstiness, request class and cache warming state. GPU/system: memory/KV occupancy, preemptions, CPU/load generator saturation, network, per worker queue depths and utilization. Slices: metrics by prompt length bucket, output length bucket, tenant priority, model/adapter, and endpoint; aggregates can conceal disadvantaged groups. 8. Avoid coordinated omission and other measurement mistakes A closed loop client that sends fewer requests during a slowdown can under sample the worst congestion. Compare scheduled arrival time with actual submission and completion times. If a benchmark client has a concurrency limit, record client queue time separately. The vLLM benchmark tooling exposes client queue related metrics with the appropriate configuration. Do not silently omit timeouts, rejections, cancellations or unfinished requests from user facing failure/SLO rates. A load generator constrained by CPU/network can make a model look faster or slower than it is; cross check achieved request rate and producer resource use. Warm caches can make repeated prompts unrealistically easy. Test cold/warm phases or reproduce the real prefix reuse ratio, with safe per tenant isolation. Tiny samples provide unstable p99 estimates. At 100 requests , the nearest rank p99 depends on almost the last observed request; collect enough observations per important traffic slice and repeat runs. Compare runs with the same request mix , network path, model and dataset seed; otherwise a changed output distribution can explain apparent speed improvements. 9. Production incident: the benchmark said “fast” Incident: A chatbot passes a laboratory test with a mean TTFT of 450 ms at 100 concurrent users, but production p99 TTFT exceeds 4 seconds during morning traffic spikes. Diagnosis: The test sent fixed 512 token prompts to a warmed prefix cache using closed loop clients, while the live system had bursty open loop arrivals, RAG prompts from 200 to 20,000 tokens and mixed decode lengths. The generator also throttled new requests during stalls. Root cause: Unrepresentative request lengths, arrival schedule and cache behavior concealed the long prefill queue and KV contention. Fix: Replay approved traffic traces with arrival timestamps and correlated token lengths; report p95/p99 TTFT/ITL and success/error outcomes by workload slice; vary offered RPS around the goodput knee; confirm the generator is not overloaded. Prevention: Make a trace based mixed workload benchmark a release gate after model, routing, scheduler or cache changes. Store benchmark configuration and outputs for repeatability. 10. Interview ready answer “I would design a two stage benchmark: a controlled model/hardware baseline, followed by privacy safe production trace replay with realistic arrival times, prompt/output length correlations, cache reuse and streaming. I'd use open loop offered rates for capacity evaluation, measure p95/p99 TTFT/ITL and per request SLO outcomes, and include errors, timeouts and client side queueing. I would validate the load generator and choose operating points by sustainable SLO compliant goodput, not peak tokens/s.” References: NVIDIA — Benchmarking LLM Inference at Scale with AIPerf (September 2026) NVIDIA AIPerf — Comprehensive LLM Benchmarking Red Hat — GuideLLM for Real World LLM Inference vLLM Benchmark CLI

How do you perform load, stress, spike, and soak testing to identify production LLM inference capacity limits?

Load, stress, spike, and soak tests exercise different failure modes of an LLM serving system. Load testing checks ordinary and expected peak demand; stress testing pushes beyond that operating range to find the saturation knee and safe failure behavior; spike testing checks sudden arrival bursts and recovery; soak testing checks long running stability, memory growth, and performance drift. Together they identify sustainable SLO compliant capacity , not merely the highest requests per second reported by a synthetic benchmark. 1. The four tests and what each proves Load test: Hold normal and forecast peak traffic long enough to measure stable p95/p99 TTFT, TPOT/ITL, end to end latency, error rate, GPU/KV memory use, and SLO compliant goodput. This tests whether planned traffic can be served reliably. Stress test: Increase offered request rate (and token work) beyond expected peaks in steps until latency targets break, errors or rejections rise, or throughput stops improving. Locate the capacity knee and confirm graceful load shedding rather than OOM or crashes. Spike test: Jump rapidly from baseline to high demand and back. Check admission control, queue limits, autoscaler lag, retries, model cold starts, p99 queue delay, and recovery without request storms. Soak test: Sustain a realistic traffic mix for hours or a workload appropriate long period. Watch for memory/KV allocation leaks, growing queues, stale prefix cache entries, fragmentation, slow increases in TTFT/ITL, unhealthy workers, and cost drift. Bonus — breakpoint test: A controlled ramp to the failure limit is a subtype of stress testing. Do not run destructive breakpoint experiments against shared production capacity without isolation and guardrails. 2. Why requests/second is not enough for LLMs A 200 token chat request generating 50 tokens and a 20,000 token document prompt generating 2,000 tokens impose very different prefill, decode, and KV cache costs. Measure uncached input tokens/s , output tokens/s, queued prompt tokens, active sequences, KV occupancy, preemptions, prefix reuse, and the share of work by tenant/workload. Preserve prompt/output length correlation , streaming, rate burstiness, cache hit ratio, deadlines and client disconnect behavior. An independent/open loop arrival schedule exposes backlog growth; fixed concurrency/closed loop clients can accidentally slow their offered load when server latency rises. Test both steady state demand and transient workload changes; an autoscaler that handles sustained load may still miss a 30 second spike. 3. Beginner friendly numerical capacity example Assume a 2 GPU LLM serving deployment with these illustrative measured results under the same realistic request mix and a joint TTFT ≤ 1 second and TPOT ≤ 50 ms target: Load at 8 offered requests/s: 8 successful completions/s with 95% meeting both SLOs → 7.6 SLO compliant requests/s ; p95 TTFT is 0.7 s. Higher load at 12 requests/s: 12 successful completions/s with 90% SLO compliance → 10.8 goodput/s ; p95 TTFT is 0.95 s. Stress at 16 requests/s: 15 successful completions/s with only 55% compliance → 8.25 goodput/s ; p95 TTFT rises to 2.3 s. Stress at 20 requests/s: 16 completions/s with 30% compliance → 4.8 goodput/s ; queues grow or overload protection rejects requests. Interpretation: Peak observed raw throughput may reach 16 completions/s , but the best tested SLO compliant goodput is 10.8 requests/s at the 12 RPS load point. A conservative provisioned operating point should be below the saturation knee with headroom for burst variation and worker loss. These figures are illustrative, not a specific product's performance claims. Queue growth check: If sustained offered demand is 20 requests/s but the system completes only 16 requests/s , an unbounded queue can grow at about 4 requests/s . After 60 seconds it would accumulate roughly 240 requests if no requests time out, cancel or get rejected. Real systems should enforce bounded admission and record all shedding. 4. Design a testing sequence 1. Freeze the environment: model weights, tokenizer, precision, GPU SKU/count, inference engine, routing, model/context limits, and reproducible traffic dataset. 2. Validate the generator: smoke test the endpoint; verify offered versus actual dispatched RPS, GPU side metrics, stream parsing and output token counts. NVIDIA warns that high concurrency benchmark clients can bottleneck before the inference server. 3. Baseline and load test: use typical and forecast peak distributions for sufficiently long stable windows, after explicitly recording warmup and cache state. 4. Stress/ramp test: increase offered RPS in manageable steps while preserving request length mix. Measure the first sustained SLO breaches, queue growth, admission rejections, and OOM/preemptions. 5. Spike/recovery test: rapidly increase rate (for example, 8 → 24 → 8 RPS ) and measure p99 latency, retry amplification, maximum queue, autoscaler reaction, and time to stable recovery. 6. Soak test: sustain a normal or moderately high but safe load for an appropriate extended period; look for statistically meaningful memory and latency slopes, cache fragmentation effects, and long term availability changes. 7. Failure injection extension: repeat selected tests with a replica restart, degraded network, or full prefill/decode worker pool to validate resilience without causing customer impact. 8. Compare runs using matched traffic seeds, warm/cold cache assumptions and confidence intervals; preserve raw traces and configuration. 5. Example capacity sweep with GuideLLM GuideLLM supports sweep profiles that begin with a synchronous baseline, discover a throughput ceiling, then test intermediate rates. Its documented example can be adapted to a compatible OpenAI style server: This is a controlled synthetic sweep , not a complete production traffic replay. Use representative trace based or heterogeneous traffic for production conclusions, and check installed tool/version options. NVIDIA AIPerf offers configurable request rates, concurrency, Poisson/gamma arrival patterns, synthetic token distributions and trace replay. For general API/application traffic, Grafana k6 documents smoke, normal load, stress, spike and soak profiles. Select the tool and observation window appropriate to the LLM endpoint and full user journey. 6. Establish pass/fail criteria before testing Normal load: p95/p99 TTFT/TPOT and end to end latency within required SLOs; acceptable errors and rejections; stable backlog and GPU/KV pressure. Stress: bounded queues and explicit, controlled overload responses instead of OOM, deadlock, cascading retries or runaway tail latency. Determine highest stable SLO compliant operating rate , not just peak raw tokens/s. Spike: quantify maximum queue delay, response rejection rate, autoscaling lag and recovery time; protect interactive users from batch traffic. Soak: no unexplained sustained rise in allocated GPU/host memory, KV cache occupancy at fixed workload, latency, error rates, or worker restarts. Per tenant fairness: ensure aggregate goodput improvements do not hide starvation of one request class or customer tier. A service may be “up” but still fail the performance test if nearly all requests violate streaming latency SLOs. 7. Production incident: short tests pass, overnight traffic fails Incident: The team ran a five minute fixed concurrency benchmark and concluded the LLM endpoint could handle 20 RPS. During a long daytime workload, TTFT worsens hour by hour; an evening surge causes queues to explode and users see many timeouts. Root cause debugging: 1. The fixed concurrency generator throttled itself as the server slowed, so its claimed 20 RPS capacity was not a sustained offered load. 2. Reused short prompts created unusually favorable prefix cache hits versus production. 3. The test ended before a sustained memory/queue growth pattern emerged. 4. The gateway retried 503 responses aggressively, amplifying the spike. Engineering solution: run open loop load and stress sweeps , reproduce the real short/long prompt mix and prefix reuse rate, introduce a bounded token aware queue and jittered retries, and run a soak test with GPU memory/KV/queue trend alerts. Separate a confirmed leak from legitimate growth caused by different context lengths. Prevention: make distinct load, stress, spike and soak suites release gates, include peak traffic and replica failure scenarios, and record a repeatable capacity envelope for each model/hardware configuration. 8. Critical observations and cost trade offs Saturation point varies with workload: do not state a single “max RPS” without token distributions, model version, GPU topology and latency SLOs. Client bottlenecks: check generators' CPU/network limits, dropped scheduled arrivals and client queueing before attributing tail latency to GPUs. Scale up cost: more replicas improve headroom but may incur GPU idle costs and inter replica KV transfer/network overhead. Aggressive batching: can improve output tokens/s while reducing SLO goodput by increasing TTFT/ITL. Early rejection: protects active requests but is not free—record availability, 429/503 and final user success separately to prevent gaming goodput. Percentile reliability: collect enough samples within each workload bucket; rare p99 delays are hard to estimate from a tiny run. 9. Interview ready answer “I would first validate a realistic open loop load generator and baseline the deployment. Then I'd run a normal load test for SLO stability, a stepped stress test for the capacity knee, a sudden spike test for queues and recovery, and a long soak test for memory or latency drift. I'd track p95/p99 TTFT and TPOT, completion/error rates, SLO goodput, KV occupancy and retries, including failures and rejections. The production capacity is the highest repeatably stable SLO compliant demand with safety headroom, not the largest observed raw token rate.” Research references: NVIDIA AIPerf — Benchmarking LLM Inference at Scale NVIDIA AIPerf — Metrics Reference GuideLLM — Finding Optimal Concurrency Grafana k6 — Types of Load Testing

How do you estimate GPU capacity and cost per SLO-compliant LLM inference request?

Estimate inference capacity from measured SLO compliant goodput, then divide the fully loaded cost of the required GPU fleet by successful SLO conforming requests. A GPU that produces more raw tokens is not necessarily the cheaper deployment if its TTFT, inter token latency, or error rates violate application objectives. This lesson builds on Question 257 (defining goodput), Question 258 (realistic benchmarking), and Question 259 (finding sustainable capacity); it focuses on converting those measurements into GPU counts, headroom, and unit economics . 1. First check that the model fits Approximate model weight memory = parameter count × bytes per stored parameter, plus quantization metadata and runtime overhead. Add memory for KV cache at expected context lengths and concurrency, CUDA/workspace buffers, and fragmentation headroom. A serving replica may need multiple GPUs via tensor or pipeline parallelism; replicas are not GPUs . Tensor/pipeline parallelism can make one copy fit across GPUs, but communication can reduce efficiency. Data parallelism duplicates a model and raises throughput only after each replica fits. NVIDIA Dynamo's deployment documentation expresses a starting check as weights + KV cache + overhead ≤ aggregate VRAM available to one model replica . This is a fit check, not a promise of latency or throughput. 2. Benchmark the correct unit of capacity For a specific deployment unit (for example, one model replica using two GPUs): 1. Replay the expected input/output token distribution, burst pattern, prefix cache behavior and streaming mix. 2. Sweep offered arrival rate and concurrency. 3. Record completed and rejected requests, TTFT, TPOT/ITL, p95/p99 end to end latency and error rates. 4. Compute SLO compliant goodput : requests completing successfully with all required latency objectives satisfied per second. 5. Choose a sustainable operating goodput with safety margin for worker loss and bursts, not the absolute peak seen in a short experiment. Goodput is workload and SLO specific: a short chat 8 RPS configuration does not imply eight 32k document requests per second. 3. Beginner friendly numerical sizing example Assume an illustrative two GPU replica , using an approved model and fixed traffic mix: Measured sustainable SLO compliant goodput: 8 requests/second per replica while meeting defined TTFT and TPOT targets. Forecast peak demand: 30 incoming requests/second . Choose a conservative 75% target operating utilization relative to that measured capacity. Sizing: 1. Target capacity per replica = 8 × 0.75 = 6 SLO compliant requests/s. 2. Required replicas = ceil(30 / 6) = 5 replicas. 3. Each replica uses two GPUs → 5 × 2 = 10 GPUs . 4. Provisioned nominal benchmark capacity = 5 × 8 = 40 requests/s. At 30 RPS, nominal loading is 75% . Failure domain check: if one of five replicas fails, remaining benchmark capacity = 4 × 8 = 32 requests/s, but at the chosen 75% target its planned load safe capacity is only 24 RPS . Thus five replicas do not satisfy the 75% utilization policy after N−1 failure . For this requirement, solve (replicas − 1) × 6 ≥ 30: six replicas / 12 GPUs are needed. This distinction prevents falsely claiming availability from nominal peak capacity alone. These figures are illustrative, not measured hardware benchmarks. Scale for token work and the relevant SLO, not generic RPS. 4. Unit cost calculation Suppose each GPU costs $3 per GPU hour (illustrative, not a current market price), and we provision five two GPU replicas = 10 GPUs for a whole hour. GPU cost per hour: 10 × $3 = $30/hour. Suppose actual observed successful SLO conforming production traffic is 20 requests/second sustained . Useful requests per hour: 20 × 3,600 = 72,000. GPU only cost per SLO compliant request: $30 / 72,000 ≈ $0.000417, or approximately $0.417 per 1,000 good requests . Do not use capacity of 40 requests/s as the denominator when only 20 requests/s actually arrive and pass SLOs. The provisioned GPUs are billed even when idle, so lower demand raises cost per useful request. For a fully loaded cost, add CPU nodes, networking, shared storage, KV transfer, control plane, monitoring, model provider/license fees, cross zone traffic, and engineering/operations charges as appropriate. Normalize to the same region, billing term and utilization. 5. Cost per token versus cost per SLO request GPU cost per output token: GPU hours cost divided by successfully generated output tokens . Input prompt tokens require distinct treatment. Cost per million tokens: useful for comparing similar request mixes, but can be misleading when one fleet processes very long prompts and another short prompts. Cost per good request: total serving cost divided by successful, SLO conforming requests. This aligns more closely with the interactive user's experience. Cost per good request at a quality target: even better when model variants differ in accuracy or format compliance; a cheap wrong answer is not a successful service. Different models can output different token counts for the same task. Benchmark equivalent workload and quality before declaring a model cheaper. 6. Compare two possible deployments Assume the same traffic mix and SLO at 30 peak RPS : Plan A: two GPU replicas at 8 good requests/s each and $3/GPU hour. Five replicas → 10 GPUs, $30/h before shared overhead. Plan B: one GPU replicas at 5 good requests/s each and $4/GPU hour. At 75% target loading, a replica contributes 5 × .75 = 3.75 RPS, so ceil(30/3.75)=8 replicas → 8 GPUs, $32/h . Plan B uses fewer GPUs but is more expensive per hour at its illustrative price. Conversely, an option with expensive GPUs may be cheaper per good request if it offers substantially higher SLO conforming throughput. Re evaluate after N−1, zoning and minimum instance requirements. 7. Production incident: expensive underutilized fleet Incident: A team buys 16 GPUs because a synthetic test reported a very high tokens/s figure. At real workloads, p95 TTFT breaches 1 second SLO whenever long prompts arrive, and normal overnight traffic leaves much of the fleet idle. Debug: Compare actual request/token distributions with benchmark inputs, tail TTFT/TPOT, prefix hit ratio, each replica's KV occupancy, accepted versus successful requests, and GPU billing hours. Root cause: Capacity planning used peak raw output tokens/s , assumed every GPU was independently serving one complete replica, and divided cost by theoretical throughput instead of actual SLO conforming traffic. Engineering solution: Run representative goodput sweeps; identify the appropriate parallelism configuration and memory fit; use peak plus failure headroom, autoscaling where practical, and compute cost per observed good request. Consider separating online and offline workload pools. Prevention: Review capacity with production traces, repeat benchmarks after model/hardware changes, budget for deployment startup, check N−1 and multi zone constraints, and alert on goodput per GPU and cost per good request trends. 8. Production trade offs and monitoring More headroom: better resilience and tail latency, higher idle GPU cost. More batching/concurrency: may raise raw throughput but reduce latency compliant goodput as KV or decode contention increases. Quantization: can reduce model/KV memory and permit additional concurrency, but confirm answer quality and kernel performance. Prefix reuse: may lower prefill costs; do not generalize warm cache synthetic results to cold, privacy isolated traffic. Disaggregated prefill/decode: size each worker class independently and include KV transfer and networking costs; not all GPUs have the same role. Autoscaling: account for model loading delay, reserved capacity, minimum GPU commitments, and short bursts that cannot wait for new replicas. Metrics: cost/GPU hour, GPUs/model replica, queue depth, KV use, average and p95/p99 TTFT/ITL, successful SLO requests/hour, GPU hours per 1,000 successful requests, fallback and rejection rates. 9. Practical decision recipe 1. Verify weights+KV+overhead fit in the planned replica topology. 2. Find sustainable SLO goodput under a representative production load profile. 3. Divide peak forecast by measured goodput × planned utilization/headroom ; round up and account for N−1 and failure domains. 4. Multiply replica count by GPUs per replica and their actual billed hourly price . 5. Divide total cost by observed or realistic forecast successfully completed SLO compliant requests , not nominal request capacity. 6. Run sensitivity scenarios (traffic ×1.5; long prompt share doubles; one replica fails; GPU rate changes). 7. Revisit periodically as model versions, latency policies, or observed traffic shifts. Interview ready answer: “I'd verify the model and KV cache fit, benchmark sustainable TTFT and TPOT compliant goodput per multi GPU replica, and size replicas for peak demand with headroom and N−1. Then I'd compute fully loaded dollars per hour divided by actual SLO passing requests per hour. I wouldn't divide by theoretical tokens/s, because it ignores tail latency, unused paid capacity, and workload mix.” References: NVIDIA — LLM Inference Benchmarking: How Much Does Your LLM Inference Cost? NVIDIA Dynamo — Model Deployment and Sizing NVIDIA NIM — Inference Benchmarking NVIDIA Dynamo — Advanced Performance Tuning