Large Language Models
Large language models are deep learning models, usually transformer-based, trained to predict tokens from large corpora. After pretraining, they are often instruction-tuned and aligned so they can follow prompts and assist users.
How do you design a cache-aware request router for distributed LLM inference?
A cache aware LLM router selects an eligible model replica using both prefix KV cache locality and current server load. Unlike round robin routing, it tries to send a request to a replica that already holds the same prompt prefix KV states, avoiding repeated prefill. But blindly following the cache can overload popular replicas, so the router also needs queue, latency and capacity signals. 1. Why ordinary load balancing can waste GPU work Imagine two replicas running the same compatible model and tokenizer . Replica A has already computed the KV cache for a large, shared system prompt; replica B has not. Round robin may send a matching request to B, forcing B to recompute that prefix. A cache aware router recognizes the reusable prefix on A and may send the request there if the time saved is greater than A's expected queueing delay . The prefix match must respect model/adapter identity, multimodal and position context, cache validity and permitted tenant sharing. A semantic similarity score is not sufficient for KV reuse. 2. Architecture and request path 1. Authenticate and select compatible endpoints. Filter by model, adapter, version, security boundary, health and hardware capabilities. 2. Compute a prefix identity from normalized token/hashed block context without putting raw sensitive prompts into shared router logs. 3. Estimate available KV cache overlap per endpoint. Options include an index/controller reporting block ownership, or approximate locality using a session or prefix hash. 4. Collect live load signals: waiting/running requests, KV occupancy, predicted prefill/decode cost and observed TTFT. 5. Rank eligible replicas using the predicted time to first token or a calibrated score that combines cache hit benefit and load; route to the best choice. 6. Update cache metadata and health state. Deal with evictions, replica restarts and scale outs; on missing or stale cache information, fall back to ordinary load aware routing. 3. Beginner friendly numerical example A 4,000 token prompt reaches two healthy replicas: Replica A: 3,000 reusable prefix tokens; estimated queue wait 220 ms . Replica B: no cached prefix; estimated queue wait 20 ms . Assume uncached prefill takes 400 ms , and reusing 3,000 of 4,000 tokens hypothetically reduces A's remaining prefill work to an equivalent 100 ms . Estimated time before the first token, ignoring other common overhead: A = 220 + 100 = 320 ms ; B = 20 + 400 = 420 ms . Route to A in this example. But if A's queue grows to 400 ms , then A ≈ 500 ms , so routing to B would be better. These are illustrative estimates. Prefill compute is not strictly linear in token count, cached suffix processing still attends to the full prefix, and actual scheduler and hardware behavior should drive the cost model. 4. A practical score One documented vLLM Production Stack load aware scoring approach is: Choose the endpoint with the highest score. β controls how strongly the router avoids overloaded replicas. A large β favors load balancing; a small β favors cached prefix reuse. This is an example policy, not a universal optimal formula. 5. Common operational failures Hot prefix concentration: many requests share one prefix and overload its cache holder. Include live load and spill to idle replicas. Stale cache index: an eviction, crash or scale out invalidates predicted KV ownership. Treat misses as normal and reconcile metadata. Unequal workloads: raw request counts ignore prompt length, output length, decode phase and cache pressure; ideally use estimated remaining work. Tenant or adapter mix ups: do not share private KV blocks across incompatible trust boundaries. Use trusted cache namespaces/salts or isolate replicas. Expensive remote KV transfers: when a distributed cache can move blocks between replicas, include network transfer latency in the placement decision; remote reuse is not always faster than recomputation. Uneven capacity: some GPUs or serving configurations are faster; normalize predicted latency rather than assuming identical workers. 6. Benchmark and rollout Compare round robin , least loaded , prefix affinity , and cache plus load aware routing using identical repeated prefix and unique prompt traffic. Measure p50/p95/p99 TTFT, overall latency, per replica queues, KV hit token rate, preemptions, GPU usage, cost and SLO compliant goodput. Test cold starts, hot prefixes, cache eviction, replicas failing, and multi tenant traffic. A cache aware policy may lose to least loaded routing when most prefixes are unique. Interview ready answer: “I'd route only among compatible and authorized model replicas, estimate how many prefix KV tokens each can reuse, and combine that cache benefit with queueing and KV capacity signals. The goal is minimum predicted response delay, not maximum cache affinity. I'd fall back safely on stale cache metadata and benchmark against least loaded routing under real workloads.” References: vLLM Production Stack — Load Aware Routing; vLLM — Prefix Aware Routing; vLLM Router load balancing policies; llm d Prefix Cache Aware Routing.
How does disaggregated KV-cache storage work across distributed LLM inference workers?
Disaggregated KV cache storage lets LLM workers move or retrieve attention key/value (KV) blocks beyond one GPU's local memory—for example from CPU RAM, another worker, or a shared cache tier. This can support prefill/decode separation and cross worker prefix reuse , but these are distinct design choices: a disaggregated prefill worker can send its freshly computed KV states directly to a decode worker without a persistent shared KV store. 1. Start from the bottleneck Prefill computes the prompt's attention KV states; decode reads those states while generating tokens. GPU high bandwidth memory (HBM) is fast but finite; long prompts and concurrent requests make KV cache capacity scarce. An inference fleet can keep reusable KV blocks on CPU RAM, secondary storage, or remote peers and pull them into the target worker when reuse is worthwhile. KV storage must preserve exact model/adapter compatibility, token prefix identity, position/attention layout, and tenant security boundaries . 2. Step by step architecture 1. A router/scheduler chooses a compatible inference worker based on load and cache locality. 2. During prefill , a worker computes the prompt KV tensors. The engine or KV connector may export completed blocks to an offload tier. 3. A cache index/metadata service records a block identity and where reusable KV blocks reside. Some deployments instead use direct handoff with request scoped coordination. 4. For a new matching request, a worker checks whether eligible blocks exist locally or remotely. 5. A KV transfer connector fetches compatible blocks over host device or network links (using available DMA/RDMA paths), promoting them into the memory needed for inference. 6. The worker computes any uncached suffix and generates the next tokens. If a block is missing, stale or incompatible, it must recompute rather than silently reuse the wrong state. Important: CPU/SSD/remote transfers cost time and bandwidth, and cache blocks are not interchangeable with the text tokens or generated answer themselves. 3. Beginner friendly numerical example A repeated 8,000 token prefix would require 400 ms of recomputation in a hypothetical deployment. A remote KV store already has reusable states for those tokens: KV lookup and transfer overhead: 120 ms . Additional suffix prefill: 40 ms . With reuse, estimated work before generation ≈ 120 + 40 = 160 ms . Without reuse, suffix plus full prefix processing ≈ 400 + 40 = 440 ms . Estimated improvement ≈ 280 ms (63.6%) for this simplified portion of the request. However, if remote transfer slows to 500 ms , the reused path costs 540 ms , worse than recomputation. Real attention computation isn't strictly additive, and cache placement, network utilization and queueing alter these numbers. 4. Prefill/decode disaggregation versus offloaded prefix reuse Prefill/decode disaggregation: one set of workers processes prompts, then transfers request specific KV states to decode workers. Operators can tune TTFT and inter token latency separately. A persistent shared cache is not required . KV offloading / tiered caching: reusable completed KV blocks move from GPU HBM to CPU RAM and optionally lower tiers. Later requests retrieve them for reuse. Peer to peer KV sharing: a worker pulls cached blocks from another worker or its host memory tier. A cache aware router can use a distributed index to choose reuse locations. These approaches can be combined , but each has different scheduler, transfer and eviction costs. 5. What can go wrong? Transfer slower than compute: especially over saturated networks, small block transfers or distant storage. Stale metadata or eviction: a cache index may advertise blocks that were already reclaimed; the system needs fallbacks and accurate ownership tracking. Incompatible KV layout: tensor parallel partitions, model/LoRA versions, attention variants, dtype and position encodings can invalidate reuse. Cache pressure: GPU still needs enough live KV capacity to process active requests; offloading alone does not remove active memory constraints. Tail latency regression: remote downloads may increase p95 TTFT if coordination/transfer stalls. Security leakage: tenant isolation, authenticated transfer, and access controlled cache namespaces are essential. 6. Production evaluation Benchmark local only , CPU offload , peer to peer , and prefill/decode disaggregated serving under the same representative prompt lengths and concurrency. Track remote block hit and byte hit rates, cache evictions, network throughput, transfer time, p50/p95 TTFT, inter token latency (ITL), model throughput, GPU HBM pressure and SLO compliant goodput. Compare transfer latency with recomputation cost , not cache hit percentage alone. Interview ready answer: “Disaggregated KV storage allows compatible inference workers to reuse attention KV blocks beyond one GPU, using an index, offload tiers and KV transfer connectors. Prefill/decode disaggregation directly hands request KV from prefill to decode; remote prefix caching reuses previously computed blocks across requests. I'd use either only when transfer plus coordination beats recomputation, with strict identity/tenant checks and fallback on cache misses.” Further reading: vLLM disaggregated prefilling; vLLM KV offloading; llm d KV cache management.
How do prefill/decode disaggregation and KV-cache transfer affect TTFT, inter-token latency, and throughput?
Prefill/decode (P/D) disaggregation runs the prompt processing (prefill) and token generation (decode) phases on separate model serving workers. The goal is to prevent long prefill jobs from interrupting interactive decoding and to tune each phase independently. The trade off is that the computed KV cache must travel from prefill to decode before generation can proceed , adding transfer and coordination overhead. 1. Three metrics to distinguish Time to first token (TTFT): delay from request arrival until its first output token. It includes queueing, prefill, KV transfer and initial decoding, where applicable. Inter token latency (ITL): elapsed time between successive output tokens. Lower or more stable ITL makes streaming feel smoother. Some systems report time per output token (TPOT) over later tokens; review metric definitions before comparing them. Throughput: successfully completed requests/s or generated tokens/s under a stated latency/service level target. Increasing raw throughput while violating TTFT or ITL targets is not necessarily useful. 2. Why separation helps—and why it can hurt 1. Prefill is often compute intensive and has high arithmetic intensity for large prompt batches. 2. Decode is often limited by memory bandwidth because each new token accesses the model and KV state. Long prefills on the same GPU may interrupt decode and worsen tail ITL. 3. With P/D disaggregation, specialized prefill workers prepare KV states, then transfer them to decode workers; decode workers produce output without competing with new prefill kernels on the same instance. 4. KV transfer lies on the critical path to the first output token. Its cost depends on cache size, interconnect bandwidth, protocol, queueing and whether transfer can overlap other work. 5. Disaggregation does not automatically improve TTFT or aggregate throughput . Better tail ITL and workload specific capacity efficiency are possible, but added transfers and resource imbalance can erase gains. 3. Beginner friendly numerical example Consider a request with a long prompt and 50 generated output tokens. All numbers below are hypothetical. Combined prefill and decode worker: Waiting + prefill + first token processing = 300 ms TTFT . Because new prefill work intermittently interrupts generation, average interval for the next 49 tokens = 45 ms . Approximate total completion time = 300 + 49×45 = 2,505 ms . Separate P/D workers: Queue + prefill + initial decode, excluding transfer = 270 ms . KV cache transfer and coordination = 80 ms . TTFT = 270 + 80 = 350 ms ( 50 ms worse ). Less prefill interference lowers average interval for the next 49 tokens to 25 ms . Completion time = 350 + 49×25 = 1,575 ms ( 930 ms faster overall ). Interpretation: P/D helps streaming and end to end latency in this scenario even though the first token arrives later. If the network transfer expands to 1,100 ms or the decode pool becomes overloaded, the advantage can disappear. These figures are illustrative, not benchmark results. 4. What about throughput? P/D lets operators size prefill replicas and decode replicas independently and possibly select different parallelism strategies. A decode worker can maintain more predictable token generation when long prefill bursts are isolated. But an undersized prefill pool queues requests (TTFT regression), while an undersized decode pool queues or preempts generations (ITL and throughput regression). Network transfer consumes bandwidth and CPU/GPU resources; each request must have its KV state available on its decode worker. Important nuance: vLLM's disaggregated prefill documentation says the feature itself does not improve throughput; llm d describes workload dependent opportunities for better throughput through specialization. Neither implies a general throughput guarantee. 5. Production checklist 1. Compare aggregated serving with chunked prefill against P/D on identical prompts, output lengths and arrival patterns. 2. Record p50/p95/p99 TTFT and ITL , request throughput, tokens/s and SLO compliant goodput (requests that meet both latency targets). 3. Instrument prefill queueing, prefill GPU utilization, KV transfer duration/bytes, decode queueing, KV cache occupancy and preemptions . 4. Model transfer costs; prefer an efficient network path such as RDMA/NIXL where appropriate, with realistic load and failure tests. 5. Tune the ratio of prefill to decode capacity; test long context bursts, mixed prompt sizes, KV transfer failures and scaling events. 6. Keep model/adapter version, KV layout, tenant isolation and transfer authorization compatible across phases. 6. Simple diagnosis TTFT high, ITL healthy: check prefill queue, compute bottleneck and KV transfer. TTFT healthy, ITL high: check decode congestion, memory bandwidth and KV cache capacity. Both bad: check worker placement, routing, excessive arrivals, network contention and model configuration. Interview ready answer: “P/D disaggregation isolates compute heavy prefill from bandwidth sensitive decoding, which can stabilize tail inter token latency. It adds KV transfer before the first token, so TTFT may increase even when later tokens arrive faster. Throughput depends on pool sizing and transfer overhead; I would benchmark TTFT, ITL and goodput with representative traffic rather than assume disaggregation is always faster.” References: vLLM: Disaggregated Prefilling; llm d: Disaggregation Architecture; llm d: P/D Disaggregation guidance; vLLM: Push Based KV Transfer.
How do you size and autoscale prefill and decode worker pools in disaggregated LLM serving?
Size prefill and decode pools separately using measured token workloads, service time capacity and latency SLOs. Prefill is mainly prompt processing compute; decode generates tokens while retaining KV state and may be limited by memory bandwidth and active KV capacity. Scale whichever phase is the actual bottleneck instead of using GPU utilization alone. 1. What do the two pools need? Prefill pool (P): prompt tokens per second, prompt length distribution, prefill queue waiting time and TTFT. Longer prompts consume more work than shorter ones. Decode pool (D): generated tokens per second, active generations, average output length, inter token latency (ITL/TPOT), remaining KV capacity and request preemptions. Between them: KV transfer time and network bandwidth. Faster prefill is not useful if KV handoff or decoding is saturated. Global goal: meet p95/p99 TTFT and inter token latency targets at minimum sustainable GPU cost, with spare capacity for bursts and failures. 2. Beginner friendly sizing example Suppose a service receives 20 requests per second in steady state. Each request averages 2,000 input tokens and 400 output tokens . 1. Prefill demand = 20 × 2,000 = 40,000 prompt tokens/second . 2. Decode demand = 20 × 400 = 8,000 generated tokens/second . 3. Load testing shows each configured prefill worker can sustain 10,000 prompt tokens/second and each decode worker can sustain 2,000 generated tokens/second while maintaining the desired latency targets. 4. Nominal requirement: 4 prefill workers and 4 decode workers. 5. With a planning target of 70% utilization : ceil(40,000/(10,000×0.7)) = 6 prefill workers ; ceil(8,000/(2,000×0.7)) = 6 decode workers . Important: These are illustrative, workload conditioned rates, not universal GPU benchmarks. Prefill cost is not perfectly linear in length, decode throughput depends on batching and KV occupancy, and heterogeneous workers need individual calibration. 3. Why the ratio changes Many long prompts and short replies : proportionally more prefill capacity. Short prompts and long generated answers : more decode capacity, especially as live KV state grows. Heavy prefix cache reuse reduces fresh prefill work and may change the required P:D ratio. A bursty chatbot and an offline batch job should be modeled separately; their p95 latency targets differ. Scale complete tensor parallel worker groups , not arbitrary fractions of a shard. 4. Autoscaling signals by phase Prefill: in flight or queued input tokens , estimated time to drain the queue, prefill running requests and measured p95 TTFT. Decode: active sequence count, output token backlog when estimable, KV occupancy, preemptions and p95 ITL/TPOT. End to end: admission/transfer queue depth, KV transfer latency and goodput meeting both user facing SLOs. Avoid GPU utilization as the sole metric : continuous batching may keep it near 100% at very different levels of customer visible saturation. 5. Production controller 1. Run representative load tests to estimate per role throughput at acceptable TTFT/ITL. 2. Export per role request/token backlog, KV occupancy and latency to a monitoring system. 3. Use KEDA/HPA or an equivalent controller to raise a role's replica count when projected queueing or SLO pressure exceeds thresholds. 4. Include headroom, cooldown/hysteresis and startup time to avoid oscillation; prewarm new workers because model load times can be significant. 5. Coordinate prefill and decode growth: a larger prefill pool can flood a smaller decode pool. The router should stop over admitting when decode has insufficient capacity. 6. Scale down carefully: allow decode workers to drain active sessions and preserve correct KV transfers. Avoid terminating a replica carrying long generations. 7. Validate under long context bursts, changing prefix cache hit rates, network saturation, mixed tenants and worker failures. 6. Practical incident Symptom: TTFT rises from 250 ms to 900 ms while decode ITL remains near 30 ms. Metrics show the prefill token queue growing, but decode KV occupancy is only 45%. Diagnosis: insufficient prefill capacity. Adding decode GPUs alone is unlikely to improve TTFT. Action: scale prefill workers or reduce prefill load through permitted prompt prefix reuse and request admission. Counterexample: low TTFT but decode KV occupancy above 90%, frequent preemptions and higher ITL suggest a decode capacity bottleneck. 7. Limitations and trade offs A static capacity equation approximates a complex queueing system; burstiness, prompt length tails and per request parallelism cause nonlinear p95 effects. Remote KV transfers may become the bottleneck even when both GPU pools look healthy. More replicas increase cost and may reduce cache locality or increase coordination. Never claim that disaggregation intrinsically improves throughput ; compare against aggregated serving and chunked prefill on the same workload. If using llm d, consult the installed release's documentation; newer autoscaling guides support token aware prefill backlog estimates and decode KV occupancy , while autoscaling strategies and component maturity can vary by release. Interview ready answer: “I'd estimate prefill and decode token demand separately, benchmark per worker service capacity under TTFT and ITL targets, and size each pool with headroom. Autoscaling should follow prompt token backlog and prefill queueing for P, plus KV occupancy, active generations and ITL for D. I'd coordinate scaling across both pools and the KV transfer network and judge success by SLO compliant goodput and cost.” References: llm d autoscaling guide; llm d disaggregated serving; NVIDIA: disaggregated inference on Kubernetes.
How do KV-cache transfer bottlenecks and network bandwidth limit disaggregated LLM inference?
In disaggregated LLM serving, a prefill worker must deliver the request's attention KV cache to a decode worker before decoding can start. Even if both GPUs are fast, inadequate network bandwidth, transfer setup overhead or decode side queueing can make the handoff the bottleneck. A practical design measures KV bytes per request, usable bandwidth, concurrency and transfer tail latency instead of assuming RDMA makes all transfers free. 1. Where KV transfer sits in the request 1. The prefill worker processes the prompt and produces K/V tensors. 2. The scheduler selects a compatible decode worker and coordinates buffer ownership. 3. A KV connector transfers needed states using a supported path, such as NIXL over an RDMA capable fabric; some connectors can also use TCP with higher overhead. 4. The decode worker waits for the required KV state, then generates output. 5. Source KV buffers must be retained until successful transfer or safe cancellation; premature eviction can break the handoff. Transfer sits on the critical path for time to first token (TTFT) . Depending on overlap and scheduling, some costs may be hidden, but network congestion raises TTFT and often p95/p99 latency. 2. Estimate how much KV data moves For a dense Transformer with ordinary full attention, an approximate KV cache size is: KV bytes = 2 × L × T × H kv × D head × bytes per element where 2 is for keys and values, L = number of attention layers, T = cached tokens, H kv = KV heads, and D head = head width. Tensor/sequence parallel layouts, sliding window attention, quantized KV, prefix hits and connector packing change actual transfer bytes. 3. Beginner friendly numerical example Assume a hypothetical model with: 32 layers , 8 KV heads , 128 features per head , 16,384 prompt tokens , and 2 bytes per KV element. KV size = 2 × 32 × 16,384 × 8 × 128 × 2 = 2,147,483,648 bytes ≈ 2.0 GiB . If the effective network/application transfer bandwidth is 20 GiB/s , the ideal payload time is 2/20 = 0.10 seconds (100 ms) . Now suppose: Fixed connector/setup/synchronization overhead = 20 ms . Network contention adds 80 ms of queueing. KV transfer path adds approximately 200 ms before decoding (100 + 20 + 80). At 5 GiB/s effective bandwidth, payload time alone becomes 400 ms ; including the same overhead and queueing gives about 500 ms . These are teaching estimates; real transfers may overlap computation and vary with measured bandwidth. 4. Network capacity under concurrent requests Suppose the service handles 10 requests/second and each requires 2 GiB of unique KV transfer. Required sustained application payload rate ≈ 20 GiB/s . A single path capable of 20 GiB/s effective is already fully utilized before bursts, protocol overhead and competing traffic ; p95 transfer delay may grow sharply. If traffic doubles, either add transfer capacity, reduce bytes/request, exploit valid reuse, or revise placement. Adding more prefill GPUs without fixing the link can make queueing worse. Remember network line rate is not the same as delivered KV bandwidth ; account for serialization, memory registration, host staging and topology. 5. Common production bottlenecks Insufficient bandwidth or congested links: transfers queue even with idle compute. Many small transfers: per transfer setup, control plane round trips and synchronization dominate; batching/pipelining or push based movement may help. Cross node topology: poor locality or hops introduce slower paths; schedule close by compatible prefill/decode pairs when feasible. Host/device staging: GPU to CPU and CPU to GPU copies can erase network gains where GPU direct paths are unavailable. Source KV retention: slow decode admission may leave KV blocks pinned on prefill GPUs, reducing capacity for new prompts. Retries and timeouts: a failed or delayed transfer can waste earlier prefill work and increase tail TTFT. 6. Engineering responses and trade offs 1. Instrument bytes transferred, achieved GiB/s, connection/setup delay, transfer queue time, p50/p95/p99 KV handoff latency, TTFT and source KV retention time . 2. Use an appropriate high bandwidth topology (e.g., InfiniBand, RoCE or EFA with supported RDMA/NIXL transport) where the workload justifies it; verify actual throughput with microbenchmarks. 3. Place workers and route requests for locality , while balancing prefill/decode pool load and avoiding hot destinations. 4. Reduce transferred bytes when valid, via compatible KV head architecture, lower precision KV or reuse of already local prefix blocks; validate accuracy and implementation support. 5. Consider overlapped/pipelined or push based transfer to hide setup/idle gaps, but verify correct buffer lifetime and coordination. 6. Limit admission or scale the network/transfer path when throughput demand exceeds capacity. More GPU replicas do not automatically fix the bottleneck. 7. Compare against aggregated serving with chunked prefill; disaggregation is only beneficial when isolation and specialization exceed transfer/coordination costs. 7. Production incident Incident: p95 TTFT jumps from 450 ms to 1.4 seconds after adding prefill replicas. Prefill compute time falls, but KV transfer queues, byte volume and source KV retention rise. Root cause: the shared interconnect is saturated and decode workers cannot consume incoming KV blocks quickly enough. Fix: cap admission based on transfer budget, place closer prefill/decode pairs, upgrade or partition the transfer fabric, and rebalance worker capacity. Verify p95 TTFT and transfer queue depth afterward rather than only measuring prompt tokens/s. Interview ready answer: “I estimate KV bytes from layer/head/token dimensions, divide by measured effective bandwidth for a lower bound transfer time, then add setup, queueing and retention delays. At scale, requests per second times bytes per request determines required sustained network capacity. I'd monitor transfer p95 and bandwidth, optimize placement/transport and admission, and compare disaggregation against colocated serving.” References: vLLM push based KV transfer (Oct 2026); vLLM disaggregated prefilling; llm d P/D disaggregation.
How do KV-cache compression and quantization reduce memory and network-transfer costs in distributed LLM inference?
KV cache quantization stores attention key/value tensors in fewer bits (for example FP8 rather than BF16). This can nearly halve KV payload size and, when the distributed KV transfer path supports the compact format, reduce network traffic. Compression is broader: it may combine reduced numerical precision, compact packing or token/block reduction. Not every compression strategy is lossless, and reduced KV bytes do not guarantee faster decoding or improved model quality. 1. Why KV memory matters During inference, every generated token uses previously computed keys and values from attention layers. The cache grows with retained sequence length, number of layers, KV heads and head width. In disaggregated serving, a prefill worker transfers request specific KV states to a decode worker. Large caches consume GPU memory and interconnect bandwidth. Lowering cache precision can permit longer contexts or more concurrent requests, but quantization scales, kernel support and transfer serialization determine actual benefits. 2. Estimate cache size For a conventional dense decoder with full KV retention: KV bytes ≈ 2 × layers × tokens × KV heads × head dim × bytes per element The factor 2 represents keys and values. Grouped query attention reduces the KV head count. Special attention types, block padding, tensor parallelism and metadata can change physical allocations. 3. Beginner friendly numerical example Suppose a hypothetical model uses 32 layers , 8 KV heads , 128 features/head , and 16,384 cached tokens . 1. BF16 uses 2 bytes/element : 2 × 32 × 16,384 × 8 × 128 × 2 = 2,147,483,648 bytes = 2 GiB. 2. FP8 uses 1 byte/element : about 1 GiB before scales, alignment and metadata. 3. Nominal KV memory and bytes to transfer drop about 50% . 4. At a measured 10 GiB/s effective payload rate , the ideal transfer payload time falls from 200 ms to 100 ms if both transfer paths can move the data in the respective compact formats. 5. But if packing/quantization and decoding add 70 ms , the total critical path gain is only 30 ms ; if the extra overhead exceeds 100 ms, it may become slower. These values are illustrative lower bound calculations , not measured production benchmarks. 4. What does FP8 quantization actually do? BF16 stores floating point KV values with a wider representation; FP8 e4m3/e5m2 represents them with fewer bits and limited dynamic range/precision. A scaling factor maps the original tensor's magnitude into the quantized range. Uncalibrated scales (such as default 1.0) may lose important values. A serving engine can use per tensor scales, or supported per attention head scales, which may preserve accuracy better. Some attention kernels work on FP8 directly; others require dequantization. Their kernel paths and hardware determine the latency benefit. Quantizing model weights and quantizing KV states are distinct; one does not imply the other. 5. Compression beyond FP8 INT8/INT4 or other low bit KV: potentially larger compression but more quantization error and possible decode/packing overhead. Availability varies by backend. Selective precision: retain sensitive layers or recent KV positions in higher precision when supported; reduce the rest. Token/block eviction or pruning: stores fewer KV positions and can change context information, unlike straightforward precision reduction. Transfer aware compression: producer and consumer must agree on exact format and layout. If KV states are expanded back to BF16 before network transfer , the network saving disappears even if GPU storage was quantized. 6. Correctness and production risks Accuracy regressions: long context retrieval, rare tokens and reasoning can be sensitive to KV precision. vLLM's April 2026 report described a severe needle in a haystack regression for one initial FP8 attention configuration, underscoring the need for realistic evaluation rather than assuming FP8 is always harmless. Compute overhead: quantize/dequantize, extra scales and format conversion may worsen TTFT or inter token latency. Incompatible workers: KV layout, model/adapter version, parallelism strategy, numerical dtype and cache block identity must agree. Bandwidth ceiling: compression is of limited value when the bottleneck is prefill compute or output generation rather than KV transfer. Tenant safety: the transfer path and any reusable KV store require authentication, authorization and isolation. 7. Evaluation and rollout Compare BF16 baseline, calibrated FP8 and any supported lower bit scheme on the same model, hardware, traffic and SLO . Track: KV cache bytes/token, GPU occupancy, available concurrency and evictions. Actual wire bytes , transferred GiB/s, p50/p95/p99 transfer latency and TTFT. Inter token latency, request throughput and SLO compliant goodput. Task accuracy, long context retrieval/needle tests, output divergence and numerical instability. Quantization calibration quality and kernel compatibility. Interview ready answer: “KV quantization reduces bytes per attention key/value element, so BF16 to FP8 can roughly halve cache size and transfer payload. But I must verify the wire format remains compact, that kernels support it and that quality holds—especially for long context tasks. I would measure transfer p95, TTFT, ITL, concurrency and accuracy against BF16; smaller KV does not automatically mean faster serving.” References: vLLM Quantized KV Cache; vLLM April 2026 FP8 KV cache accuracy study; Hugging Face KV cache strategies.
How do you evaluate the accuracy and latency trade-offs of FP8 versus BF16 KV-cache quantization?
Evaluate FP8 KV cache quantization against a BF16 baseline on the same model, workload, serving backend and hardware. FP8 reduces nominal KV tensor storage by about half, but performance depends on supported attention kernels, calibration scales and cache pressure; accuracy can degrade on specific long context or reasoning tasks. Question 246 explained how cache bytes shrink; this lesson focuses on a rigorous go/no go experiment . 1. Keep the experiment controlled Fix the model weights, tokenizer, prompt templates, generation settings, maximum context length, GPU type, tensor parallelism, batch/load profile and software versions. Change only KV cache precision first: BF16 baseline vs FP8 with default scales. Then add calibrated FP8 as a third experiment. If backend selection also changes the attention computation (for example, FP8 attention quantizes Q as well), disclose that the experiment is FP8 KV + attention rather than KV precision alone. Benchmark cold/warm caches and low/peak concurrency separately. 2. What to measure Quality: Task accuracy/pass@1, exact match or retrieval correctness, long context needle/retrieval tests, hallucination rate and output validity. Slice by prompt length (e.g., 2k, 8k, 32k, 128k), generation length, language and difficult reasoning tasks. Use several randomized generations for stochastic decoding; one mismatched response does not demonstrate quality regression. Performance: GPU KV memory and cache capacity, actual KV transfer bytes if disaggregated, achievable concurrency, p50/p95 TTFT , p95 inter token latency, generated tokens/s and SLO compliant goodput. Include quantize/dequantize overhead, scales and metadata, preemption counts and hardware/kernel differences. 3. Beginner friendly numerical example Imagine evaluation on 1,000 labeled long context questions plus a fixed high load serving test: BF16 KV cache 920 correct / 1,000 = 92.0% accuracy . p95 TTFT = 700 ms ; p95 inter token latency = 38 ms . Cache budget occupied = 40 GiB in a particular test. Uncalibrated FP8 KV cache 880 correct / 1,000 = 88.0% accuracy . p95 TTFT = 650 ms ; p95 ITL = 35 ms . Approximate same token KV payload = 20 GiB , excluding scaling/metadata. Calibrated FP8 KV cache 913 correct / 1,000 = 91.3% accuracy . p95 TTFT = 660 ms ; p95 ITL = 36 ms . Approximate KV payload remains 20 GiB plus scale/metadata. Calculate the trade offs: Uncalibrated FP8 loses 4.0 accuracy percentage points relative to BF16, while TTFT improves 50/700 ≈ 7.1% . Calibrated FP8 loses 0.7 percentage points , while TTFT improves 40/700 ≈ 5.7% . If the release rule permits no more than 1.0 percentage point degradation and TTFT must not regress, calibrated FP8 passes this illustrative point estimate screen; uncalibrated FP8 fails . Do not ship based solely on these 1,000 examples : evaluate uncertainty, important task slices, tail behavior and incident thresholds before rollout. All figures are illustrative rather than claims about a specific model. 4. How calibration changes quality FP8 has less precision and dynamic range than BF16. Scale factors map tensor magnitudes into the representable interval. Per tensor scaling shares a scale across a tensor; per attention head scaling uses finer granularity and may help when heads have different distributions. vLLM documents default scales of 1.0, dataset based calibration with llm compressor , and per head scales where its supported backend allows them. Do not assume calibration always fully restores BF16 quality. Test model families, atypical activations, long prompts, sliding window layers and supported selective BF16 fallbacks. 5. Latency and capacity do not scale linearly with bit width BF16 → FP8 cuts nominal bytes per KV element 2 → 1 , but does not guarantee 2× requests/s or half the TTFT . Reduced KV memory may indirectly increase allowed concurrency and reduce preemptions, improving throughput. Some kernels operate directly on FP8; others dequantize KV before attention, adding compute. Cache block packing and host/network format matter too. FP8 attention can change query precision as well as KV precision; results are backend dependent. In distributed serving, verify that data crosses the network in compact FP8 format before claiming a bandwidth improvement. 6. Production rollout 1. Run repeatable offline quality suites and matched load tests. 2. Track long context accuracy against BF16 by length bucket; explicitly test failure prone slices. 3. Shadow test with real traffic and compare answers where evaluable. 4. Canary a small percentage of eligible requests, watching p95/p99 TTFT, ITL, throughput, errors and quality signals. 5. Roll back if a predeclared accuracy or latency guardrail fails; preserve BF16 as a safe fallback for unsupported or sensitive configurations. 6. Re evaluate after changing models, hardware, attention backends, calibration dataset or context window. Interview ready answer: “I would compare BF16, uncalibrated FP8 and calibrated FP8 under identical model/hardware/load settings, tracking per task and long context accuracy plus p95 TTFT, ITL, concurrency and memory. FP8 approximately halves KV payload bytes but may affect quality and kernels. I'd ship only if the measured accuracy budget and latency SLOs hold under canary load.” References: vLLM Quantized KV Cache; vLLM April 2026 FP8 KV accuracy and performance research.
How do you calibrate FP8 KV-cache quantization scales using representative inference data?
FP8 KV cache calibration estimates the numerical scales used to store Transformer attention key/value (KV) tensors in 8 bit floating point format. Rather than assuming every tensor fits a default scale of 1.0, calibration runs representative prompts through the original model, observes KV activation ranges, and chooses scales that reduce clipping and quantization error. This is numerical quantization calibration , not the probability calibration of LLM answer confidence covered in Question 203. 1. Why calibration is needed KV states can have very different magnitudes across layers, attention heads, and keys versus values. FP8 stores fewer bits than BF16; a poorly chosen scale can cause clipping of large values or loss of precision for smaller values. vLLM supports FP8 KV cache with default scales of 1.0 , or dataset derived scales using llm compressor . Per tensor and, on a supported FlashAttention backend, per attention head calibration are available. Weights and KV cache are distinct targets : you can retain original BF16 model weights and change only the KV cache, subject to backend support. Some FP8 attention paths also quantize Q, so identify the complete experiment. 2. How scaling works A simplified convention is to encode a floating point value x using q = FP8(x / s) and reconstruct approximately x hat = s × float(q), where s is the calibrated scale. The exact storage convention and kernel implementation vary by backend. Beginner friendly numerical example: Suppose a sampled key tensor contains values with maximum absolute magnitude 896 . Assume an FP8 E4M3 representation whose largest finite magnitude is 448 (for the relevant encoding). A simple maximum based illustrative scale is s = 896 / 448 = 2. A key activation of 600 is first divided by 2 → 300 , then rounded to a representable FP8 number, and multiplied by 2 during reconstruction. Using s = 1 would put 600 above the illustrative FP8 maximum of 448, so a saturating quantizer would clip it. Important: The formula is a teaching example of abs max scaling , not a guarantee that production calibration simply uses the maximum; outlier aware or percentile based policies can behave differently. FP8 levels are not uniformly spaced, and scale estimates are subject to backend requirements. 3. Choose representative calibration inputs 1. Gather a small, authorized, privacy reviewed sample of real workload patterns or realistic synthetic data. 2. Cover short and long prompts, chat templates, system instructions, document retrieval, code/math, multilingual inputs and uncommon formatting relevant to your service. 3. Match the token length distribution and challenging context windows. Calibrating only on short, generic text can fail at 32k/128k contexts. 4. Keep a separate holdout evaluation dataset . Do not select scales by repeatedly tuning to the final test set. 5. Track tokenizer/model revision, attention backend, dtype, calibration seed and sample counts for reproducibility. 4. Practical calibration workflow using vLLM tools Start from an unquantized, compatible checkpoint and its tokenizer. Prepare the calibration prompts using the same chat template and tokenization as production; select samples and maximum sequence lengths deliberately. Use llm compressor and its FP8 attention/KV cache calibration recipe, selecting per tensor scales initially and per head scales where supported. Run calibration to observe attention activations and write the computed Q/K/V scales into the saved model's quantization metadata as supported. Load that calibrated artifact in vLLM with kv cache dtype="fp8"; verify that the calibrated scales, not default 1.0 scales, were loaded and that the selected attention backend supports the chosen granularity. The official commands and supported parameters evolve: follow the pinned vLLM/llm compressor version's executable example rather than copying an old recipe blindly. 5. Per tensor versus per head example Imagine two KV heads in the same layer: Head A observed maximum absolute value = 448 → illustrative scale 1.0 . Head B observed maximum absolute value = 896 → illustrative scale 2.0 . One shared tensor scale of 2.0 can accommodate both heads but reduces the representational resolution available for smaller Head A values. Per head scales of 1.0 and 2.0 can preserve more useful detail for each head, at the cost of added scale metadata and backend constraints. Separate K and V scales matter; some runtimes also calibrate Q scales when using quantized attention. Do not assume every serving kernel supports per head scales. 6. Validate before deployment Compare BF16 baseline , FP8 default scale , FP8 calibrated per tensor , and FP8 calibrated per head (if supported), keeping the model and workload fixed: Accuracy: long context retrieval, needle in a haystack tests, exact match, reasoning/code correctness and task specific failure rates. Numerics: saturation/clipping frequency, quantization error by layer/head and rare activation outliers. Performance: KV bytes, p50/p95/p99 TTFT, inter token latency, cache preemption rate, concurrency and actual wire transfer bytes in distributed serving. Safety and release: predeclare acceptable accuracy degradation, use shadow/canary traffic, and revert to BF16 or selected higher precision layers for sensitive regressions. Calibration should be rerun or checked after changes to model weights, attention backend, chat template/data distribution, supported context lengths, or quantization scheme. 7. Typical troubleshooting Long context quality worsens: extend calibration samples to realistic long contexts and test per head or mixed precision options. FP8 metrics unexpectedly differ: ensure scales were actually loaded; check default scale fallbacks and the selected kernel's Q/K/V quantization. KV memory is smaller but latency is not: inspect quantize/dequantize overhead, occupancy, attention kernels and network transfer format. Rare prompt types fail: improve domain coverage; inspect head level outliers and protected layers rather than simply lowering a global quality threshold. Interview ready answer: “I'd collect representative prompt traces, run the BF16 checkpoint through an FP8 KV calibration recipe, estimate compatible per tensor or per head Q/K/V scales, save them with the model and verify vLLM loads them. I'd compare BF16, default scale FP8 and calibrated FP8 on long context quality and p95 latency, with rollback if accuracy or backend compatibility fails.” References: vLLM — Quantized KV Cache; LLM Compressor — KV Cache Quantization Example.
When should you use mixed-precision KV caching to preserve accuracy in sensitive Transformer layers?
Mixed precision KV caching assigns different cache storage precisions to different Transformer attention layers (or other supported groups) instead of quantizing every layer identically. For example, keep particularly sensitive layers in BF16 and use FP8 for less sensitive layers. The goal is to recover accuracy, or avoid unnecessary low precision kernel overhead, while retaining most of the KV memory benefit. 1. Why uniform FP8 is not always ideal KV cache quantization reduces memory, but different layers respond differently to reduced precision: some are relatively tolerant, while others cause much larger model output or long context retrieval regressions. A layer containing short window or local attention may not benefit enough from the smaller cache to offset FP8 conversion and kernel overhead. vLLM's 2026 FP8 study describes a kv cache dtype skip layers setting for such hybrid configurations and sensitive layers. Do not confuse per head calibration scales (Question 248) with changing the bit width of a whole layer. Calibration helps FP8 represent values; mixed precision decides which layers should use a higher precision cache . Feature support is backend and release dependent . Do not assume an arbitrary model or every attention kernel supports different KV dtypes per layer. 2. Beginner friendly numerical example Imagine a 32 layer Transformer where each BF16 KV cache layer uses 100 MiB for a specific workload. All BF16: 32 × 100 = 3,200 MiB . All FP8: 32 × 50 = 1,600 MiB (idealized half size, ignoring scales and metadata). Suppose testing identifies 8 sensitive layers to retain in BF16 and 24 layers suitable for FP8. Mixed precision: 8 × 100 + 24 × 50 = 2,000 MiB . Saving versus all BF16: (3,200 − 2,000) / 3,200 = 37.5% . Extra cost versus all FP8: 400 MiB, which may be worthwhile if quality is restored. Illustrative evaluation: On a 1,000 question long context holdout, suppose all BF16 produces 940 correct , all FP8 910 , and the mixed layout 936 . Accuracy is 94.0%, 91.0% and 93.6%, respectively. Mixed precision loses only 0.4 percentage points relative to BF16 while saving 37.5% of idealized KV bytes. These are fabricated teaching results, not measured results from a paper or a model release . 3. How to find sensitive layers 1. Establish a BF16 reference on representative workloads, including long context retrieval, code/reasoning, multilingual and unusual inputs. 2. Run a calibrated all FP8 configuration, then quantify the quality gap by task and context length. 3. Ablate layers: restore a candidate layer/group to BF16 while keeping others FP8 and observe the quality change. Repeat systematically; layer effects may interact. 4. Rank layers by recovered quality per additional byte of BF16 KV and measure latency impact. 5. Select a practical hardware/backend supported subset rather than arbitrarily assigning fractional bits to unsupported layers. 6. Verify on a held out dataset. Studies such as KVTuner use offline layer sensitivity search to select hardware friendly mixed precision KV policies. 4. When mixed precision helps Uniform FP8 fails an accuracy threshold , but restoring a small subset of layers recovers the needed quality. Long context retrieval or reasoning is disproportionately sensitive to quantization noise in certain attention layers. Some short window attention layers run faster in BF16 because FP8 quantization costs outweigh the smaller KV movement. GPU memory or network budgets remain too tight for all BF16 caching. Your chosen serving runtime explicitly supports heterogeneous cache layouts and compatible attention kernels. It may not help if nearly every layer is sensitive, scale calibration fixes the issue without mixing, or a simpler BF16 only configuration already meets memory and SLO targets. 5. Production pitfalls and trade offs Backend support: mixed FP8/BF16 storage may require explicit per layer skip controls and compatible attention kernels; the global kv cache dtype setting alone does not imply arbitrary per layer precision support. Allocation and fragmentation: heterogeneous block sizes can complicate paged cache allocation, packing, memory accounting and prefix cache identity. Transfer correctness: prefill and decode workers must agree on each layer's KV format, scales, model/adapter version and layout. Do not assume a mixed cache transfers as one uniform FP8 tensor. Not automatically faster: extra BF16 bytes and kernel dispatch/conversion costs affect TTFT, inter token latency and network bandwidth. Sensitivity is workload dependent: a layer choice optimized for short prompts may fail under long contexts or another domain. Quality policy: keep a BF16 fallback and roll back when holdout/canary accuracy or latency guardrails fail. 6. Measure before shipping Compare BF16, calibrated uniform FP8, and supported mixed FP8/BF16 using the same model, GPU, traffic and inference configuration. Track: Correct answers and task failures by context length, especially worst case slices. Actual allocated KV bytes, bytes moved over the network and cache occupancy. p50/p95/p99 TTFT, inter token latency, throughput and SLO compliant goodput. Mixed precision kernel compatibility, per layer quantization error and rollout errors. A reasonable selection rule is minimize KV bytes/cost subject to an explicit accuracy floor and latency SLO , not simply maximize the number of FP8 layers. 7. Interview ready answer “I'd use mixed precision KV caching when some Transformer layers exhibit a disproportionate quality loss from uniform FP8, or when FP8 overhead is counterproductive in short window layers. I'd rank layer sensitivity through ablations, keep the important layers in BF16, quantize the others, and benchmark the supported per layer implementation against all BF16 and calibrated all FP8. The goal is to recover accuracy while retaining most of the memory saving, without violating latency SLOs.” References: vLLM, The State of FP8 KV Cache and Attention Quantization (April 2026) KVTuner: Sensitivity Aware Layer Wise Mixed Precision KV Cache Quantization (ICML 2025) KVmix: Layer Importance Aware Mixed Precision KV Quantization (AAAI 2026)
How do you design admission control and backpressure to prevent overload in production LLM inference?
Admission control decides whether a new LLM request can enter the inference system now. Backpressure slows or rejects incoming work when the model, KV cache, queue, or downstream workers are near saturation. The objective is not maximum raw requests accepted; it is high SLO compliant goodput without runaway queueing, out of memory errors, or unfair tenant starvation . 1. Why LLM inference needs more than a request per second limit A 100 token prompt generating 20 tokens and a 20,000 token prompt generating 2,000 tokens cost very different amounts of GPU work and KV memory. Prefill requires prompt processing work, while decode repeatedly reads model/KV state; long outputs occupy running slots and memory longer. If every request is allowed into an unbounded queue, time to first token ( TTFT ) increases even before GPUs run out of memory. Admission control sets limits at the gateway, queue and model scheduler; backpressure propagates saturation to upstream clients through bounded waiting, explicit rejection, or client side concurrency controls. 2. Beginner friendly numerical example Assume a hypothetical deployment can safely run 12 concurrent LLM requests and retain 8 additional waiting requests while meeting its latency objectives. 1. Total accepted/in flight budget = 12 + 8 = 20 requests. 2. A burst of 35 simultaneous requests arrives while all slots are initially available. 3. Admit 12 to running, queue 8 , and reject or shed the remaining 15 promptly . 4. Without limits, all 35 might remain in flight and 23 wait; that long queue can miss deadlines and amplify retries. 5. Rejected clients can receive a bounded retry hint; accepted requests should still respect their deadlines. This example is deliberately simplified: a real scheduler also budgets prompt tokens, output tokens and KV memory , not request counts alone. For a second, token aware example, suppose a prefill pool sustains 6,000 input tokens/s under its SLO. A 300 ms queue work budget corresponds to roughly 6,000 × 0.3 = 1,800 queued prompt tokens . If the next eligible request adds 2,500 prompt tokens to a nearly full budget, the gateway may defer, reroute or reject it instead of hiding the overload in a long queue. This is a capacity planning approximation , not an exact TTFT guarantee; prefix cache hits, mixed prompt sizes, batching and queueing variability matter. 3. Where to enforce limits 1. Edge/API gateway: authenticate, limit per tenant request rate and in flight requests, constrain prompt and maximum output token budgets, and reject obviously invalid or abusive workloads early. 2. Shared inference router: monitor worker health and estimated workload; route to a compatible worker with available queue/KV capacity rather than flooding a busy replica. 3. Model scheduler: cap active sequences, batched tokens, pending queue size, KV allocation, and long prefill concurrency. 4. Disaggregated prefill/decode pools: maintain separate limits for prefill backlog, KV transfer bandwidth and decode occupancy; otherwise faster prefill can overload the decode side. Current vLLM serving controls include max num seqs (model runner sequence capacity), max num active seqs (RUNNING admission), max num queued reqs (an API server in flight/queue valve), max num queued tokens (a prefill token backlog cap), and watermark for KV allocation headroom. The documented queued request and queued token caps reject with HTTP 503 when reached. Supported flags and precise defaults must be checked against the deployed release. 4. Choose sensible rejection and retry behavior 429 Too Many Requests: useful when the client or tenant exceeds a policy quota or rate limit. 503 Service Unavailable: useful when healthy infrastructure is temporarily saturated; vLLM's documented queue caps use 503. Clients should use exponential backoff with jitter , a strict retry budget and an end to end deadline, preferably honoring a valid Retry After indication. Avoid automatic, synchronized retries that magnify overload. Do not indiscriminately retry a side effecting or partially executed agent/tool workflow. Cancel abandoned generation when the client disconnects or its deadline expires, where the serving backend supports it. Continuing an unused long decode wastes KV capacity and GPU time. For overload, a fast explicit rejection can be better than allowing requests to time out after seconds in a queue. 5. Fairness and priority Protect latency sensitive user traffic from large batch or low priority jobs with separate queue budgets, priority bands or weighted fair scheduling . Prevent one tenant with long prompts from monopolizing KV blocks and blocking others; bound requests and estimated token work budgets per tenant. Priorities should not imply infinite waiting for low priority users. Apply admission and aging/fairness rules, plus clear shedding policies. llm d documents queueing and backpressure combined with multi tenant flow fairness and priority . Distinguish priority policy from physical worker capacity. 6. Production incident and debugging Incident: A chatbot's p95 TTFT rises from 0.6 s to 5 s after batch summarization jobs start. GPU usage is near 100%, pending requests rise, and timeouts trigger aggressive client retries. Root cause: No bounded queue or prompt token admission, so long prefills and accumulating generations exceed service capacity. Retried requests create an overload feedback loop . Engineering response: Separate batch traffic, impose running and queued token limits, reserve KV headroom, return 503 quickly on saturation, apply retry jitter, and cancel expired requests. Increase pool capacity only after measuring which stage is limiting. Prevention: Exercise load spikes, mixed prompt lengths, tenant traffic and worker outages in tests. Alert on backlog growth before OOM or mass timeout. 7. Measure success Goodput: responses meeting correctness and TTFT/ITL/deadline objectives per second, not just raw token/s. Admission: accepted, queued, rejected, timeout and cancelled requests, sliced by tenant/priority. Queues: p50/p95 wait, waiting requests, queued prompt tokens and age of oldest waiting work. GPU and scheduler: running sequences, KV cache occupancy and preemptions. User experience: p50/p95/p99 TTFT, inter token latency, end to end latency and retry amplification. Capacity: compare guarded/unbounded behavior under bursts; ensure fast rejections occur as configured instead of hidden queueing. Interview ready answer: “I’d enforce bounded in flight requests, token aware prefill limits, decode/KV capacity controls and tenant fairness. When capacity is exhausted I’d reject quickly with an appropriate status and retry guidance, rather than queue indefinitely. I’d use cancellation, jittered retries and load aware routing to prevent amplification, and tune limits against p95 TTFT, ITL, error rates and SLO compliant goodput.” References: vLLM serve CLI — active sequences, queued requests/tokens and KV watermark llm d — Queue and Prioritize Requests (Flow Control) NVIDIA Triton — Request Cancellation NVIDIA Triton — Metrics