Large Language Models
Large language models are deep learning models, usually transformer-based, trained to predict tokens from large corpora. After pretraining, they are often instruction-tuned and aligned so they can follow prompts and assist users.
How do you design chaos experiments and disaster-recovery procedures for distributed LLM inference systems?
Chaos engineering for distributed LLM inference deliberately injects a bounded, observable failure to verify that routing, GPU worker replacement, queueing controls and recovery procedures actually preserve user facing service objectives. Disaster recovery (DR) expands the scope to catastrophic node, zone, region or control plane failures and proves that traffic, configurations, model weights and essential state can be restored to an agreed service level. The objective is not to break GPUs for its own sake. Each experiment starts with an explicit hypothesis and a stop condition, tests a single fault domain, measures real request outcomes, and produces a change to improve the system. Unlike Question 255 (designing failover), Question 263 (graceful rollout draining), and Question 270 (triaging individual GPU/CUDA incidents), this lesson teaches how to validate resilience systematically with controlled fault injection and complete DR drills . 1. Establish the steady state and blast radius Before injecting failures, record: Customer SLIs: original request success rate; p95/p99 time to first token (TTFT), inter token latency (ITL/TPOT), full stream completion, error/rejection rates and SLO compliant goodput. Dependency health: ready versus desired replicas, prefill/decode worker counts, KV cache occupancy, network transfer wait, router health, GPU hardware status and load balancer endpoint propagation. Recovery SLIs: time to detect the incident, time to stop routing to the failed worker, time to ready replacement capacity, and time for p95 TTFT/ITL to return below SLO limits. Workload: representative, privacy safe production like prompt/output length mix, open loop arrival schedule, streaming, burstiness and tenant priorities. Pick a steady state hypothesis such as: “With one decode worker lost, 99% of original user requests still complete within our target latency, no active streams silently disappear, and full rated capacity is restored within five minutes.” Start in an isolated staging cluster with synthetic or approved redacted prompts, then move toward a narrowly scoped production experiment only with change approval, on call coverage, monitoring and an immediate abort switch. 2. Define what to break and what it teaches Worker process kill Terminate one noncritical inference engine process deliberately; observe how readiness, routing, active streams and replacement behave. A crash is not a graceful drain. If in flight streams cannot resume, count the lost user requests as failures rather than hiding them in retry totals. GPU device or node loss Simulate node outage / eviction / device unavailability in a safe test environment. Check GPU quotas, alternative placement, tensor parallel group rebuilding and cold start overhead. A single GPU failure can invalidate an entire multi GPU tensor parallel replica, so the fault domain is larger than “one GPU worth of requests.” Prefill or decode worker loss In disaggregated inference, failure of a prefill worker may cause a different failure shape from decode worker loss. The former can stall new prefills, while the latter can interrupt active output streams and strand KV transfer state. Verify the router does not schedule new requests to dead workers and that stale KV handles are reclaimed. Network degradation/partition Inject latency, packet loss, disconnects or partitions between gateway/router, prefill/decode workers and KV transfer services. Test deadlines, circuit breakers, transfer cleanup and backpressure. Example tool: LitmusChaos pod network latency , which can target specific pods/destinations and delay traffic for a bounded time. Validate supported experiment options against the deployed version. Model storage or registry outage Remove access to remote checkpoints or model registry for a bounded test; verify existing ready replicas keep serving while new cold replicas might fail to start. Test availability of verified checkpoint copies, access controls, image caches and model version pinning. Control plane or region disruption Test what happens when autoscaling/control plane updates or a full availability zone becomes unavailable. Existing inference traffic may survive some control plane failures but not all data path failures. For region level DR, route only to an explicitly approved region with compliant data residency and model/version policies. Overload combined with a fault Combine a controlled worker loss with a representative arrival burst only after passing one fault experiments. This tests remaining capacity, retry amplification and queue protection. 3. Beginner friendly numerical example Suppose a distributed LLM deployment uses four independent serving replicas , each able to sustain 6 SLO compliant requests/s on the same measured workload. Total safe modeled capacity = 4 × 6 = 24 requests/s . Current offered demand = 18 requests/s . An experiment kills one replica. Remaining nominal safe capacity = 3 × 6 = 18 requests/s . At the average incoming rate, the remaining fleet has zero modeled spare headroom . Any short surge or longer prompt distribution may cause SLO failure. A second unexpected loss leaves 2 × 6 = 12 requests/s . The excess demand is 18 − 12 = 6 requests/s . An unbounded queue would accumulate approximately 6 × 60 = 360 requests after one minute of sustained deficit, ignoring cancellations/timeouts/rejections. If a replacement becomes inference ready after 180 seconds , the naive unbounded backlog would have grown by roughly 6 × 180 = 1,080 requests during the two replica shortfall. Correct response: cap work admitted by token budgets and user deadlines, shed/defer excess traffic, bring up compatible ready replicas, and include rejections/stream aborts in success metrics. A queue that absorbs 1,080 requests long after their deadlines is not resilience. The figures are illustrative. The independent “six good requests/s per replica” assumption can fail when shared network, storage, router and disaggregated worker dependencies saturate. 4. Hypothesis, procedure and stop conditions A reusable experiment specification should record: 1. Experiment ID and owner: affected model/version, namespace, GPU pool, approved time window and on call operator. 2. Steady state hypothesis: specific SLO thresholds, error tolerance and recovery time objective. 3. Fault: one target process/pod/GPU or narrow network rule, duration, trigger method and reversibility. 4. Blast radius limit: test namespace, one eligible replica or a fixed small percentage of capacity, and excluded high priority tenants. 5. Baseline: normal traffic rate, prompt mix, warm cache state, version, ready GPU count and pre fault metrics. 6. Abort conditions: for example sustained 1% stream failures, p95 TTFT above a critical threshold, failed rollback, or any unexpected tenant/security impact. Set thresholds from approved SLOs, not generic defaults. 7. Observation window: pre fault, fault injection and post recovery metrics plus traces and incident timeline. 8. Rollback action: remove injection rule, uncordon eligible healthy nodes, restore routing and validate request completion. 9. Outcome: hypothesis passed/failed/inconclusive, supporting graphs, follow up changes and retest owner. If telemetry, abort tooling or healthy spare capacity is missing, do not proceed with production chaos injection. 5. Practical Kubernetes fault injection approach LitmusChaos supports container kill, pod delete, pod network loss/latency/partition, CPU stress and node level experiments. Its network latency experiment describes a controlled packet delay injection at targeted pods/destinations. A safe conceptual sequence (not a command to execute on production automatically): Identify the dedicated test namespace and one disposable inference worker. Run a synthetic streaming client continuously and record TTFT, ITL, full response success and error codes. Inject a bounded single worker termination or 200 ms network delay, with predeclared stop and time limits. Observe endpoint removal, failure detection, request results, standby capacity, GPU provisioning, KV cleanup and latency recovery. Remove the fault and verify the system returns to steady state performance. Kubernetes PodDisruptionBudgets: PDBs constrain voluntary Eviction API disruptions but cannot prevent involuntary hardware failures; ordinary direct pod deletion and workload rolling update behavior require separate protections. A PDB alone is therefore not a chaos safety switch . Use explicit target allowlists and permissions. 6. Recovery objectives: RTO and RPO RTO (recovery time objective): maximum acceptable elapsed time from a qualifying outage until the defined service is restored to its agreed state. Specify whether this means first response possible or normal SLO performance restored . RPO (recovery point objective): maximum acceptable age/amount of persisted state lost after recovery. In a stateless inference only API , model weights/configuration usually come from versioned storage; RPO for user state may be not applicable. In flight KV caches are ephemeral and normally cannot be reconstructed token perfectly after a worker crash. In an agentic or conversational product , session history, tool execution records, authorization context and conversation metadata may live outside inference GPUs. Their RPO must be defined and enforced with backups/replication, and replaying a tool action needs idempotency. Do not promise lossless continuation of a partially streamed generation unless the serving stack explicitly implements and tests a compatible continuation protocol. 7. A disaster recovery runbook 1. Declare and scope the incident: identify failing region/zone, missing model replicas, storage/network dependencies and customer impact. 2. Freeze risky automation: pause deployments that could amplify the incident; rate limit retrying clients and cancel impossible deadlines. 3. Select approved recovery region/zone: confirm GPU inventory, topology, model availability, data residency and tenant permissions. 4. Restore control and data dependencies: image registry, version pinned model checkpoints, configuration/secrets, routing metadata, auth, observability and any required session databases. 5. Provision whole serving groups: load model weights, compile/warm runtime, validate tensor parallelism, adapters, KV allocation and readiness with a real inference probe. 6. Shift eligible new traffic gradually: apply circuit breakers and guarded traffic weights; preserve healthy remaining regions and monitor error/latency/quality changes. 7. Handle active user streams explicitly: report interrupted streams and let clients resume/retry from application state where possible, not by assuming migrated GPU KV state. 8. Verify recovered SLOs and state: measure original request success, p95/p99 TTFT/ITL, SLO goodput, tool action integrity, auth, model checksum and backups' recovery point. 9. Return/operate in DR mode: keep production in approved fallback until stable and confirm rollback/failback readiness. Record outage and actual achieved RTO/RPO. Important: keeping a checkpoint in storage does not guarantee a rapid RTO. A new region must also have compatible GPU capacity, sufficiently provisioned network, cached model weights and functioning dependencies. 8. Recovery experiments versus vendor specific fast restart NVIDIA Dynamo documents a shadow engine recovery feature in which an initialized standby engine on the same GPU can reduce engine process recovery time. A published demonstration (August 2026) showed 7.3 seconds to restore a second worker, versus 283 seconds for cold restart in a specific two worker B200/GLM 5.2 experiment. Scope of the evidence: this addresses a class of process/engine failures where the GPU remains available . It does not automatically recover from GPU destruction, node/zone loss, corrupted persistent user state or a region outage. In chaos tests, distinguish process recovery , device recovery , worker group rebuild , and full disaster recovery rather than mixing their success criteria. A fast process restart can reduce lost capacity without preserving in flight stream progress; validate what happens to already active requests. 9. Example production incident Scenario: A GPU node hosting a decode worker becomes unavailable while the router continues sending new decode assignments to stale endpoints. Meanwhile, client retries multiply incoming work. Prefill GPUs appear healthy, but KV transfers fail and TTFT spikes. Root cause: incomplete worker health propagation and stale routing entries, combined with retry amplification and inadequate decode headroom. Immediate containment: Withdraw failed decode endpoints, bound admissions and retries, and preserve healthy worker group capacity. Invalidate orphaned KV transfer handles and account for interrupted streaming requests. Bring up a compatible decode worker/group on available healthy GPU capacity and wait for inference readiness. Prevention experiment: Kill exactly one test decode worker during a representative open loop mixed workload. Validate that the router removes its endpoint within the target detection window, no new transfer is sent to it after convergence, uncompleted requests fail clearly rather than hanging, and overall service returns to its latency SLO inside RTO. Avoid false success: a pod returning to Running state does not prove that a compatible model is loaded, the KV data path works, or previous requests were correctly accounted for. 10. Measure resilience and learn For each experiment publish: Before/during/after: offered/accepted/completed RPS, original request SLO success, errors/429/503, stream completion, p95/p99 TTFT and ITL. Recovery timings: detection latency, route withdrawal, GPU replacement start, model time to ready, and SLO restoration. Distributed state: KV transfer failures, orphaned buffers, failed tensor parallel ranks, session/tool state integrity. Safety: whether stop conditions triggered, whether the injection was removed, and whether customer data/residency policies stayed enforced. Coverage: process kill, node/GPU failure, network degradation, model storage outage, prefill/decode split, and region DR as applicable. Trade offs: Larger spare GPU pools make failover faster but cost more; calculate cost per successfully SLO conforming request during fault scenarios. Frequent chaos tests improve confidence but should not increase real customer risk; start small and expand only with evidence. Reconstructing KV and replaying prompts costs latency/tokens and may produce a different answer. Preserve application level state and guard side effects. Multi region standby may be constrained by GPU scarcity and data residency. Test the actual hardware/model deployment path, not merely DNS switching. Interview ready answer: “I would define the steady state SLIs, fault hypothesis, blast radius, stop conditions and RTO/RPO before injecting anything. I'd start with controlled worker and network failures, then exercise GPU/node loss and full region recovery in dedicated environments. For LLMs I'd explicitly test in flight stream loss, prefill/decode/KV dependencies, model warmup and retry storms. I'd measure original request goodput and the time to restore SLOs—not just pod restarts—then turn each failed experiment into a runbook and regression test.” References: Kubernetes — Disruptions and PodDisruptionBudgets LitmusChaos — Pod Network Latency NVIDIA Dynamo — Inference Architecture NVIDIA — Shadow Engine Recovery (2026) Fast Recovery for LLM Serving via Decoupled Device Memory Lifetime (2026)
How do you design multi-region failover and capacity planning for highly available LLM inference?
Multi region LLM inference places independent, compatible model serving capacity in more than one geographic cloud region so requests can be served despite a region wide outage—or routed closer to users. The crucial question is whether the surviving region has enough ready, SLO compliant GPU capacity for redirected traffic , not whether DNS can point somewhere else. Large checkpoints, GPU scarcity, long model cold starts, KV cache locality, streaming connections, and data residency requirements complicate recovery beyond typical stateless web services. This lesson builds on Question 255 (health aware replica failover), Question 260 (GPU sizing), Question 261 (SLO aware autoscaling), Question 262 (cold start prewarming), and Question 271 (chaos and DR experiments). It specifically covers geographic fault domains, active active versus active passive inference, failover capacity math, global routing, cross region state and cost . 1. Decide if multiple regions are justified A multi availability zone design within one region may already meet many reliability requirements. Multi region deployment becomes appropriate when the service must survive an entire regional outage , meet geographically separated recovery or data residency obligations, or reduce latency for users in distant locations. It adds extra GPU and storage spending, checkpoint replication, traffic management complexity, data consistency concerns and more failure modes. Define a measurable regional outage SLO , recovery time objective (RTO), recovery point objective (RPO), and whether degraded model quality or partial traffic shedding is acceptable during recovery. Do not interpret “multi region” as automatic immunity to shared control plane, identity provider, DNS, cloud account, model registry or software version failures. AWS Prescriptive Guidance emphasizes choosing multi region architecture when a documented driver requires it, rather than making it a default for all workloads. 2. Four recovery patterns Backup and restore: replicate versioned infrastructure configuration, model artifacts, databases and backups to a recovery region, but create most compute only after an outage. Lowest ongoing GPU expense; recovery may be long and depends on quota, GPU availability and weight downloads. Pilot light: keep critical control/state dependencies running in the DR region but provision most model serving GPUs on demand. Faster than full restoration but still requires node creation, checkpoint loading and engine warmup. Warm standby: keep a smaller working, healthy inference deployment ready in the backup region. It serves little/no user traffic until a failover and then scales out. Existing warm capacity protects some urgent requests, but expanding to full capacity may take minutes. Active active: run user serving GPU pools in two or more regions and distribute live traffic. Routing can shift traffic when one region fails, but surviving regions still need failover headroom or they will be overloaded. RTO/RPO labels in vendor diagrams are architectural tendencies, not guaranteed durations . An LLM warm standby with insufficient GPU quota or a 15 minute model download can miss its recovery objective. 3. Beginner friendly numerical capacity example Assume a single one GPU serving replica has a measured, safe operating capacity of 5 SLO compliant requests/second for the real prompt/output token distribution, including chosen headroom. Two regions each receive 20 requests/second during peak business hours. Normal active active operation: Region A: \ceil(20 / 5) = 4 replicas\. Region B: \ceil(20 / 5) = 4 replicas\. 8 GPUs total for 40 RPS; each region is sized for its local share. If Region A fails: The surviving Region B receives 40 RPS if all traffic is eligible for redirection. Required replicas: \ceil(40 / 5) = 8\. Its existing 4 replicas can serve only 20 RPS at the planned safe rate. Shortfall = 20 RPS until four additional compatible GPU replicas become ready, even though global failover is functioning. True immediate regional outage resilience: Preposition 8 ready replicas in each region to handle the entire 40 RPS demand from either side. Fleet size: 16 one GPU replicas , compared with 8 GPUs when only normal demand is covered. This is an illustrative, expensive N region capacity design; actual replication, GPU topology and shared bottlenecks alter the count. Warm standby alternative: Region B keeps 2 ready spare replicas (10 RPS safe capacity) and scales to 8 on failover. With a 40 RPS surge, 30 RPS remains above immediate warm capacity. This offers some fast service but cannot promise full peak SLO compliance until more replicas are ready. In practice it may require priority tiers, queue limits and shedding. Important: passing SLOs at 5 RPS/replica under steady load does not prove a 40 RPS redirected burst can be handled by aggregate averages alone. Verify queueing, load balancing, cache coldness, network path, tenant mix and fault domain independence. 4. Capacity planning formula with loss of a region For two equivalent regions with normal demand \D A\ and \D B\: Total eligible demand = \D total = D A + D B\. Let \c\ = benchmarked safe goodput of one complete serving replica at the specified SLO. To survive loss of either region immediately , each surviving region must have at least \R = ceil(D total / c)\ ready replicas , subject to token mix, routing and shared resource constraints. If a replica requires \g\ GPUs (e.g., four way tensor parallelism), each region requires R × g GPUs , not R GPUs. If the requirements instead allow a few minutes of degraded service, you can deploy fewer warm replicas and explicitly model the recovery/scale up interval and allowed traffic reduction. For nonidentical GPUs or model variants, use measured per region capacity envelopes rather than multiplying unlike replica counts. Recalculate with p95 peak load, extra burst headroom, model startup time, and separately defined N−1 replica/zone requirements. 5. Recovery time budget: traffic switch is not capacity readiness Break the actual failover timeline into: 1. Detect loss of user facing model service , not just a reachable TCP socket. 2. Validate the incident to avoid flapping or falsely isolating a slow but healthy region. 3. Update health aware global routing and wait for its effective propagation to clients/proxies. 4. Confirm the target region has compatible ready inference replicas and room for more sessions. 5. If capacity is missing, allocate GPUs, start containers, stage checkpoint shards, load weights, compile/capture kernels and warm the serving stack. 6. Verify p95 TTFT/ITL, errors, original request success and full streaming completion at redirected load. Illustrative recovery example: failure detection 20 s , global route convergence 15 s , replacement provisioning+model warmup 180 s . Time until full targeted capacity = about 215 seconds , assuming sequential critical path stages. Prepositioned warm GPU capacity could eliminate the 180 second component for the work it can already handle . These are hypothetical times. DNS health checks and low TTLs can assist but do not guarantee every existing client stops using an old endpoint immediately: caching resolvers, persistent HTTP connections and streaming sessions can outlive a DNS change. Global anycast or application gateways have different convergence behavior; measure client observed recovery , not just control plane status. 6. Region level global traffic routing Route only among healthy, model compatible and authorized endpoints. A region containing a healthy web gateway but unavailable prefill or decode GPUs is not fully healthy. Combine geographic/latency aware routing with capacity admission signals so a “nearest healthy” region cannot attract more token work than it can serve. Provide transparent region exclusion , failover weights and optional manual override; automate decisions with hysteresis to prevent traffic flapping. AWS Route 53 supports active active and active passive failover routing using health checks. A DNS policy does not itself assess model specific TTFT, KV pressure or ready GPUs; integrate an application level health endpoint or supported load balancer policy. Google Cloud documents serving in multiple regions behind global load balancing and health based regional failover for supported deployments. Preserve per tenant routing constraints; a user assigned to an EU only residency boundary must not be redirected to an unapproved region merely because capacity exists there. 7. What state must be replicated? Immutable or versioned artifacts: Model weights, tokenizer, chat templates, quantization configuration, adapters, inference engine/container manifests, safety policy and tool schema. Store immutable digests/checksums and verify the recovery region runs the same approved version , unless an explicitly tested fallback is allowed. Keep cross region artifact copies or caches near recovery GPUs; a single region object store dependency makes a multi region deployment fragile. Persistent application state: Auth/session metadata, conversation turns, retrieval index or vector store, tool execution ledger, quotas/billing and any customer data needed to answer requests. Define consistency and RPO for each component. Async replication may lose the latest writes; active active writes can conflict without ownership or conflict resolution. Use idempotency keys and transaction status for side effecting tools, especially when a client retries after a cut off stream. Ephemeral GPU serving state: KV cache blocks, batching queues and an in progress decoder state are generally bound to a running worker and not replicated automatically across regions . A failed region's live streams normally terminate, and restarting a generation may produce different tokens or repeat side effects. Replaying a prompt from persisted conversation history is an application level retry , not bit identical token continuation. State recovery must preserve tenant authorization, retention, encryption, data locality and model licensing . 8. Disaggregated prefill/decode across regions: a common mistake Prefill and decode workers may exchange large KV state over high bandwidth, low latency fabrics; locating a prefill GPU in one distant region and decode GPU in another can add severe transfer delay and cross region bandwidth cost. Prefer regional self contained serving cells with colocated or region local prefill/decode/KV dependencies and local routers. Fail over whole compatible regional serving paths instead of trying to move live KV blocks over intercontinental links. Keep local GPU topology (NVLink/PCIe), KV network and storage budgets separate from the global request routing layer. If the product supports explicit cross region context migration, benchmark transfer cost, consistency and in flight recovery; do not assume that ordinary model serving has it. NVIDIA Dynamo describes a distributed serving architecture with routing, disaggregated inference and KV state paths. Its ability to scale workers within a serving environment does not, by itself, provide automatic regional disaster recovery or global session state replication . 9. Production incident: DNS switches but recovery still fails Incident: Region A receives 70% of user traffic and Region B receives 30%. A regional outage triggers global health checks and sends the failed region's traffic to B. DNS switches within seconds, but B's p95 TTFT jumps from 800 ms to 9 seconds , and error rates surge. Root cause debugging: 1. Routing shows the global controller worked as intended, but Region B has only enough ready GPUs for its usual 30% share plus small spare headroom. 2. Autoscaler asks for new GPUs, but the target region lacks immediately available quota/stock and must download large checkpoints from a cross region store. 3. Cold prefix caches lower prefill reuse, increasing actual token work versus the benchmark. 4. User retries compound the overload, and long queues retain work beyond request deadlines. Engineering solution: Preposition compatible warm capacity sized to the explicit degraded mode SLO and regional loss requirement. Pre stage checkpoints and container images in B; validate usable GPU quota and model startup p95. Make global routing aware of healthy ready capacity and per tenant constraints, not merely DNS endpoint health. Bound admission, honor deadlines, jitter/back off retries and allow defined priority based degradation during recovery. Benchmark full redirected peak traffic with representative long context and cold cache conditions. Prevention: Regularly conduct a controlled entire region traffic evacuation test , measuring routing convergence, ready replica counts, SLO goodput, stream aborts and actual RTO/RPO. A healthy DNS endpoint alone is not a passed disaster recovery drill. 10. Recovery economics and failure trade offs Active active: stronger immediate regional survival and lower geographic latency, but requires duplicated GPUs and careful cross region state synchronization. Warm standby: lower idle GPU spending, but the service must declare what percentage of requests it can handle before additional model replicas become ready. Pilot light: may reduce idle costs but depends heavily on GPU supply, checkpoint download and warmup during an incident. Model fallback: an approved smaller or alternate model can preserve some workflows at different cost/accuracy; only use it when contract, quality, safety and data residency conditions permit. Cross region egress: moving prompts, retrieval context, output and persistent state consumes network bandwidth and may have financial/privacy impact. Avoid large KV transfers across distant regions without measured benefit. Cold cache penalty: loss of regional prefix/KV reuse raises prefill work, so compute failover capacity using the expected post disaster cache state , not only warm steady state benchmarks. Total cost metric: spend per SLO successful user request , including idle warm GPU capacity and replicas provisioned for regional failure. 11. A validation checklist and runbook Before rollout: 1. Record steady state load by region, peak total demand, measured safe goodput per compatible GPU replica, and chosen RTO/RPO. 2. Verify two region loss of one region capacity (or document the explicit temporary degraded service contract). 3. Confirm immutable model artifacts, tokenizer/adapters, routing, auth, retrieval state and versioned configuration are available in each approved region. 4. Test routing failover, unhealthy model workers, GPU cold starts, quota limits and shared external dependency failures. 5. Test long lived streaming sessions, idempotent retries and tool call side effects across regional cutover. 6. Confirm telemetry distinguishes original requests, backend retries, errors, p95 TTFT/ITL, full stream completions and SLO goodput per region . 7. Rehearse failback gradually; do not immediately move all requests to the recovered region before it is warmed and verified. 8. Validate data residency and security rules in routing, backups and cross region observability. Interview ready answer: “I would decide whether the workload needs warm standby or active active based on a regional loss RTO and explicit SLOs. Then I'd size every surviving region for the total eligible failover demand using measured per replica SLO goodput , not normal regional load. I'd keep immutable model artifacts and critical state available regionally, prewarm GPU capacity or document reduced service during scale up, and route only to ready, compatible, authorized regions. Finally, I'd drill full region evacuation with cold caches, live streams and retries and report actual SLO restoration time and RPO.” Research references: AWS Prescriptive Guidance — Multi Region Architecture AWS Well Architected — Disaster Recovery Options Amazon Route 53 — Active Active and Active Passive Failover Google Cloud — Serve Traffic From Multiple Regions NVIDIA Dynamo — Overall Architecture
How do you optimize LLM inference cost using GPU reservations, spot capacity, and workload-aware scheduling without violating SLOs?
Cost aware LLM inference scheduling places requests on the cheapest eligible and sufficiently reliable GPU capacity while maintaining each workload's latency, correctness, data residency and availability objectives. The foundation is a protected serving baseline for latency sensitive requests, with flexible surplus capacity for deferrable or interruption tolerant workloads. Buying the cheapest GPU hour is not necessarily cheapest: interrupted generations, failed SLOs, cold start delays and wasted prompt tokens can make the cost per successfully completed SLO compliant request worse. This lesson follows Question 260 (GPU sizing and unit economics), Question 251 (tenant fairness), Question 261 (autoscaling), Question 263 (stream draining), and Question 272 (regional capacity planning). Here the new skill is combining purchasing models with workload class placement, interruption risk, and explicit economic/SLO constraints . 1. Four distinct cloud purchasing concepts On demand: Pay for instances as needed; useful for elastic baseline/overflow when available. On demand is not by itself a guarantee that scarce GPU instances can be launched instantly. Capacity Reservations: Reserve specified eligible compute capacity in a defined zone, helping ensure the necessary instances can be launched. This is a capacity assurance mechanism , not inherently a discount; unused reserved capacity may still be billable. Actual product options and terms vary. Savings Plans / Reserved Instances / commitments: Reduce eligible compute billing under a usage/configuration commitment. A Savings Plan does not by itself reserve GPU capacity . Understand term, region/family flexibility, underutilization, and the effects of commitments when architecture changes. Spot / preemptible capacity: Discounted spare capacity that the provider can reclaim. Availability and interruption behavior are variable. Excellent for checkpointable, repeatable batch work; risky as the sole pool for long, real time streaming sessions. AWS specifics: AWS Spot normally provides a two minute interruption notice ; an earlier rebalance recommendation may be available, but no two minute window is a promise that all generations will finish. AWS separately documents Capacity Reservations and Savings Plans. Google Cloud differs: Spot VMs can be configured for a 120 second advance preemption notice , while the default is 0 second dedicated notice followed by a best effort shutdown of up to 30 seconds . Do not apply the AWS two minute rule to every provider. 2. Classify requests before choosing a GPU pool Tier A — interactive with strict SLO User facing chat, tool using assistants and interactive coding need low TTFT/TPOT, predictable success and stable streaming. Prefer adequately sized, warmed noninterruptible/contracted or reserved capacity for the protected baseline. Allow Spot only when failure is explicitly tolerable and reliable fallback/readmission is tested. Tier B — elastic but still user facing Lower priority chat or asynchronous APIs may tolerate longer deadlines or model fallback. Route to cost efficient GPUs only when measured queue and token workload indicate sufficient headroom. If a risky pool becomes unhealthy, reject/defer or fall back within the stated policy. Tier C — deferrable, restartable batch Offline summarization, embeddings/backfills where applicable, evaluation sweeps or bulk document generation can often tolerate interruption. Favor Spot/preemptible resources with checkpointing, idempotent job IDs, retry budgets and a deadline aware fallback to on demand. Tier D — highly sensitive or constrained Tenant isolation, strict residency, model licensing, regulated workflows or minimum accuracy requirements may prohibit some pools, regions or fallback models regardless of cost. The scheduler should decide based on workload eligibility + predicted token work + SLO deadline + actual ready capacity , not on cloud price alone. 3. Beginner friendly numerical example: two capacity choices Assume a production workload receives 20 interactive requests/s and 8 batch requests/s . A compatible one GPU serving replica safely handles 5 requests/s for this particular measured mix and SLO , after planned per replica headroom. Illustrative GPU prices: On demand GPU: $3.00/hour . Spot GPU: $1.00/hour . All quantities and prices are educational assumptions, not current cloud quotes. Plan A — all on demand: \ceil((20 + 8)/5) = 6 GPUs\. Hourly GPU cost = \6 × $3 = $18/hour\. Plan B — workload separated pools: Interactive baseline: \ceil(20/5) = 4\ on demand GPUs. Batch pool: \ceil(8/5) = 2\ Spot GPUs. Hourly GPU cost = \4 × $3 + 2 × $1 = $14/hour\. Nominal savings = \($18 − $14)/$18 ≈ 22.2%\. Important assumptions: the model and SLO safe rate really are comparable across pools; jobs can use dedicated capacity without unreasonable imbalance; Spot GPUs are continuously available; the four on demand GPUs have enough protected serving headroom; and storage/network costs are excluded. In reality, long batch generations can have very different token work. Independently benchmark each tier instead of extrapolating the same 5 RPS capacity to unlike workloads. 4. Why Spot interruption can erase a discount Continue the example: batch pool contains two Spot GPUs costing $2/hour , but preemption makes only 80% of attempted batch work yield useful successfully finished output (after accounting for wasted work and before any retry). If throughput is otherwise the same, the naive effective GPU cost per successful unit scales roughly as: \$2 / 0.80 = $2.50\ per equivalent hour's successful work, not $2 . The formula is a simplified normalization; true cost also includes the time of retries, checkpoint writes, cold starts, duplicate tokens, quality verification and fallback capacity. A real system should compute: \effective unit cost = (all billed GPU hours + retry/checkpoint/storage/network + operational overhead) / successfully completed eligible work\ For an online SLO, replace the denominator with successfully finished SLO compliant requests . Work rejected, interrupted or timed out must not be silently excluded from availability reporting. 5. Capacity floor and burst protection Use measured goodput per replica from realistic load tests: 1. Let \D interactive\ = forecast peak interactive offered RPS (or a calibrated token work rate). 2. Let \c protected\ = sustainable SLO compliant rate per eligible protected replica. 3. Minimum protected replicas \R = ceil(D interactive / c protected)\ with additional requirements for N−1 failures, zoning and burst headroom. 4. Use discounted/interruption tolerant capacity only for demand that can be deferred or safely displaced. 5. Allow overflow to protected capacity only if interactive headroom remains ; never let a cheap batch surge consume interactive queue/KV budgets. 6. Reserve or prewarm capacity as required by actual model loading time. A GPU available on paper but still loading weights is not ready service capacity . Failure case: if the four on demand replicas in the worked example each handle five RPS, interactive traffic of 20 RPS fully consumes their planned safe rate. Therefore there is no guaranteed protected slack for a lost Spot batch pool or a simultaneous replica failure. The plan is financially attractive only if batch work can wait and the strict interactive tier's additional reliability policies are addressed. 6. Workload aware scheduling policy A practical scheduler may use: Hard eligibility filters: approved region, GPU memory/model fit, hardware topology, tokenizer/adapter/version, encryption, tenant policy and tool safety. Request classification: online/async, deadline, priority, estimated prompt and output tokens, streaming flag, expected prefix reuse and cancellation semantics. Capacity checks: available ready model replicas, queue token work, active sequences, KV blocks, recent TTFT/ITL, goodput and health. Economics: effective $/GPU hour, remaining commitments, interruption risk, data egress, cold start cost and estimated retries. Routing: online Tier A to protected low latency pools; batch Tier C to cheaper interruptible pools subject to deadline and progress checkpoints. Escalation: nearing deadline → promote eligible batch requests to reliable capacity only if budget and protected workload headroom permit; otherwise return an explicit deferral/failure status. Never rely on an uncalibrated prediction of output length as a hard admission fact. Output tokens are uncertain; use conservative estimates, maximum generation limits and ongoing queue signals. 7. Kubernetes placement example Create appropriately labeled, separately managed protected and spot GPU node pools. Enforce the intended selection using node affinity/selector plus resource requests and, where appropriate, taints/tolerations. Illustrative batch Job template fragment: \\\yaml spec: template: spec: restartPolicy: Never nodeSelector: capacity class: spot tolerations: key: "capacity class" operator: "Equal" value: "spot" effect: "NoSchedule" containers: name: batch inference image: registry.example.com/approved llm job:PINNED DIGEST resources: limits: nvidia.com/gpu: 1 \\\ This is a conceptual manifest, not a complete deployment : your cluster must actually apply the label/taint, provide GPU scheduling/device plugins, use an authentic digest and configure the full Job, checkpoint storage and security. A toleration only allows scheduling ; it does not force placement. The node selector provides the strict class match in this example. Do not schedule critical control plane components exclusively on interruptible nodes. 8. Interruption handler and state safety 1. Monitor the correct provider specific preemption signals and capacity risk recommendations. 2. Immediately stop new assignments to a threatened worker. 3. For a long batch generation, write recoverable progress and a durable idempotency/ownership record at a safe application boundary, if the job supports it. 4. Attempt to drain active streams within the actual remaining termination period; classify unfinished streams as incomplete rather than successful. 5. Requeue only jobs whose effects are idempotent or explicitly reconciled. An LLM decoder's KV state on a lost GPU normally does not survive preemption . 6. Restart from an approved checkpoint or from earlier application level state. Exact token for token continuation is not guaranteed, especially with stochastic generation. 7. Back off retries and avoid stampeding the model object store after a large Spot reclamation. The interruption warning may be absent, delayed or too short for long generations. Never design correctness around guaranteed notice delivery. 9. Cost aware autoscaling and purchasing Baseline commitments: use historical stable usage and forecast confidence to decide what fraction of GPU hours warrants long term discounted commitments. Excess commitments still cost money when traffic shrinks or models change. Capacity reservations: consider when GPU scarcity threatens a strict RTO/SLO; separately evaluate whether discount instruments apply. Autoscale batch pools: use job queue depth and estimated queued token work rather than interactive GPU utilization. Scale down empty interruptible capacity promptly after safe checkpoint/termination. Model aware packing: schedule compatible inference jobs to avoid loading too many copies of an enormous checkpoint, while protecting tenant isolation and avoiding KV/OOM contention. Heterogeneous hardware: benchmark equivalent model quality and SLO success on each GPU SKU; the lowest $/GPU hour may have the highest $/successful request. Multi region: the cheapest geographic pool may violate data residency or add enough latency/egress cost to fail the workload's objective. 10. Production incident: batch jobs evict live chat capacity Scenario: A service puts both user facing chat and offline summarization on a shared Spot GPU pool. A Spot interruption terminates two workers during a burst. Chat completions stop midstream, the remaining workers receive retries, queue wait spikes, and p95 TTFT increases from 0.8 to 4 seconds . The lower hourly price does not compensate for failed customer SLOs. Root cause: no protected Tier A capacity; the scheduler ignored workload deadline/restartability and treated Spot GPUs as equivalent to stable warmed replicas. Fix: Move the minimum interactive serving footprint to protected, warm GPUs with explicit N−1 headroom and admission budgets. Run only interruption safe batch work on Spot; isolate queues and preemption handling. Implement provider interruption detection, bounded retry and job checkpoint/idempotency. Calculate $ per SLO compliant user request and $ per verified completed batch job , not aggregate GPU rate discounts. Prevention: run synthetic full Spot pool reclamation drills while interactive traffic stays at peak; verify no critical streaming tier depends on ephemeral capacity, and confirm the job queue recovers without duplicate side effects. 11. Monitoring and common mistakes Track separately by capacity class : Billed GPU hours, unused commitments/reservations, actual Spot prices, interruptions/hour, job retries and lost compute work. Ready serving replicas, time to re acquire GPUs, checkpoint restore/load time, queued token work and batch job age. Original request p95/p99 TTFT, TPOT, error/rejection/stream truncation, joint SLO goodput and cost per good request. Per tier quality and model version, tenant isolation and data residency policy violations. Avoid: Assuming Savings Plans imply a guaranteed GPU supply. Counting merely requested or still warming Spot GPU instances as ready capacity. Treating every batch request as safely restartable without durable progress and idempotency. Making Spot the only production tier for sub second streaming SLOs. Mixing user facing and batch request classes without scheduling/fairness isolation. Treating instance hour discounts as proven unit cost savings without lost work, cold starts and failure costs. Depending on an AWS specific two minute notice across all cloud providers. 12. Interview ready answer “I'd benchmark real SLO goodput and create a protected GPU baseline for interactive traffic, using reservations where capacity availability warrants it and commitments only when the expected usage is steady. I'd schedule checkpointable batch inference on Spot/preemptible GPUs with provider specific interruption handlers, idempotent retries and separate queue budgets. The optimizer should consider actual ready capacity, token work cost, deadlines, quality and residency rather than hourly price alone. I'd validate savings as total dollars divided by successfully completed SLO conforming requests and run interruption drills before production.” Research references: AWS — EC2 Purchasing Options AWS — Spot Instances and Interruption Notices AWS — Capacity Reservations Google Cloud — Spot VM Preemption Kubernetes — Taints and Tolerations GKE — Spot VMs with GPU Node Pools
How do you design deadline-aware batch inference scheduling and checkpoint recovery on interruptible GPUs?
Deadline aware batch inference scheduling assigns large, deferrable LLM workloads to suitable GPU pools while ensuring jobs have a realistic chance of finishing before their deadlines . When cheaper Spot/preemptible GPUs disappear, workers should recover from durably committed job progress , avoid duplicate side effects, and escalate only work whose deadlines require more dependable capacity. This lesson follows Question 273 (GPU purchasing options and online/batch pool separation), Question 253 (online request deadlines/cancellation) and Question 251 (tenant fairness). Here the focus is on batch job feasibility, remaining work estimates, durable progress manifests, preemption safe retries and deadline driven escalation —not streaming failover or raw cost optimization. 1. Understand the different deadlines Job deadline: the externally agreed time by which all required outputs must be complete and validated, for example midnight UTC. Queue deadline: latest safe time to acquire enough GPUs and start computation so that the overall job remains feasible. Task timeout: upper bound for one document/shard attempt. Task timeout is not necessarily the overall job deadline. Recovery reserve: time budget for expected interruptions, checkpoint restore, model startup, output validation and object store consistency. Hard versus soft deadline: a hard deadline may require reserved/on demand GPUs, workload shedding and explicit failure; soft deadlines may trade increased waiting or cost for best effort completion. Important: Kubernetes Job \activeDeadlineSeconds\ is an elapsed runtime cap for a Job, not an absolute wall clock business deadline. If exceeded, Kubernetes terminates running Pods and marks the Job failed; the scheduler must translate business due times into appropriate orchestration policies and requeue decisions. 2. Beginner friendly feasibility calculation Suppose a batch contains 12,000 documents , each requiring on average 2 seconds of measured GPU service for the approved model and input/output distribution. Total nominal GPU work: \12,000 × 2 = 24,000 GPU seconds\. With 4 comparable GPUs at ideal full effectiveness: \24,000 / 4 = 6,000 seconds = 100 minutes\. At an observed end to end 80% effective processing efficiency (startup, packing, queue, variable lengths and write overhead): \6,000 / 0.8 = 7,500 seconds = 125 minutes\. Add 15 minutes for a conservative measured failure/restore and quality validation allowance: 140 minutes . If the deadline is 150 minutes away , only 10 minutes of slack remain. If admission predicts 20 minutes of waiting for Spot GPUs, the job is no longer feasible under this configuration because \20 + 140 = 160 150\ minutes. The scheduler should increase reliable capacity, change the approved execution plan, or report infeasibility , not quietly admit it as an on time job. All throughput and recovery numbers are illustrative. In real LLM batches, long prompt and generated output lengths produce heavy tailed task durations; use workload calibrated upper quantile estimates and re estimate remaining work throughout execution. 3. Define slack, predicted finish time and scheduling priority For a job \j\: emaining work j\: estimated prompt/decode token work, corrected using measured service rates per model and GPU SKU. \estimated duration j(pool)\: remaining processing time given available ready GPUs , expected batching efficiency, checkpoint/restoration and failure risk. \slack j = deadline j now estimated duration j\. Negative slack means the job is not forecast feasible under the current placement. Do not claim it is impossible under all placements; test larger or faster eligible capacity. A practical policy combines: 1. Hard constraints: authorized model/adapter, data residency, GPU memory/model fit, deadline class, quality and dataset access. 2. Admit only with a plausible completion plan: include queue wait and checkpoint reserve. 3. Prioritize urgent feasible work: use earliest deadline first or least slack time first with fairness and starvation limits . Neither policy automatically guarantees all deadlines on multiprocessor, non preemptible, variable length LLM workloads. 4. Pack work into manageable shards: group similar token lengths for efficiency but keep chunks small enough to recover within the deadline. 5. Escalate by slack: when predicted remaining completion time approaches deadline, move checkpointable tasks from Spot to reliable capacity if authorized GPU budget and availability allow. 6. Reject/defer transparently: if no compliant capacity can meet the deadline, mark the job at risk/failing and notify upstream workflows. 4. Build a durable work and checkpoint model Recommended unit of recovery: a completed input document or microbatch , not the ephemeral decoder KV cache state. For each batch job persist: Immutable job ID , model weight digest, tokenizer/chat template version, prompt/template version, generation parameters, input dataset snapshot and output schema version. Partition/shard IDs and a manifest enumerating every required input record. Status per shard: pending, leased, succeeded, retryable failure, terminal failure; bounded lease expiry and attempt count. Durable output URI + checksum for each committed shard, plus quality/schema checks and audit metadata. Checkpoint manifest version and timestamp; store in persistent object storage or database outside the Spot node. Exactly once effects need an application protocol. Kubernetes Jobs and queues usually provide at least once execution in failure cases. Two workers can try the same shard. Prevent duplicate published outputs using an atomic conditional claim/commit, durable idempotency keys and a single authoritative manifest. Do not assume one Pod equals exactly one execution. 5. How to checkpoint safely A reliable shard process: 1. Claim a shard using an expiring lease or transactional compare and swap (CAS) with a unique attempt ID . 2. Read the immutable input snapshot and verify approved model/configuration. 3. Generate outputs and validate schema, correctness constraints and safety requirements. 4. Write results to an attempt scoped temporary object , with integrity checksum. 5. Atomically mark that shard succeeded in the authoritative manifest if this attempt still owns the lease; publish the committed output reference. 6. If the manifest commit fails, treat the attempt as uncommitted and safely discard or garbage collect its temporary object. 7. On worker loss, reclaim expired leases and retry unfinished shards up to the job's deadline and retry budget. Object stores do not universally support atomic cross object transactions. Use the database/manifest CAS as the commit point , and design readers to ignore uncommitted objects. A temporary output existing in storage does not mean the corresponding shard has completed. Caution: token level decoder checkpoints require exact KV/cache and RNG/runtime compatibility, which most serving stacks do not promise across GPUs or engine versions. Replaying a shard from the prompt is often safer, though stochastic decoding can yield a different completion. Restore from stable committed work , not an imaginary portable KV snapshot. 6. Preemption notices and emergency checkpointing For AWS EC2 Spot: A rebalance recommendation may arrive before a two minute interruption notice, but is not guaranteed to arrive first. A normal Spot interruption notice typically provides approximately two minutes before interruption; it can be unavailable or late. Correctness must survive abrupt termination without notice . On notification, stop leasing new shards immediately; complete only those likely to finish within the remaining budget, commit finished outputs, and relinquish leases or allow expiry for unfinished work. Do not start a costly checkpoint operation that cannot finish or corrupt a durable manifest if interrupted mid write. Keep completed outputs in a persistent store, not only on instance local ephemeral storage. AWS recommends small tasks and durable checkpoints, and explicitly suggests testing sudden terminations. Cloud providers have different notice semantics ; never hardcode AWS behavior into a multi cloud scheduler. 7. Checkpoint interval and wasted work trade off Suppose a running worker takes 40 seconds to finish one document/shard and checkpoints only after each 10 shard microbatch : Checkpoint interval = \10 × 40 = 400 seconds\ plus commit overhead. If an interruption occurs uniformly within that interval, expected lost compute is about 200 seconds (half the interval), assuming no prior durable intermediate commit. If it checkpoints after every 2 shards , interval = \2 × 40 = 80 seconds\, and expected lost work falls to 40 seconds . More frequent commits incur more object storage/database writes and CPU time; fewer commits increase expected recomputation after failure. An approximate checkpoint optimization model with fixed checkpoint overhead \C\ seconds and mean time between interruptions \M\ seconds is \overhead fraction ≈ C/T + T/(2M)\ for checkpoint interval \T\, under simplifying independent failure and constant work assumptions. Minimizing gives \T ≈ sqrt(2CM)\. This is only a heuristic: actual LLM output lengths, shard sizes, checkpoint cost and nonstationary Spot interruptions can violate its assumptions. Practical choice: commit on stable item/microbatch boundaries, then use a load test and interruption simulation to choose an interval; do not blindly checkpoint every emitted token. 8. Kubernetes and Kueue orchestration Kubernetes Indexed Jobs assign stable completion indexes to partitions and support \backoffLimitPerIndex\ (stable in current Kubernetes versions). With estartPolicy: Never\, a failed shard index can retry without restarting all successful indexes. Still, the application must provide durable output/idempotency , because duplicate execution can occur. Illustrative Job configuration: \\\yaml apiVersion: batch/v1 kind: Job metadata: name: llm batch inference spec: completions: 100 parallelism: 8 completionMode: Indexed backoffLimitPerIndex: 2 template: spec: restartPolicy: Never nodeSelector: capacity class: spot containers: name: worker image: registry.example.com/llm batch@sha256:REPLACE WITH REAL DIGEST resources: limits: nvidia.com/gpu: 1 \\\ Do not deploy this unchanged: replace the example image/digest with a real approved immutable image, verify labels/GPU resources, configure checkpoint storage and IAM, and decide whether a Spot only placement is acceptable for the job deadline. A Pod index is a partition identity , not a durable completed result guarantee. Kueue provides quota/resource flavor admission for Kubernetes batch jobs. Queueing priority, borrowing and topology aware GPU scheduling can help allocate scarce resources, but Kueue does not automatically calculate LLM specific deadlines, estimated remaining generated tokens or portable KV checkpoints . Implement those in a separate workload planner or integrate them with custom admission policies. 9. Recovery algorithm after an interruption 1. Receive preemption signal or detect missing worker heartbeat/expired shard lease. 2. Stop assigning new work to a departing worker and persist any fully completed shard outputs. 3. Reconcile in progress attempts against authoritative manifest and object storage. 4. Return uncommitted shards to pending only after fencing old lease holders, to avoid late duplicate commits. 5. Recompute emaining work\, estimated GPU service rate and \slack\ for the deadline. 6. If enough Spot capacity and slack remain, retry on an eligible Spot pool with backoff and rate limits. 7. If slack is insufficient, escalate to pre approved on demand/reserved capacity, reduce optional work only if contract permits, or explicitly declare the deadline at risk. 8. Once every required index is committed and validated, atomically mark the job complete; only then notify consumers or downstream workflows. Be careful with external side effects : if generated outputs trigger emails, purchases or database mutations, process them in a separate idempotent downstream step, not directly from retryable inference attempts. 10. Production incident: “99% finished” but misses deadline Scenario: A document summarization batch of 50,000 inputs runs cheaply on Spot GPUs. A preemption near the deadline destroys several large uncheckpointed work partitions. Kubernetes restarts Pods, but earlier outputs are mixed with retries and duplicate writes; the batch misses its deadline despite the dashboard showing 99% of GPU work processed. Root causes: Progress was counted from attempts rather than durably committed outputs. Long partitions made a single interruption erase many minutes of GPU work. The scheduler ignored shrinking deadline slack and continued waiting for cheap Spot GPUs. Output writes were not fenced by per attempt ownership. Fix: split into manageable durable shards, commit each through a manifest CAS, monitor committed/required output counts, and add deadline aware promotion to reliable GPUs. Configure retry budgets and backoff, and simulate interruptions at the worst possible commit boundary. Prevention: require deterministic or acceptance validated outputs, end to end checksum verification, interrupted worker experiments and a completion audit before exposing “Done” status. 11. Metrics and common mistakes Monitor: Job deadline and slack distribution, predicted finishing time, queued work in input/output tokens, and ready GPU capacity per purchasing class. Committed shards / total required, active leases, oldest outstanding shard, per shard attempts, terminal failures and late completions. Checkpoint/manifest write latency, average redo compute after interruption, failed CAS updates, duplicate attempt suppression and output integrity. Spot preemption and unavailable GPU rate, restart/warmup duration, reliable capacity escalation cost and late job frequency. Effective dollars per validated, on time completed batch job , not only dollars per GPU hour. Avoid: Treating \activeDeadlineSeconds\ as an absolute business due timestamp. Claiming zero data loss from ephemeral KV cache state. Publishing a result merely because temporary objects were written. Retrying non idempotent tool actions on every duplicate worker attempt. Prioritizing solely by earliest deadline without token work estimates and fairness. Assuming interrupt notice is always delivered or sufficiently long. Counting Kubernetes Job \Succeeded\ as proof all model outputs passed application validation. 12. Interview ready answer “I'd model each batch job's deadline, estimated remaining token work, ready GPU processing rate, recovery reserve and slack. I'd admit only feasible jobs, prioritize urgent work with fairness, run restartable shards on Spot when slack permits, and escalate to reliable capacity before a deadline becomes unreachable. Each shard would commit validated output through a durable, lease fenced manifest so duplicate retries are safe. I'd test abrupt interruption, checkpoint commit races and final output completeness, then optimize dollars per verified on time job rather than spot hour discounts.” Research references: Kubernetes — Jobs, Indexed Completion, Retry and Deadlines Kubernetes — Pod Failure Policy Kueue — Workload Admission AWS — Prepare for Spot Interruptions AWS — EC2 Spot Best Practices
How do you optimize GPU utilization across mixed interactive and batch LLM inference workloads while preserving latency isolation?
Mixed workload LLM inference scheduling increases useful GPU work by combining latency sensitive interactive requests with throughput oriented batch requests— without letting batch prefill, long decodes, or KV cache pressure violate interactive service level objectives (SLOs) . The challenge is to reclaim genuinely idle GPU capacity while providing a measurable, enforceable isolation boundary. A high aggregate GPU utilization percentage is not success if chat users experience worse time to first token (TTFT), inter token latency (ITL/TPOT), errors or truncated streams. This lesson is a production design continuation of Question 220 (why chatbot and batch GPU demands differ), Question 251 (tenant priority/fairness), Question 273 (Spot/reserved GPU purchasing) and Question 274 (batch deadlines/checkpointing). Here the specific focus is GPU time and memory isolation, token aware co scheduling, borrowing idle capacity, eviction semantics and measurable safe utilization . 1. Why interactive and batch work interfere Interactive chat: often short or bursty prompts, unpredictable arrivals, streaming responses and strict TTFT/ITL targets. It needs quick admission and reliable KV memory headroom. Batch summarization/research: may generate many tokens from large prompts, tolerate higher queue delay, and prefer large batches or sustained GPU occupation to maximize tokens per GPU hour. Prefill versus decode: prompt prefill tends to be compute intensive; decode frequently depends on memory bandwidth/KV cache access. Mixing phases intelligently can help device efficiency, but very large prefills can delay already streaming users. Long context requests: consume more KV blocks and may cause preemption/recomputation even if the immediate GPU utilization looks moderate. Shared network/CPU/storage: two pools on different GPU allocations can still interfere through checkpoint downloads, tokenizer CPU, router queues or KV transfer links. Output length uncertainty: estimated tokens are planning signals, not guarantees. Enforce explicit request budgets and monitor actual progress. 2. Choose the isolation architecture Option A — separate GPU serving pools (strong baseline) Maintain an interactive pool sized for forecast SLO demand, burst headroom and required replica failure tolerance. Keep a separately admission controlled batch pool , potentially on cheaper or interruptible GPUs where jobs are restartable. Router sends each class only to approved compatible model/version/region pools. This is easiest to reason about and protects chat from batch related OOM and scheduling pauses. Cost: interactive GPUs may sit partly idle when chat demand is low. Option B — shared serving engine with priority aware token scheduling (controlled borrowing) Use a serving engine supporting per request priorities, maximum token budgets, preemption and KV aware admission. Let batch work borrow surplus engine capacity only while observed interactive latency and queue age remain healthy. Protect interactive decode steps, limit long prefill chunk size and cap active batch sequences and KV block use. Risk: logical scheduling does not guarantee strict GPU memory/compute isolation; an oversized batch may already hold KV state or occupy kernels. Option C — hardware partitioned GPU resources On supported NVIDIA GPUs, MIG can partition hardware into GPU instances with isolated resources, but each slice has limited HBM and compute; many large LLMs will not fit. GPU time slicing may improve access for smaller models/jobs, but NVIDIA explicitly documents no memory isolation and no throughput guarantee with time slicing. It is not a hard SLO boundary. Multi Process Service (MPS) is another sharing approach with its own support and isolation constraints. Benchmark the exact configuration and ensure tenant security requirements. For strict isolation of large models, whole GPU or whole replica pools are usually a safer starting point. 3. Beginner friendly numerical example: utilization versus SLO Assume four equivalent GPU replicas are deployed: Baseline interactive demand uses 40% of the fleet's GPU capacity during quiet periods. Batch backfill can use idle capacity but has no user facing TTFT requirement. Interactive SLO: p95 TTFT ≤ 1 second , p95 ITL ≤ 50 milliseconds and request success ≥ 99% . The results below are illustrative controlled measurements , not published GPU benchmark claims. A. No batch borrowing: GPU active utilization = 40% . Interactive p95 TTFT = 500 ms ; p95 ITL = 35 ms ; success = 99.9% . B. Unrestricted batch borrowing: GPU active utilization = 90% . Interactive p95 TTFT = 1,800 ms ; p95 ITL = 75 ms ; success = 98% . Conclusion: the fleet is busier, but the interactive service violates all three objectives; calling this an optimization would be misleading. C. Token aware, limited borrowing: GPU active utilization = 72% . Interactive p95 TTFT = 700 ms ; p95 ITL = 42 ms ; success = 99.5% . This configuration satisfies all three example goals, while increasing GPU utilization by 32 percentage points versus the quiet baseline (a relative 80% increase , \(72−40)/40 = 0.8\). Do not choose 72% as a universal target. Safe GPU occupancy changes with model, token length distribution, active sequences, KV memory, quantization, concurrent tenants and underlying hardware. A setting that passed at 40 interactive requests/s may fail during peak demand. 4. A simple capacity borrowing policy Define: \L i\: observed interactive offered token work rate and active/request backlog. \C safe\: load tested safe capacity under the real interactive mix (not theoretical GPU FLOPS). \H\: reserved headroom for interactive bursts, model warmup and worker failures. \B budget\: permitted batch token, queue and KV budgets at the current condition. A conceptual controller: 1. If interactive p95 TTFT and ITL have healthy margin, queue age is low, KV occupancy below the tested pressure point, and spare capacity exists: allow limited batch tokens/requests. 2. If interactive queue token pressure rises or SLO burn increases: immediately stop new batch admissions, reduce/stop future batch prefill chunks and drain or preempt eligible batch work. 3. If KV pressure approaches unsafe levels: reclaim batch capacity only using the serving engine's supported preemption/cancellation mechanisms. A reclaimed batch KV cache may require recomputation; never assume free/instant migration. 4. When interactive conditions recover for a sustained quiet interval: slowly reopen batch borrowing with hysteresis and a cap. 5. Always enforce per tier maximums: batch concurrency, pending token work, allowed output length, maximum memory/KV allocation and deadline/retry budget. This is a conceptual policy , not a documented built in vLLM autoscaler. GPU utilization alone should not drive the decisions. 5. Understand vLLM scheduling controls Current vLLM scheduler documentation describes: FCFS and priority policies: request priority is a numeric field; lower numeric values receive higher priority when the priority policy is configured. Priorities have semantics within the engine, not automatically across every router or model replica. \max num batched tokens\: caps a token budget per scheduling step. It influences how many prefills can mix with ongoing decodes and can materially change TTFT versus ITL. \max num seqs\: bounds active sequences. More concurrency can improve throughput but also increases KV pressure and may harm tail latency. Chunked prefill: splits oversized prefills into smaller pieces to interleave with ongoing generations, improving the chance to protect decode responsiveness. Preemption: when KV cache is constrained, the scheduler may preempt work; in vLLM V1 this commonly requires recomputation. Even if a batch request is preemptible, its lost GPU work is a cost . The vLLM V1 scheduler is token budget based and does not require a strict global alternation between separate prefill and decode stages. Avoid presenting prefill/decode batching as permanently independent fixed phases. Version caveat: vLLM scheduling defaults and optional flags evolve. Verify the deployed version, supported request priority field and model serving endpoint behavior before copying flags. 6. Protect latency with two layers Layer 1 — routing and pool level safeguards Assign a minimum ready interactive replica pool and enforce per class admission before the GPU scheduler. Reserve an interactive share of compatible GPUs or token work; stop scheduling new batch tasks when forecast peak load approaches. Separate queue budgets, auth/tenant policies and timeouts. Batch deadlines should not override a customer's latency critical request. Prefer quick start, warm interactive capacity so scaling and routing changes do not create cold start latency. Layer 2 — per engine safeguards Keep ongoing interactive decode requests responsive with token budget and supported priority policies. Break long prefill batch requests into bounded chunks; tune chunk sizes against representative short and long prompts. Set batch concurrency/KV thresholds based on observed preemptions and memory. When the engine cannot offer sufficient class isolation, move batch work to another replica/pool rather than assuming priorities are enough . Priority is not the same as GPU kernel preemption: a high priority request cannot necessarily interrupt an already running CUDA kernel or immediately reclaim live KV state. 7. Production incident: 95% GPU utilization, poor chatbot experience Scenario: A team colocates document summarization jobs with an interactive chatbot because the GPUs appear underutilized at night. When batch requests with 32k token prompts start, GPU usage reaches 95% . Streaming chat p95 ITL worsens from 40 ms to 110 ms and p95 TTFT from 600 ms to 2.4 seconds . Root cause debugging: 1. Slice TTFT, ITL and queue wait by interactive versus batch , request length and worker. 2. Trace large batch prefills through scheduler steps; they fill token budgets and delay interactive prefill admission. 3. Inspect KV cache occupancy and preemption/recomputation counts; long batch generations leave less headroom for chat. 4. Verify whether the router admits batch work onto the very replicas required for critical user streams. 5. Compare a no batch control replay to progressive borrowing rates. Fix: partition an interactive baseline; bound batch prefill chunks and active KV work; apply engine aware priorities/queues; stop batch admission when interactive queue/SLO pressure increases; ensure unfinished batch work is checkpointable at shard boundaries if moved to other workers. Prevention: run mixed class load tests at normal and peak interactive demand, and enforce per class SLO gates and safe borrowing limits before enabling production co location. 8. Evaluate efficiency correctly Track three groups of metrics: Customer protection: interactive p95/p99 TTFT, ITL, successful stream completion, error rate and SLO compliant goodput . Batch usefulness: completed and validated batch items/hour, checkpoint redo work, job lateness, preemptions, effective GPU cost per completed batch item. Resource efficiency: GPU utilization, prompt/output tokens per second by workload class , KV cache occupancy, memory pressure, preempted/recomputed tokens, CPU/network bottlenecks, total GPU hours and cost per accepted/successful outcome. Report outcomes under equivalent offered interactive traffic, with bursty arrivals and realistic prompt/output length distributions. Do not compare “utilization before” and “utilization after” if chat load differs. An appropriate optimization objective is: \maximize validated batch throughput per incremental GPU hour\ subject to interactive TTFT/ITL/success SLOs, tenant safety and batch deadline constraints. Batch speedups that cause user visible SLO failures are not feasible solutions , regardless of aggregate throughput gains. 9. An engineering rollout checklist 1. Establish the interactive only p95/p99 baseline and forecast peak plus failure headroom. 2. Load test dedicated pools to estimate class specific resource consumption. 3. Define hard isolation requirements (tenant, model, region, GPU memory, KV boundaries and support for shared hardware). 4. Start with dedicated pools; add controlled borrowing on a small portion of eligible replicas. 5. Increment batch concurrency or token budget in measured steps, retaining enough sample volume for tail latency. 6. During each step, inspect interactive TTFT/ITL, queue token age, KV occupancy and preemptions. 7. Require a fast batch admission stop switch ; validate that stopping new batch work actually restores interactive SLOs within the required time. 8. Test an interactive burst and one GPU/replica failure while batch runs. 9. If batch jobs are interrupted, use durable shard checkpoints and idempotency from Question 274. 10. Continuously report incremental batch completion versus its impact on interactive SLO compliant goodput and total cost. Limitations: queue priorities do not guarantee strict compute isolation; hardware sharing may be unavailable on the chosen GPU profile; heavy KV reuse and large dynamic context lengths complicate fairness; and global route level SLO controls require integration beyond one local engine. 10. Interview ready answer “I'd protect a ready interactive GPU baseline first and benchmark real p95 TTFT/ITL and goodput under peak traffic. Batch inference can use separate cheaper pools by default, then borrow tested surplus capacity via token aware queues, bounded chunked prefills, priority scheduling and per class KV/concurrency limits. If chat queue age or SLO burn rises, batch admissions stop and the system safely drains/preempts eligible batch work. I'd avoid using GPU time slicing as a hard isolation guarantee and optimize verified batch completions per GPU hour subject to user facing latency and reliability targets.” Research references: vLLM — Scheduler Configuration and Priority Policy vLLM — Architecture and Token Budget Scheduler vLLM — Agentic Mixed Traffic Scheduling (September 2026) NVIDIA — GPU Time Slicing and Its Isolation Limits NVIDIA — MIG Partitioning NVIDIA Dynamo — Inference Scheduler and GPU Fleet
How do you design token-aware GPU quota allocation and chargeback across teams and tenants in production LLM inference?
Token aware GPU quota allocation controls how much LLM inference work each team or tenant is allowed to submit, run concurrently and consume over time. Chargeback attributes the actual cost of shared serving infrastructure to the teams that benefit from it; showback reports those costs without necessarily invoicing them. In a shared vLLM or comparable serving fleet, token accounting must be combined with GPU capacity, latency, KV cache pressure, workload priority and accurate completion records . A count of requests alone is not enough: one 200 token chat and one 80,000 token long context request have very different serving costs. This lesson continues Question 251 (tenant fairness and priority), Question 260 (GPU cost per SLO compliant request), Question 267 (inference observability), and Question 275 (mixed interactive/batch GPU scheduling). Its distinct contribution is quota policy, precise metering, usage reconciliation, fair shared infrastructure cost allocation and financial governance . 1. Separate quota, admission control, rate limit and chargeback Entitlement/quota: a tenant's contracted or organizational allowance, such as 10 million input tokens/day, 2 million output tokens/day, 25 concurrent requests and four reserved GPU slots. Rate limits: protection against bursts: requests/minute, estimated input tokens/minute and maximum ongoing generated token work. Admission control: decides whether a particular request fits its token budget, context length limit, concurrency, memory and deadline. Rejecting an impossible request early is better than consuming GPU work before timing out. Scheduling/fairness: orders accepted work across tenants. Fair scheduling cannot enforce a monthly budget if no accounting and quota mechanism exist. Metering: measures what actually happened: attempts, prompts, cached/recomputed tokens, generated tokens, errors, retries, GPU seconds and allocated GPU hours. Chargeback/showback: applies a published policy for distributing shared costs to cost centers or customers. Never confuse namespace GPU quotas with per token consumption. Kubernetes ResourceQuota can limit a namespace's requested extended resource (for example equests.nvidia.com/gpu\), but cannot directly meter one organization's tokens in a multi tenant shared model process . That requires an authenticated inference gateway and a ledger. 2. What makes LLM quota accounting difficult? Input versus output: input prefill and autoregressive output decode have different GPU cost profiles. Pricing both as one undifferentiated “token” may systematically cross subsidize some teams. Prefix caching: cached prompt tokens may avoid significant compute, but still consume memory and may need separate policy/pricing. The cache hit count, provider billable cached tokens and compute reuse are different facts. Long context: higher memory/KV pressure can lower safe concurrency even when request counts or output tokens remain low. Streaming and cancellation: clients may cancel after some tokens; usage must reflect observed generated work, terminal status and agreed billing rules, not assume every started request was completed. Retries: a gateway may attempt the same logical user request multiple times. Avoid charging a customer twice by accident while still accounting for the real compute consumed. Model/hardware heterogeneity: 100k tokens on a small efficient model and 100k tokens on a very large model have radically different unit costs and latency envelopes. Multi tenant isolation: trust verified auth derived tenant identity; do not allow the caller to choose an arbitrary chargeback group in request headers. 3. Define a practical hierarchical quota An organization may establish: 1. Organization budget: monthly currency spending cap or committed GPU hour entitlement. 2. Team/project quota: daily token envelopes, maximum GPU seconds or credit balance. 3. Tenant/application rate limits: input token/min, generated token/min, requests/min and concurrency. 4. Per request limits: maximum input context length, maximum output tokens, deadline, approved models and region. 5. Class limits: strict interactive requests have protected capacity; batch tasks have deferrable queue budgets and lower priority. 6. Burst rules: an agreed small token burst bucket, paid overage, or documented rejection/429 policy. 7. Safety reserve: a fraction of the whole fleet retained for baseline SLO and node failures, not sold as simultaneously guaranteed to every tenant. Use consistent hierarchical admission : a request must pass org, team, app and per request constraints. A team quota is not an implicit permission to bypass model safety, residency or hardware limits. 4. Numerical example: token quota and rate limits Suppose Team A has: Daily allowance of 2,000,000 input tokens and 500,000 generated tokens . Token throughput allowance of 50,000 estimated input tokens/minute . Output generation allowance of 10,000 tokens/minute . Maximum 20 concurrent generations . A new request contains 4,000 input tokens and sets \max tokens=1,000\ output tokens: Admission reserves 4,000 input tokens and up to 1,000 output tokens against conservative quota/rate budgets, subject to the current policy. Actual response uses 4,000 input tokens and 300 generated tokens . Once final usage is committed, refund or release the unused 700 output token reservation . The gateway must reconcile incomplete streams separately. If Team A had already used 499,000 output tokens today, reserving 1,000 leaves zero remaining headroom, so the next request requiring output must wait/reduce its declared maximum if approved or receive a clear limit response. Why reserve before generation? Without estimated output reservations, 100 concurrent requests can all pass a simple “remaining daily output tokens” check, then overspend their shared allowance as decode proceeds. Use atomic reserve/commit/release updates or an equivalent distributed quota service. Caution: max output token limit is an upper bound on generation, not a prediction of actual output. Different models may expose visible and non visible output/reasoning usage differently. Follow the deployed API's authoritative usage schema. 5. Numerical example: cost allocation with different token classes Suppose a month of self hosted inference incurs $12,000 in combined GPU infrastructure cost for one shared serving pool. Usage ledger totals: Team A: 20 million uncached input tokens , 5 million cached input tokens , 4 million output tokens . Team B: 10 million uncached input tokens , 15 million cached input tokens , 6 million output tokens . To illustrate a transparent internal weighted token policy, assume weights: Uncached input = 1 charge unit/token . Cached input = 0.2 charge unit/token . Output = 4 charge units/token . Team A weighted units: \20 + (5 × 0.2) + (4 × 4) = 37 million units\. Team B weighted units: \10 + (15 × 0.2) + (6 × 4) = 37 million units\. Total = 74 million units ; each team gets 50% × $12,000 = $6,000 under this particular policy. These weights are fictional policy choices, not a universal GPU cost model or provider prices. Real unit weights should come from measured model/GPU prefill/decode cost profiles, cache reuse, committed capacity and published contractual terms. You may choose to allocate a guaranteed reserved baseline by contracted GPU hours and only allocate incremental variable cost by token weights, rather than sharing all GPU charges purely by token volume. 6. Chargeback policy options and trade offs Option A: Per actual GPU second Use attributable compute duration and GPU device allocation where instrumentation makes it credible. Better alignment with time based infrastructure invoices. Shared batch kernels and overlapping requests make exact per request GPU seconds ambiguous; allocation must be approximate or use a documented cost model. Option B: Weighted token cost Simple, explainable: \charge = uncached input tokens × rate in + cached input tokens × rate cached + output tokens × rate out\. Different model families and hardware SKUs need different rate cards; token ratios alone do not capture every long context/low concurrency memory cost. Option C: Reserved capacity + usage Allocate contracted GPU hours, dedicated replicas or minimum guarantees as a fixed fee; attribute dynamic burst costs by metered tokens or incremental GPU work. Often fairer for teams with very strict SLO guarantees and large standing capacity requirements. Requires clear policy for idle reserved GPU time. Option D: Provider pass through For hosted inference, apply model/provider bills and actual usage reports, including cached input and output/reasoning components where supported. Do not double add cache write tokens when already included in input token categories. Pricing and usage schemas differ between vendors and model versions. Recommendation: use a two part internal chargeback for shared GPU fleets: capacity reservation charge for guarantees plus a metered variable charge for actual work. Show both to teams and disclose any shared overhead allocation rule. 7. Design the authoritative usage ledger At request ingress, create an immutable logical request ID , verified org/team ID , model/config version, quota policy version, request class, region and admission decision. The ledger should record: \logical request id\, \attempt id\, \tenant id\, \team id\, \model revision\, egion\, equest class\, \cost center\. Accepted/rejected status, start/end times, terminal outcome (completed, cancelled, deadline, error), and whether any stream produced visible output. Prompt tokens , cached input tokens when supported, generated tokens , any provider reported reasoning/output detail, and whether values are measured or estimated. Versioned rate card , internal cost estimate, infrastructure allocation period, and separate adjustments/refunds. Idempotent commit key per attempt and separate logical request reconciliation to prevent double billing. Durable audit trail for late arriving metering records, corrections and disputes. Important: vLLM exposes aggregate counters like \vllm:prompt tokens total\ and \vllm:generation tokens total\ via Prometheus. These are useful to reconcile fleet totals but are generally insufficient as a canonical tenant billing ledger , because aggregate series need not carry verified tenant identity or exactly once logical request boundaries. Collect authorized request level metering at the gateway/engine integration and retain sensitive identifiers only in controlled storage—not as high cardinality Prometheus labels. 8. A safe two phase quota accounting algorithm \\\text 1. Authenticate request; resolve tenant/team/cost center server side. 2. Validate model, region, policy, prompt size, declared output cap. 3. Atomically reserve input/output budget and concurrency slot. 4. Admit to an eligible GPU/queue only if live capacity and deadline permit. 5. Record one attempt with a unique attempt ID. 6. As streaming progresses, update bounded provisional work metering. 7. At terminal completion/cancel/error, commit authoritative measured use. 8. Release unused reserved budget and concurrency. 9. Reconcile orphaned attempts from worker death with timeout and audit rules. 10. Deduplicate external customer billing by logical request ID and explicit retry policy. \\\ Do not write a distributed quota counter without atomicity. Concurrent requests can oversubscribe a quota during races. For financial records, use durable transactional ledgers or properly fenced reservation/commit operations and publish a documented policy for retries/cancellations. If a worker dies before reporting final usage, classify the record estimated/unresolved and reconcile rather than assuming zero tokens consumed. 9. Kubernetes quota example and limitations For a dedicated team namespace requesting up to four GPU resources with a device plugin resource named vidia.com/gpu\: \\\yaml apiVersion: v1 kind: ResourceQuota metadata: name: team gpu quota namespace: team a spec: hard: requests.nvidia.com/gpu: "4" \\\ This sets a namespace quota on requested GPU resources using Kubernetes extended resource quota rules. It does not limit per request tokens, guarantee that four GPUs are physically available, allocate a fraction of a GPU to a shared inference service, or provide a bill. Dynamic Resource Allocation (DRA) quotas have their own resource names/rules; follow the cluster's allocation model. If teams share a centrally deployed gateway and inference engine under one namespace, its GPU quota applies to the engine namespace , not automatically to end tenants. Enforce tenant quotas in the application control plane. 10. Design fairness and avoid quota gaming Rate limit by estimated token work and active generation , not just requests/second. Budget input and output separately: output decode is often more expensive per token and less predictable. Consider cached token discounts only if the inference cache saves real work under the chosen deployment and policy. Prevent a user from selecting an arbitrary team for billing. Authenticated identity and approved project membership must determine the cost center. Version rate cards and meter corrections; cost calculations must be reproducible. Give teams timely usage reports and dashboards: remaining allowance, daily burn, projected month end spend, per model cost and top workloads. Protect against retry amplification : a tenant may consume compute in repeated failed attempts even when customer billing applies only once. Enforce fairness for small tenants even when a large tenant has unused quota; unused budgets do not justify blocking all smaller tenants. Never include raw prompts, tool arguments or private data in financial telemetry unless absolutely necessary with authorization and retention controls. 11. Production incident: “free” retries create a surprise bill Incident: Tenant A's application times out on long generations. The gateway retries requests twice without preserving a logical request ID. The metering system counts only successful responses and bills Team A for one completion, but the backend performed three partial decodes. GPU invoices rise 35% despite stable user visible requests; other tenants' p95 TTFT degrades because retries fill KV memory. Root cause: No linkage from logical request to downstream attempts. Failed and cancelled attempts omitted from capacity metering. Quota reserved only on the first attempt and not on aggregate compute budget. No tenant level limits on parallel/retried generation. Engineering fix: propagate logical request ID and unique attempt IDs, enforce bounded retry policy, meter all actual GPU work even on failure, and explicitly distinguish internal infrastructure chargeback from user billable completions. Add admission limits for token work across retries and showback dashboards for expensive unsuccessful work. Prevention: reconcile gateway request logs with vLLM aggregate token counters and cloud GPU invoices, test cancellation/retry races, and alert when billed infrastructure cost diverges from attributed usage by more than the documented tolerance. 12. Monitor reconciliation and business outcomes Track: Per tenant/team: admitted/rejected requests; input/uncached/cached/output tokens; token rate/active concurrency; quota reserves, commits, refunds and budget exhaustion. Shared fleet: GPU hours by hardware/region, token throughput, KV occupancy/preemptions, actual idle capacity, failed decode work and SLO compliant goodput. Financial: allocated versus unallocated GPU costs, unit cost by model/workload, reserved baseline charges, rate card version, contested adjustments and invoice drift. Reliability: admission denial precision, p95 TTFT/TPOT by class, fair share service ratios and timeout/retry amplification. Reconciliation: compare (1) durable request ledger token totals, (2) vLLM engine aggregate counters over the same time interval, and (3) infrastructure billing and reserved capacity costs. Expect some definitional differences from prefixes, retries, cancellation, cache and telemetry losses; document each reconciliation adjustment. Treat a mismatch as an investigation trigger rather than automatically modifying customer invoices. 13. Interview ready answer “I'd enforce hierarchical org/team/application quotas using an authenticated gateway, reserve estimated input/output token budgets before admission, and settle actual usage from a durable per request ledger. I'd track retries as distinct attempts under one logical request and reconcile cancelled/failed GPU work separately from customer billable output. Kubernetes GPU quotas can limit dedicated namespaces but cannot solve tenant token quotas inside a shared inference engine. For cost allocation I'd combine reserved GPU capacity commitments with weighted, model specific actual token usage, expose showback dashboards, and reconcile ledger totals against vLLM metrics and cloud bills.” Research references: Kubernetes — Resource Quotas and GPU Extended Resources vLLM — Production Metrics OpenAI — Prompt Caching Usage and Accounting OpenTelemetry — Generative AI Usage Conventions FinOps Foundation — Allocation
How do you reconcile LLM inference token usage, GPU infrastructure costs, and billing discrepancies across tenants?
LLM inference billing reconciliation verifies that three records of activity agree closely enough to explain, audit and correct the cost attributed to each tenant: (1) request level usage events , (2) inference engine operational telemetry , and (3) infrastructure/provider invoices . Their totals are not supposed to be identical units : request events may count billable tokens, engine counters measure work performed, and cloud bills measure time based GPU resources plus storage, networking, commitments and discounts. A sound reconciliation system links these views through explicit definitions and maintains a documented unallocated cost and variance ledger . Prerequisite distinction: Question 276 designs hierarchical quotas, metering fields and chargeback policy. This question focuses on month end and daily close operations, data quality tests, exact discrepancy investigation, cost normalization, immutable financial corrections and invoice dispute handling . 1. Why totals differ even when nothing is broken Different units: input/output tokens are not GPU hours; prefill and decode cost per token varies with context, batching, hardware, caching and utilization. Different scopes: the engine may serve batch jobs, shadow traffic, health checks, internal evaluations and retries that the customer ledger intentionally does not bill. Different clocks: gateway timestamps, inference completion timestamps, cloud billing timestamps and invoice settlement can cross midnight or a billing period boundary. Different notions of success: a gateway may count one logical request while an engine performs multiple backend attempts ; timed out or cancelled work still consumes GPUs. Caching and token categories: input token totals commonly include cached input as a subset . Reasoning output may be a subset of output tokens. Summing a subtotal with its parent double counts. Hardware idle time: fixed or reserved GPU capacity accrues cost even when no tokens are processed. Shared costs: regional routers, replicas, object storage, networking, logging and observability often lack a unique owner. Pricing and credits: discounts, commitments, credit adjustments, taxes, support charges and amortized costs may arrive in different datasets or revisions. Telemetry gaps: restarts, missing final stream usage, lost events and poorly labeled metrics lead to real measurement uncertainty. Do not force token totals to equal dollar totals. Reconcile each source within its own semantic domain, then apply a versioned, independently auditable allocation model. 2. Three independently controlled data sources A. Financially authoritative request ledger For each authenticated original request, keep immutable logical request ID, tenant/project/cost center identity, model, API version, source region, submitted and completed times, billable input/output usage, cached/reasoning subsets where known, pricing/rate card version, charge/refund and final status. Distinguish each backend attempt using a separate attempt ID. Retried attempts can consume GPU work even if policy bills one completed logical request. Store provenance : server measured, provider reported, estimated, corrected or missing. B. Operational usage and capacity telemetry Capture vLLM compatible \vllm:prompt tokens total\ and \vllm:generation tokens total\ as counters, request success/finish reason, queue/cancellation metrics, model/region/pod revision, GPU running hours and fleet idle capacity. Use the deployed engine's actual \/metrics\ exposition because names and labels vary by version. Operational counters cannot automatically supply tenant billing events . Avoid tagging Prometheus time series with individual request IDs or unbounded tenant identifiers. C. Supplier financial statement Ingest cloud provider detailed usage and cost exports or invoices with billing account ID, resource ID, SKU, region/zone, usage period, price, discounts, commitments, credits, adjustment records and currency. Where supported, preserve raw versus amortized/effective cost semantics rather than quietly mixing them. FinOps cost allocation guidance recommends consistent owner/account/project/tag metadata and transparent rules for shared costs; tags may be missing or introduced too late for prior charges. 3. Beginner friendly numerical example: trace the mismatch For one day, suppose a ledger contains: Team A billable: 1.0 million uncached input tokens, 0.2 million cached input tokens and 0.3 million output tokens. Team B billable: 0.6 million uncached input tokens, 0.4 million cached input tokens and 0.2 million output tokens. Therefore billable input = \1.0 + 0.2 + 0.6 + 0.4 = 2.2 million\, billable output = \0.3 + 0.2 = 0.5 million\. Engine counters over the matching, normalized time window report: 2.5 million prompt tokens and 0.58 million generated tokens . Raw difference = 300,000 input and 80,000 output tokens . A drill down matches: 1. Shadow/integration test attempts: 150,000 input, 40,000 output . 2. Client cancelled or timed out backend attempts: 100,000 input, 30,000 output . 3. Internal retried work on logical requests: 50,000 input, 10,000 output . Input explanation: \150k + 100k + 50k = 300k\. Output explanation: \40k + 30k + 10k = 80k\. The engine work reconciles in this simplified example: the customer billable ledger intentionally excludes these classes. However , a cancellation policy may bill some partial completed usage; inspect the actual contract rather than assuming every failure is free. Infrastructure example: the cloud cost export shows $1,500 of GPU costs and $150 of other allocatable support/storage/network costs. Assume $300 in already identified fixed guaranteed capacity allocation is charged to Team A and the remaining $1,350 is split using approved workload weights: A weighted share 60% → \$810\ variable allocation + \$300\ fixed = $1,110 . B weighted share 40% → $540 . Check \$1,110 + $540 = $1,650\. These are invented numbers and policies . They illustrate that matching engine tokens does not directly determine a unique dollar split : the documented contract and measured resource attribution choose the weights. 4. The close pipeline: raw, normalized, matched, explained, published 1. Collect immutable raw exports from gateway, worker/event queue, Prometheus rollups and provider billing datasets; record source revision, extraction time and checksum. 2. Normalize identity and time: map service account/resource to owner; fix UTC boundaries, request start versus completion semantics and late arriving settlements. 3. Normalize token taxonomy: input total including cached, output total including reasoning if reported; preserve cache read/cache write and modality subsets without adding them again . 4. Dedupe and link: one logical user request → one or more backend attempts; link attempt IDs to worker/model/region and terminal status. 5. Measure three variances separately: ledger versus engine tokens, engine GPU time versus provisioned/billed GPU time, and allocated departmental costs versus provider total. 6. Explain variance in reason coded buckets: retries, cancellations, shadow tests, late records, cache semantics, idle baseline, shared network, missing ownership or rate card difference. 7. Apply corrections: append adjustment events rather than overwriting the closed ledger; preserve old invoice snapshots and audit who approved changes. 8. Publish a reconciled statement with residual unexplained amount, accepted tolerance, timestamps, source versions, cost allocation method and unresolved issues. 9. Reopen only under controlled process for late bills, refunds or confirmed defects. A successful close needs an explicit residual . For example, “98.8% allocated, 1.2% still unallocated” is auditable; silently assigning the leftover to the biggest team is not. 5. Reconcile tokens using explicit equations For a matched model, region and interval: \\\text engine input = billable input + nonbillable shadow input + nonbillable health and eval input + retries and failed attempt input + boundary or provenance adjustments + unexplained input engine output = billable output + nonbillable shadow output + nonbillable health and eval output + retries and failed attempt output + boundary or provenance adjustments + unexplained output \\\ Warning: this identity only works if classes are mutually exclusive , the engine and ledger count tokens the same way, and all terms refer to the same model/time scope. A cached token subtype is inside input totals; do not treat it as another top level addend. Some backend counters may count actual processed prompt tokens differently from API reported billable input, especially with prefix caching or partial failures; explicitly introduce separate definition adjustments rather than manufacturing equality. Report absolute token variance and relative variance : elative variance = abs(engine explained ledger total) / max(engine, ε)\. Use tolerances determined by expected sampling, pipeline lag and engine semantics. A daily target like ≤0.5% unexplained could be used as an organizational gate, but is not a universal standard. 6. Attribute cloud spend without double counting Create a cost fact table keyed by invoice month, account, cloud resource/SKU, region, effective cost basis and billing line . Separate: Direct inference GPU instance hours. Guaranteed idle/reserved GPU baseline and commitment or amortized charges. Batch/spot GPU usage and interruption overhead. Shared ingress/router, storage, cross zone/region data transfer, observability and recovery capacity. Provider adjustments, credits, taxes and unsupported to allocate lines. Cost basis discipline: choose whether reporting uses unblended , net , or amortized effective cost, state it, and reconcile to the provider's equivalent invoice metric. Amortized commitment charges and usage records can double count if blindly summed. Discounts and credits may be company wide rather than tenant specific; document whether they are passed through or retained centrally. A simple allocation: \\\text tenant allocated cost = directly owned GPU cost + guaranteed capacity fee + share of shared GPU pool + share of shared network storage observability + approved adjustments \\\ \\\text provider in scope cost = sum(tenant allocated cost) + unallocated or central cost + separately recorded out of scope cost \\\ The equations must use the same cost basis and scope. A cloud invoice may include taxes or support items that should be excluded from inference chargeback; keep them visible in an out of scope reconciliation bucket, not silently ignored. 7. Handle out of order events and partial streams A streaming LLM response can terminate without a final provider usage message. Use a state machine : \RECEIVED\ → \ADMITTED\ → \ATTEMPT STARTED\ → \STREAMING\ → \COMPLETED | CANCELLED | ERRORED | TIMED OUT\. Persist provisional token observations without treating estimates as authoritative settled amounts. At terminal transition, prefer engine/provider authoritative usage when available, with an explicit confidence/source flag. On worker death, create a missing final usage item, not a zero cost completed request; reconcile against engine level work and documented contractual rules. Use a reconciliation watermark (e.g., wait for an agreed late event horizon before a day is final), but don't delay safety/overage limits on live traffic. Do not let a late completion from an expired backend attempt overwrite a more recent logical request settlement. For retries, preserve multiple attempt records and a single logical request customer billing policy . Prevent repeated side effecting tools with separate idempotency protection. Important: a customer facing output that stops midstream can be operationally incomplete even when many tokens were generated. Determine billability by written policy and observed usage, not only HTTP 200 at first byte. 8. Example SQL reconciliation model This query is illustrative and uses hypothetical tables; adjust to the real schema and time boundaries. The \usage events\ table is an immutable fact table with one finalized record per backend attempt and explicit customer billing status. \\\sql WITH tenant usage AS ( SELECT model version, region, SUM(input tokens) FILTER (WHERE customer billable) AS billable in, SUM(output tokens) FILTER (WHERE customer billable) AS billable out, SUM(input tokens) FILTER (WHERE NOT customer billable) AS nonbillable in, SUM(output tokens) FILTER (WHERE NOT customer billable) AS nonbillable out FROM usage events WHERE completed at = :start utc AND completed at < :end utc AND source quality = 'measured' GROUP BY model version, region ), engine AS ( SELECT model version, region, SUM(input token delta) AS engine in, SUM(output token delta) AS engine out FROM engine usage windows WHERE window start = :start utc AND window start < :end utc GROUP BY model version, region ) SELECT e.model version, e.region, e.engine in COALESCE(t.billable in,0) COALESCE(t.nonbillable in,0) AS residual input, e.engine out COALESCE(t.billable out,0) COALESCE(t.nonbillable out,0) AS residual output FROM engine e LEFT JOIN tenant usage t USING (model version, region); \\\ Pitfalls: actual backend attempt records may lack measured usage on crashes, and the time periods shown use event completion versus engine work window boundaries. Account for these with a lag buffer and missing usage categories. If a logical request has multiple attempts, ensure \customer billable\ on an attempt record has well defined semantics; ideally keep separate attempt level physical usage and logical request settlement to avoid charging multiple retry attempts unintentionally. 9. Production incident: a 27% unexplained GPU cost spike Scenario (fictional): Monthly GPU spend rises from $20,000 to $25,400 (+27%), but recognized customer billable tokens rise only 5% . Finance suspects tenant B is underbilled; engineers suspect GPU price changes. Investigation: 1. Normalize invoices by region, GPU SKU, period, discounts, commitment amortization and usage hours. 2. Compare engine generated token work and original request completions. Backend attempts rose 30% , including many retried long prompts after an engine timeout regression. 3. Worker logs reveal interrupted streams whose final usage was never recorded in the customer ledger; engine counters still include some of the work. 4. Reconcile hours: a new standby GPU deployment adds fixed spend but little new billable volume. 5. Separate causes: retry amplified GPU work , underutilized standby capacity , and some missing usage events . Do not assume the 27% cost delta must be assigned entirely to tenant B. Remediation: fix retry storms and timeout handling; improve terminal usage capture, log/metric reconciliation and immutable attempt linkage; allocate standby costs under the published reserved capacity policy; issue approved invoice adjustments only when contractual billable usage is proven. Prevention: daily anomaly tests for \GPU hours per successful request\, \engine attempts per logical request\, \unexplained tokens by model\, and \unallocated spend\, plus a documented monthly close procedure. 10. Controls and alerts for financial integrity Recommended checks: Data completeness: distinct logical IDs and attempt IDs, duplicate event rate, terminal status coverage, missing usage share. Token taxonomy: cached input ≤ total input, reasoning output ≤ total output, no negative usage, valid model/tokenizer and modality. Aggregate drift: summed ledger tokens versus engine counters by model/region with explicit nonbillable and definition adjustments. Cost drift: provider charged instance hours versus scheduler run hours, stable capacity versus idle time, unmatched resource IDs and tag coverage. Allocation coverage: % fully assigned cost, unallocated amount, shared cost basis and changes in overhead. Freshness: lagged bills and missing report partitions, external credits, currency changes and backfills. Governance: versioned rate cards, separation of raw facts and adjustments, auditor read access, approval trail and reproducible month end snapshots. Privacy/security: financial records carry authenticated tenant/cost center IDs; don't add raw prompts, retrieved content, secrets or request IDs to high cardinality monitoring labels. Set error thresholds proportionally to materiality, volume, expected pipeline lag and approved policy. A small absolute discrepancy can be material to a tiny tenant; a low percent mismatch in a large fleet can represent substantial dollars. 11. Source system caveats vLLM: exposed \vllm:prompt tokens total\ and \vllm:generation tokens total\ are aggregate engine counters, not an itemized tenant invoice. Compute Prometheus counter rates with proper resets; never sum gauge samples to derive token counts. OpenTelemetry GenAI conventions: input token totals contain cached subsets and output token totals may contain reasoning subsets. Choose the documented version of semantic conventions; newer token metric families may be in development. Prevent parent plus subset double counting. FinOps: chargeback requires documented allocation for shared and unallocated costs, owner metadata and defensible treatment of unused commitments. Cloud invoices: detailed cost exports may be delayed or revised. Retain raw snapshots and corrections and define when a period is provisionally versus finally closed. 12. Interview ready answer “I'd reconcile three views: a durable per attempt and per logical request ledger, engine token and GPU workload counters, and the cloud detailed cost export. I'd normalize model, region, UTC periods, cache token subsets, retry/nonbillable work, and effective versus amortized costs. Then I'd explain residual differences with reason codes and explicitly carry unallocated costs. Streaming cancellations need estimated or missing usage status, not fabricated zero usage. I'd close periods with immutable source snapshots and approved adjustments, and alert on unexplained token or GPU hour drift.” Research references: vLLM — Metrics and Prometheus Counters OpenTelemetry — GenAI Token Usage Semantic Conventions OpenTelemetry — GenAI Inference Span Usage FinOps Foundation — Cloud Cost Allocation Guide FinOps Framework — Cost Allocation
How do you detect and prevent cost anomalies caused by retries, token spikes, idle GPUs, and inefficient LLM inference configurations?
LLM inference cost anomaly detection identifies unexpectedly expensive serving behavior before it becomes a large billing surprise. Unlike normal financial reconciliation, which explains historical invoices, cost anomaly operations watch leading signals —backend attempts per user request, input/output token growth, idle GPU hours, failed generation work, prefix cache misses, deployment changes and the cost of meeting latency SLOs—and trigger safe investigation or containment. A cost anomaly is not necessarily an incident : a planned product launch may legitimately increase spend. The objective is to distinguish expected demand growth from waste, runaway behavior, broken configurations or unauthorized usage without disrupting healthy customer traffic. This continues Question 260 (unit economics), Question 254 (retry storm reliability), Question 273 (GPU purchasing), Question 276 (quota/chargeback), and Question 277 (billing reconciliation). New focus: detection algorithms, root cause attribution, safe automation, alert routing, and prevention of recurring spend anomalies. 1. Define the right unit economics first Do not page solely because total daily GPU cost increases. Split costs into: Offered demand: distinct original requests and input workload by tenant/model. Actual work: all backend attempts, uncached/cached prompt tokens, generated tokens, recomputation and failed work. Useful output: validated complete user requests or batch items, optionally restricted to SLO compliant results. Infrastructure: billed GPU hours, reserved idle capacity, per SKU price/region, checkpoint/network/storage expense and model startup overhead. Cost rate: estimated dollars/hour and dollars per 1,000 successful SLO compliant requests or per million verified, model specific output tokens. Example metric: \unit cost = total in scope GPU and shared cost / completed SLO conforming original requests\. Track the request count independently of retry attempts. When the denominator is near zero, use safe minimum volume rules; a unit cost series can spike because output dropped, not because raw cloud cost rose. 2. Four anomaly families A. Retry amplification Signal: \backend attempts / unique original requests\ rises sharply. Other signals: 429/503 and deadline timeouts increase; more prompt tokens are processed without matching successful completions. Causes: aggressive client retries, an overloaded route, failures after partial decode, missing retry budgets, or bad backoff. Safe responses: cap retry count, enforce global deadlines and jitter/backoff, investigate source errors, add per tenant limits and circuit breakers. Do not indiscriminately disable retries required for a defined recovery contract; measure actual success and SLO impact. B. Token spikes Signal: p95 input/output tokens per request and total processed tokens diverge from historical workload matched baselines. Causes: accidentally repeated document context, broken truncation, a new chat template, poor stop sequences, too high \max tokens\, runaway agent loops, response verbosity changes, prompt injection or abusive callers. Safe responses: validate input construction, stop conditions, context ceilings, task aware output budgets, tool hop budgets and per tenant authorization; escalate suspicious requests for security review. A large token spike may be legitimate , such as a scheduled long document summarization job; treat change calendar and product releases as context. C. Idle or stranded GPUs Signal: GPU hours and fixed reserved costs rise while completed useful work and demand stay flat; a large ready/idle fleet persists. Causes: stale warm standby, duplicate deployments, a failed scale down policy, unused capacity commitments, or GPUs still initializing. Safe responses: reclaim unneeded batch replicas first, right size pool floors, fix autoscaler blockers, and review commitment/reservation allocations. Do not scale away required failure headroom , data residency capacity, streaming workers or multi region disaster recovery guarantees just to improve utilization. D. Inefficient runtime/configuration Signal: cost per good request increases at matched request mix, despite normal external demand. Causes: disabled prefix caching, changed tokenizer/template, too small batches, excessive preemption/KV pressure, slower kernels, model/quantization change, wasted repeated prefill, or new GPU type with lower effective goodput. Safe responses: compare version fingerprints and controlled matched load benchmarks; restore proven settings only after checking output quality, TTFT/ITL and memory safety. GPU utilization alone cannot diagnose inefficiency: high occupancy can coexist with retry waste, and low occupancy may be deliberate SLO headroom. 3. Beginner friendly numerical example: trace a 50% cost spike Suppose a stable service processes 10,000 distinct original requests/hour on a GPU fleet costing $120/hour . Normal backend attempt ratio = 1.05 attempts/original request → 10,500 backend attempts/hour . Normal completed SLO compliant requests = 9,500/hour . Normal cost per SLO compliant request = \$120 / 9,500 = $0.01263\ (about 1.26 cents ). After a deployment: Original demand remains 10,000/hour . Attempts rise to 1.50 per original → 15,000 attempts/hour . GPU cost increases to $180/hour because additional capacity scales up. Successful SLO compliant requests fall to 9,000/hour . New cost per good request = \$180 / 9,000 = $0.0200\ ( 2 cents ). Analysis: 1. Raw hourly GPU cost: \($180−$120)/$120 = 50%\ increase. 2. Backend attempt volume: \(15,000−10,500)/10,500 ≈ 42.9%\ increase. 3. Good request unit cost: \($0.0200/$0.01263−1) ≈ 58.3%\ increase. 4. Original user demand did not grow; attempt amplification is a high priority hypothesis. 5. Confirm with correlated request/attempt IDs and error/timeout traces before claiming retries caused all 50% of the bill increase. Conclusion: alert on dollars per good original request and attempt amplification , then investigate the deployment and downstream errors. Charging for 15,000 user requests would be wrong because there were only 10,000 distinct originals. All figures are hypothetical teaching data. 4. A second example: token heavy prompt regression Imagine a RAG application previously averaged 2,000 input tokens per request but a new retrieval template repeatedly embeds the same passages, raising the average to 8,000 tokens . At 20,000 requests/day , the input token load rises from \40 million\ to \160 million tokens/day\. Increase = 120 million input tokens/day , or 4× the previous input token volume. This does not necessarily imply a 4× GPU bill: prefix reuse, context lengths, packing, output tokens, rates and provisioned GPU fleet determine actual costs. Compare unchanged workload slices, actual uncached prefill tokens, GPU service time and SLO goodput. Use a regression test that inspects deduplicated retrieved context rather than deleting relevant information indiscriminately. 5. Build a layered detector Layer 1 — immediate operational guardrails (seconds/minutes) Limit requests, token work reservations, maximum output, concurrent generations and agent/tool hops. Alert on runaway retries, cancellation rate, request queue pressure and stream errors. Halt new low priority work when quotas or capacity safety limits are reached; preserve high priority customer SLOs. Layer 2 — short window anomaly models (5–60 minutes) Compare attempts per request, mean/p95 token lengths, generated output tokens, prefills per successful request and estimated cost rate to seasonality aware historical windows. Join with release/deployment events, tenant classes, batch schedules, GPU SKU and active model revision. Require minimum volume, consecutive anomalous intervals and both relative and absolute dollar impact to reduce noisy alerts. Layer 3 — billing anomalies (hours/days) Join provider cost exports and spend monitor alerts as they arrive, validate with actual cloud invoice data and write durable incident outcomes. AWS Cost Anomaly Detection and Google Cloud cost anomaly tools provide useful broad financial signals, but timing and available cost basis differ. They should supplement, not replace , application level early detection. Important: near real time cost is usually an estimate , not the provider's final invoice. Label estimates and corrections clearly. 6. Baselines and robust thresholds Choose baselines by model, region, tenant/workload class, hardware SKU, day of week/hour and version . This prevents incorrectly flagging planned batch windows or comparing unlike model releases. Useful methods: Static safety thresholds: mandatory ceilings for runaway \max tokens\, tool hops, retry budget and tenant quotas. Appropriate for known dangerous behavior. Rolling median + MAD: robust to isolated spikes; for an illustrative alert, flag a cost rate greater than median + \k × 1.4826 × MAD\, provided history is sufficient and seasonality is accounted for. Seasonal forecasting: expected hourly GPU spend or token work baselines with prediction bands; compare observed versus expected with model quality checks. Change point detection: sustained shifts following deployments or routing changes. Multivariate rules: cost rate increases and original demand is flat and attempt ratio rises; or capacity cost increases while useful output remains unchanged. Do not make a universal promise that any fixed k/MAD threshold works for every workload. Test historical incidents, precision, recall, seasonality and alert fatigue; use absolute dollar materiality and minimum sample counts. 7. Prometheus and dashboard examples In vLLM V1, verify which counters exist in your installed engine's \/metrics\. Common names include \vllm:prompt tokens total\, \vllm:generation tokens total\ and \vllm:request success total\. Generated tokens/s across all measured engine replicas: \\\promql sum(rate(vllm:generation tokens total[5m])) \\\ Prompt tokens/s: \\\promql sum(rate(vllm:prompt tokens total[5m])) \\\ KV usage by model (fraction 0–1): \\\promql max by (model name) (vllm:kv cache usage perc) \\\ Application specific retry amplification (assumes a trusted gateway emits both counters and uses these illustrative names): \\\promql sum(rate(llm backend attempts total[5m])) / clamp min(sum(rate(llm original requests total[5m])), 1e 9) \\\ Caution: those gateway metrics are custom examples, not native vLLM metrics . The ratio is reliable only if numerator and denominator refer to the same tenant/model/time scope and sampled period; very low volume needs a minimum count gate. Engine token counters cannot distinguish customer billable tokens, shadow traffic, cache related definitions or retries without other metadata. For cost per good request , compute using a billing/usage pipeline that joins cost estimates and verified original request SLO outcomes , rather than pretending a GPU memory gauge is a dollar value. 8. Anomaly triage: a production runbook When cost rate/efficiency alerts fire: 1. Confirm scope: exact model, version, account, tenant, region and GPU SKU affected; separate service health from spend impact. 2. Compare demand: unique original requests and prompt/output length classes; was this a real customer traffic increase? 3. Compare attempts: backend attempts per original, failures, timeouts, cancelled streams, retries and shadow/test traffic. 4. Check token and cache changes: input/output distribution, prompt length, prefix cache hit fraction, uncached prefill work and context/template revisions. 5. Check capacity: ready GPUs versus allocated/billed GPUs, utilization, SLO goodput, idle reservation, autoscaler behavior and deployment duplication. 6. Check configuration diff: batch token budget, max active sequences, KV allocation, compilation, quantization, model version, routing and stop sequences. 7. Estimate financial exposure: excess dollars/hour, forecast until repair, actual confidence interval and required intervention urgency. 8. Pick a safe action: rollback clearly defective change, cap runaway requests, halt noncritical batch work, fix retries or remove explicitly unneeded warm capacity. 9. Measure results: affected service success rate, TTFT/ITL, goodput, actual token work rate and cost per good request after mitigation. 10. Close incident: preserve graphs, annotated event and owner; reconcile eventual billing with estimates (Question 277) and add a targeted prevention check. Never automatically delete all GPUs, revoke essential quotas or downgrade models solely because a cost monitor crosses a threshold. Cost controls must be subordinate to customer SLIs, quality, contracts and disaster recovery requirements. 9. Production incident: runaway retries plus idle GPUs Scenario: Overnight, a deployment mistakenly adds two retries to every long request after an overly short timeout. Backend prompt token work and GPU utilization rise; the autoscaler adds four GPUs. After the timeout is repaired, the four extra replicas remain provisioned because a minimum replica setting was inadvertently raised. Observed behavior (illustrative): Hour 1: 10,000 original requests, 10,500 backend attempts and $120 GPU spend. Hour 2: 10,000 originals, 15,000 attempts and $180 spend. Hour 3: 10,000 originals, 10,500 attempts again, but GPU spend remains $180 because idle capacity has not scaled down. Root causes: aggressive retries increased actual GPU work, then a persistent replica floor change stranded paid capacity. A rule monitoring only retry ratio would consider Hour 3 normal, while a rule monitoring only GPU utilization could miss the contractual need for spare capacity. Fix: correct timeout/retry budgets, bound original request SLO deadlines, verify autoscaling floor and spare capacity policy, remove only excess non required GPU replicas, and retest load plus one replica failure. Prevention: two independent detection paths— retry work amplification and idle provisioned GPU hour cost —and deployment annotation to explain why the fleet capacity changed. Verify post deploy configuration and scale down behavior. 10. Safe automated remediation Low risk automated actions: Block clearly invalid requests that exceed published token/context limits. Rate limit isolated abusive or noncritical excess work under predefined tenant policy. Halt new speculative batch tasks when an explicit budget ceiling is reached. Roll back a known bad configuration only if automated functional and latency gates establish regression. Trigger an incident with diagnostic links and a proposed action; notify the accountable service/FinOps owner. Actions requiring stronger approval: Shrinking reserved interactive pools or cross region failover capacity. Changing model quality/quantization, silently truncating prompts or altering generation output caps for user facing features. Disabling retries for workflows that rely on them for correctness. Revising customer bills based on estimated token use or incomplete telemetry. Changing spend caps in ways that can interrupt essential production workflows. Add safeguards: dry run, per model/tenant blast radius, SLO guardrails, cooldown/hysteresis, owner approval, measured rollback path and immutable action history. Distinguish soft budget alerts from hard enforcement; cloud budget notifications are not automatically immediate spending stops. 11. Evaluate detector quality and savings Precision and false positive rate: what fraction of alerts actually indicate actionable unexpected waste? Planned launches or batch backfills are common false positives. Recall: how many historical waste incidents were identified? Detection and mitigation delay: minutes from root cause onset to alert and to corrective action. Excess cost: estimated spend beyond an approved workload matched baseline, with exclusions for legitimate growth. Cost avoided: counterfactual projection of ongoing unnecessary cost minus mitigation expenses; state assumptions and do not overclaim. SLO impact: ensure intervention did not worsen p95/p99 TTFT, TPOT, success or full stream completion. Policy compliance: did owners approve riskier actions and was the anomaly correctly categorized? Illustrative economics: if a verified bug burns an extra $40/hour , stopping it 10 hours earlier than the previous manual procedure avoids approximately $400 in incremental compute, assuming the excess would have persisted at that rate. This is a forecast of avoidance, not an automatic provider refund. 12. Common mistakes Alerting on total bill rather than cost conditioned on actual workload demand and model mix . Counting retries as new customers or using only HTTP 200 to define success. Counting cached prompt tokens twice or treating output/reasoning subsets as independent usage totals. Treating high GPU utilization as intrinsically efficient or low GPU utilization as automatically wasteful. Using provider billing feeds as if they were instantaneous/immutable. Comparing different GPU hardware or model revisions without normalization. Hiding the true effects of retries, error streams and shadow traffic from infrastructure cost analysis. Auto scaling down disaster recovery standby or critical interactive headroom to satisfy a budget alert. Changing safety/quality sensitive inference settings with no acceptance tests. 13. Interview ready answer “I'd build cost anomaly detection around workload normalized unit economics , not just total dollars: backend attempts per original request, input/output token distributions, cost per successful SLO request, GPU hours, idle reserved capacity and cache/serving efficiency. I'd use immediate policy limits plus seasonality aware multivariate anomaly alerts. On an alert I'd isolate demand versus retries, token spikes, idle GPUs and configuration regressions, estimate dollars/hour at risk, and mitigate only within approved SLO and DR guardrails. Finally I'd reconcile the eventual invoice and track false positives, time to detection and avoided excess cost.” Research references: FinOps Foundation — Anomaly Management AWS — Cost Anomaly Detection Google Cloud — View and Manage Cost Anomalies vLLM — Production Engine Metrics Microsoft Learn — Anomaly Management in FinOps
How do you implement automated cost-optimization guardrails and policy-based remediation for production LLM inference?
Automated cost optimization guardrails are explicit, versioned decisions that constrain or remediate expensive inference behavior without silently violating latency SLOs, safety requirements, tenant contracts, or disaster recovery capacity guarantees . A safe system does not merely alert that spending rose. It evaluates the request or resource change against an approved policy, selects a permitted action, records the decision, verifies the outcome, and rolls back if customer health deteriorates. This is not the same as automatic cost anomaly detection. Question 278 explains how to recognize retry amplification, token spikes, idle GPU hours and inefficient engine settings. This lesson asks which actions automation may take, what evidence it needs, who approves them, how to prevent policy conflicts, and how to prove the intervention actually reduced waste . 1. Separate three controls Preventive admission policy: before model invocation or deployment, reject an unauthorized model/region, invalid token budget, missing cost center attribution or unsupported GPU configuration. These are deterministic rules. Runtime protective policy: under measured overload or budget pressure, reject new, eligible noncritical work, cap retries, defer batch tasks or limit tenant specific bursts. Preserve protected interactive requests and existing streams under the published contract. Remediation policy: after a confirmed condition, propose or execute a reversible change—stop excess batch admission, remove unneeded replicas, roll back a known bad configuration, or require manual review for major GPU pool adjustments. Use a risk tier for each action: \observe\, ecommend\, equire approval\, \automate reversible\, \prohibited\. A financially urgent change is still prohibited if it breaks mandatory reliability or residency rules. 2. Define non negotiable invariants Before automation, specify: 1. Interactive p95/p99 TTFT, TPOT and request success SLOs and approved error budget policy. 2. Minimum ready GPU capacity for peak demand, N−1 worker loss, zone failure and documented multi region DR requirements. 3. Per tenant contract: allowed model, maximum tokens, concurrency, region, data residency, quality requirements and budget override rules. 4. Safety and correctness: no implicit switch to a weaker model, silently truncated prompts, disabled tool approval or changed safety filter. 5. Job constraints: admitted batch deadlines, retry/checkpoint guarantees and idempotent side effects. 6. Operator controls: change approval, emergency suspension, maintenance windows and bounded rate of change. Precedence matters: security/legal and data residency rules customer safety and contractual guarantees healthy service SLOs and DR minima approved cost objectives. If a policy conflicts with a higher priority invariant, do not execute the change automatically . 3. Beginner friendly numerical example: a safe versus unsafe scale down Suppose a fleet runs 10 ready GPU replicas at an illustrative $3 per GPU hour . Matched load testing shows 4 SLO compliant requests/s per replica after per replica operating headroom. Current offered traffic = 20 requests/s . Minimum replicas for current demand = \ceil(20/4) = 5\. Reliability policy requires two additional ready replicas for an accepted failure/burst scenario, so the protected floor is 7 replicas . A naive scale down policy proposes 5 replicas , but that would violate the approved reliability floor. A policy aware controller permits reducing 10 → 7 , assuming no streams would be interrupted and measurements confirm sufficient capacity. Gross modeled hourly saving = \3 replicas × $3/h = $9/hour\. Over 24 hours, estimated saving = $216 , only if sustained demand and prices remain unchanged . One off drain/warmup, commitments and billing effects can reduce realized savings. Observe and revert: If p95 TTFT rises from an acceptable 650 ms to 1.1 s against a 1 s SLO during the controlled change, stop further reduction and restore healthy capacity. The successful unit of optimization is lower effective cost with maintained SLIs , not “three pods deleted.” All prices, capacities and latency measurements are hypothetical teaching values. 4. An action policy matrix Allowed automatically, when well defined and tested: Deny new requests exceeding a tenant's published maximum context/output token cap. Enforce a fixed retry attempt ceiling, backoff and an overall request deadline. Pause new noncritical batch admission when a separately approved batch budget is exhausted. Reclaim idle disposable batch workers above their documented floor, using draining and confirmed idle state. Route eligible best effort traffic to lower cost approved capacity if the SLO, model, residency and ready capacity checks pass. Approval usually required: Decrease the protected interactive replica floor or cross zone/region standby reserves. Change model size, quantization, reasoning effort limits, prompt context or response quality guarantees. Disable infrastructure or scale down workloads that still hold live conversations or agent state. Change tenant rate cards, contractual hard caps or customer facing billing. Introduce new Spot dependence for latency critical traffic. Prohibited without a separate contractual/emergency process: Sending private context to an unapproved region/provider because it is cheaper. Stopping all production inference when an advisory cost budget alert arrives. Breaking paid workload commitments silently or replaying tool actions without idempotency. Disabling safety controls to reduce tokens or latency. 5. End to end policy control loop Build a reconciliation aware control plane: 1. Observe : ingest original request counts, successful SLO completions, token metrics, GPU hour estimates, model/version, ready replicas, KV pressure and provider billing alerts. 2. Validate freshness and completeness : reject stale cost samples, missing ownership, missing health metrics and inconclusive SLO estimates as grounds for destructive automatic changes. 3. Attribute scope : identify the tenant, model, GPU pool, region and workload class responsible; do not punish unrelated tenants. 4. Evaluate policy : input a signed/versioned rule set, current cluster state, minimum capacity, tenant contract and action risk classification. 5. Simulate : dry run expected savings, resulting ready capacity, p95 TTFT/TPOT margins and affected workloads; require positive confidence and headroom. 6. Authorize : get explicit operator approval if the action exceeds the allowed automation risk class. 7. Act narrowly : perform a bounded, idempotent action with an expiry, cooldown and rollback plan. 8. Verify : compare real customer SLIs and cost rate against the matching baseline; automatically reverse if guardrails are breached. 9. Record : save decision ID, owner, policy revision, inputs, evidence, execution result, SLO verification and estimated versus realized savings. Use least privilege service identities for the controller. A cost management bot should not have unrestricted access to terminate all GPUs, alter billing identities or change security policies. 6. Policy as code versus provider budget alerts Open Policy Agent (OPA) / Gatekeeper can enforce declarative policy on Kubernetes resource creation or updates: for example, requiring cost center labels, limiting allowed node pools or rejecting unapproved GPU instance resource configurations. Its decisions should be reviewable, tested and deployed like code. Important limit: admission webhooks inspect Kubernetes API operations; they do not automatically intercept every inference HTTP request or know real time model tokens. Implement per request policy in a trusted LLM gateway/quota service; use OPA for decisions where its input data and latency make sense. AWS Budgets Actions can run a defined action automatically or after manual approval, including IAM/SCP controls or actions targeting particular EC2/RDS instances. An IAM/SCP restriction on future provisioning is different from stopping a running inference fleet. Scope and test those permissions carefully. Google Cloud Billing budgets may issue notifications through Pub/Sub, but ordinary alerts only budgets do not automatically cap spending . Google also documents service specific spend cap budgets for eligible services , which can pause new usage; they are not universal GPU budget controls . Do not conflate billing alert cadence with an immediate token admission control. Application gateways are the appropriate place for low latency token budget decisions. 7. A simple policy as code example The following is pseudocode for an application controller , not a built in vLLM policy engine: \\\python def evaluate reduction(state, candidate): if not state.telemetry fresh or not state.contracts loaded: return {"decision": "BLOCK", "reason": "incomplete evidence"} if candidate.action == "scale down gpu pool": if candidate.pool == "interactive": if state.slo burning or state.failure headroom at risk: return {"decision": "REQUIRE APPROVAL", "reason": "protected capacity"} required = max( state.demand based replicas, state.reliability replica floor, state.disaster recovery floor, ) if candidate.target ready replicas < required: return {"decision": "BLOCK", "reason": "below protected floor"} if state.active streams on targets 0: return {"decision": "BLOCK", "reason": "drain first"} return {"decision": "ALLOW REVERSIBLE", "max change": 1, "verify slo": True} return {"decision": "REQUIRE APPROVAL"} \\\ Engineering caveats: equired\ must be derived from approved measurements, not guessed counts; a regional DR floor is not always reducible to a single local \max\ rule. This is a simplified illustration. In practice check zone distribution, model compatibility, pending reservations and disruption budgets before acting. 8. Prevent races and automation loops A naive controller can cause oscillation : Low utilization → scale down. Queueing rises → autoscale up. Fleet becomes temporarily idle while warming → scale down again. Repeated cold starts raise cost and TTFT while the policy thinks it is saving money. Countermeasures: Hysteresis: separate thresholds for enabling and disabling cost saving mode. Cooldown: enforce a minimum observation period after each change. One step limit: shrink only one eligible replica or a bounded percentage per cycle. Leases/fencing: use a singleton controller lease or durable action ID so duplicate budget Pub/Sub events cannot produce duplicate shutdowns. Manual freeze: on call can suspend cost actions during a customer outage, incident or planned traffic event. Rollback health gate: SLO regression, worker OOM, loss of standby minimum or error rate spike reverses the experiment. Controller conflict management: coordinate with HPA/KEDA/Kueue and cloud autoscalers so independent policies do not fight over replicas. Google Cloud budget Pub/Sub notifications are at least once, potentially duplicated or out of order ; do not blindly execute every received message. Use monotonic policy revisions, idempotent action IDs and current state revalidation. 9. Separate business budget controls from runtime availability Soft guardrails: warning at 70% and 90% of predicted monthly spend, then require spend owner review. These percentages are company policy, not universal best practice. Hard tenant guardrails: published contractual API quotas may reject new requests after a verified usage threshold, returning a clear \429\ or documented quota status. The gateway must settle reserved versus actual token use so many concurrent streams do not overshoot. Infrastructure budget interventions: pause optional batch GPU expansion, enforce approved cap on new expensive experimental model deployments, or require human signoff for commitments. Critical exception: do not make a cloud billing account disablement the normal cost guardrail for an always on inference system. It can interrupt all dependent workloads and may not reverse instantly. Treat global disablement as an emergency manual process with clear blast radius approval. 10. Production incident: an automated budget action causes an outage Scenario: A budget alarm forecasts 110% of monthly cloud spend and invokes a script that terminates five “idle” GPU instances. Two of those GPUs are required for regional failover and one serves a low traffic premium customer. A follow on zone fault leaves insufficient capacity; p95 TTFT rises sharply and premium requests fail. Root cause: Automation looked at GPU utilization, not contracted capacity and ready failover headroom . Budget notification was mistaken for proof a specific resource was safe to remove. Resource ownership and active stream/zone policy were not rechecked at execution. No canary reduction or rollback verification existed. Remediation: disable the destructive workflow, restore healthy capacity, verify premium user SLOs and DR posture, then implement protected floors and a policy risk matrix. Run automated deletions only on labeled, disposable batch capacity after proving no active work remains. Prevention: dry run every scale down policy against peak/load and zone failure simulations, require approval to alter DR/interactive minima, and alert if the planned change reduces N−1 capacity. 11. Measure whether the automation really saves money Savings evidence should show: Measured GPU hours removed, actual cost basis and any continuing reserved/committed charges. Completed SLO conforming original requests and their unit cost before/after the action under comparable demand. Added model warmup, storage/egress, restart, checkpoint and operator expenses. Number of actions approved/rejected/rolled back and time from detection to safe resolution. Customer impact: p95/p99 TTFT/ITL, full stream completion, errors and fair share/tenant violations. Policy quality: false interventions, rule conflicts, expired exceptions, attempted prohibited actions and audit completeness. An idle GPU under a non cancellable commitment may produce zero immediate cash saving when shut down, even if utilization improves. Report realized bill reduction separately from avoided variable usage and future commitment optimization . 12. Security, audit and governance Treat budget notifications and metrics as data, not authority . Verify signature/source, sanitize external fields and re evaluate current state. Require signed/versioned policies, code review, testing and bounded rollout. Authenticate tenant IDs at the gateway; do not allow user chosen cost center headers to grant a budget override. Audit who approved changes, what version of a rule ran, affected resources, and measured rollback/SLO effect. Restrict remediation privileges by workload class, region and action type; no wildcard “terminate anything expensive” capability. Distinguish development experimental budgets from protected production commitments. Avoid storing private prompts, tool arguments or credentials in cost audit events. Document exception expiry and a path for customers/operators to request review of unexpected quota denials. 13. Interview ready answer “I'd implement a versioned policy controller with hard invariants for tenant contracts, data residency, model quality, SLOs and DR capacity. Deterministic token and retry caps run at the authenticated inference gateway; Kubernetes OPA can reject noncompliant resource changes. Cost alerts trigger a dry run evaluation rather than a blind shutdown. Small, reversible batch pool adjustments may run automatically with hysteresis, cooldown, fencing, and SLO rollback gates; protected GPU/region/model changes require human approval. I'd prove savings through actual GPU bill differences and cost per successful request while measuring customer impact.” Research references: AWS — Configuring Budget Actions Google Cloud — Budget Alerts and Their Limits Google Cloud — Programmatic Budget Notifications Google Cloud — Eligible Service Spend Caps Open Policy Agent — Kubernetes Admission Control FinOps Foundation — Usage Optimization
How do you evaluate and enforce cost, latency, quality, and reliability trade-offs when selecting LLM inference deployment configurations?
Selecting an LLM inference deployment configuration is a constrained, multi objective engineering decision . A candidate is not “best” because it has the lowest GPU price, highest tokens/second, shortest average latency or highest offline accuracy in isolation. It must satisfy mandatory quality, safety, availability, tenant/data residency and tail latency limits under realistic production demand ; only then should the team compare the total cost per successfully completed SLO conforming original request or another contract appropriate objective. A configuration includes the model and tokenizer , quantization and KV dtype, GPU SKU and count, serving engine/version, tensor/pipeline parallelism, batching and prefill budget, prefix caching, routing, replicas and regions, autoscaling and fault tolerance policies. These choices interact: optimizing one axis may worsen another. This lesson synthesizes established techniques without duplicating them. Question 258 covers constructing realistic benchmarks, Question 260 covers unit GPU economics, Question 264 covers live rollout and rollback, Questions 273–275 cover fleet scheduling/cost and Question 279 covers cost remediation policies. New learning objective: a reproducible decision framework that discards infeasible configurations, identifies Pareto efficient candidates, checks uncertainty and cohort specific constraints, and enforces release and post release contracts. 1. Define the service contract before measuring Write objectives per workload class ; the chatbot and overnight summarizer should not be evaluated using one aggregate winner. Quality: task accuracy/acceptance, groundedness for RAG, code/test pass rate, structured output validity, appropriate tool call behavior and safety violations. State whether a metric is a noninferiority floor versus a required improvement. Latency: client observed p95/p99 TTFT, per request TPOT, stream finish time and deadline miss rate, including prompt length slices. Do not substitute average latency or engine internal timing for user visible time. Reliability: original request completion, timeout/5xx/stream truncation, failure domain resilience and tested recovery time. A successful first byte does not mean a complete successful stream. Cost: GPU/host, model loading, reserved unused capacity, storage/network, retries and observability divided by successful acceptable output, with price and billing assumptions. Policy constraints: tenant isolation, geographic residency, model licensing, version/feature compatibility, approved fallback, budget and availability of the required GPU supply. Operating envelope: peak and burst arrivals, input/output token distributions, prefix reuse, long context incidence, tenant priority and N−1/node loss conditions. Key principle: safety, legal, mandatory quality and latency/reliability minima are hard constraints , not negotiable inputs to a single weighted score. 2. Build a comparable candidate matrix For each candidate, record: 1. Exact model weight, adapter/tokenizer and chat template hashes; model specific prompts and decoding parameters. 2. GPU count, type, topology, tensor parallel degree, utilization/reservation policy and region. 3. Engine container, kernel/driver, KV cache dtype/capacity, chunked prefill, active sequence and batched token limits, speculative decoding or prefix caching. 4. Warm/cold behavior, versioned benchmark dataset, load generator and timestamps. 5. Same offered traffic, request mix, quality labels/rubrics, time window boundary, SLOs and expected failure scenario. 6. Aggregated and per slice metrics, confidence intervals and provenance for every decision. Use independent samples or matched paired prompts for quality. For throughput/latency, keep arrival patterns and user visible timing measurement consistent; do not quietly benchmark one candidate at a lighter load . The vLLM serving benchmark allows joint TTFT, TPOT and E2E goodput targets. NVIDIA's AIPerf describes configurable, production like arrival patterns and percentile latency reports; MLPerf Inference documents workload specific reference quality and server latency constraints. Vendor benchmarks are useful references but cannot replace application specific quality and hardware/load tests . 3. Beginner friendly numerical example: choose the feasible option Assume a production assistant has 3,600 original requests per hour , evenly offered across the test, and three proposed GPU configurations. The predeclared minimums are: Quality pass rate at least 94% on an independently labeled task suite. p95 client TTFT at most 1.0 second . p95 request level TPOT at most 50 ms/token . Original request fully completed success rate at least 99% . N−1 capacity must continue to serve the required peak load within its agreed degraded mode SLO. All values below are hypothetical teaching measurements , not real model benchmarks. Candidate A — cheapest hourly GPU bill $9/hour ; quality 92% ; p95 TTFT 0.8 s ; p95 TPOT 45 ms ; completion 99.5% . Decision: REJECT because quality is below the 94% hard floor , even though it is cheap and fast. Candidate B — high nominal throughput $12/hour ; quality 95% ; p95 TTFT 1.4 s ; p95 TPOT 40 ms ; completion 99.7% . Decision: REJECT because TTFT exceeds the 1.0 second hard ceiling , even if decoded tokens/s is high. Candidate C — feasible production deployment $15/hour ; quality 95% ; p95 TTFT 0.9 s ; p95 TPOT 46 ms ; completion 99.5% ; independently passes N−1 tests. Suppose 3,528 of 3,600 original requests are complete, correct under the defined acceptance rubric, and meet the joint per request latency objectives . Joint success fraction = \3,528/3,600 = 98%\. SLO and quality compliant goodput = \3,528/3,600 seconds = 0.98 requests/s\. Cost per joint successful original request = \$15/3,528 ≈ $0.00425\ ( 0.425 cents ). C is the only feasible choice of these three , given the contractual thresholds; it is not universally optimal. The quality pass rate of 95% is measured on an evaluation suite , whereas the 98% joint rate is illustrative traffic acceptance data; do not assume they are interchangeable or mathematically derive the joint rate by multiplying independent marginal metrics. Important: percentile TTFT and TPOT thresholds do not mean that exactly 95% of requests meet both ; measure joint request level conformance directly. Passing fleet wide aggregate statistics is insufficient if a critical tenant or long context cohort fails. 4. Why a weighted average score can be dangerous A team might compute: \score = 0.4 × cost score + 0.3 × quality score + 0.3 × latency score\. This can wrongly favor a configuration that is 10% cheaper yet violates a safety or reliability requirement. Better decision order: 1. Hard policy/security eligibility: approved data region, tool safety, model and licensing, supported GPU/runtime. 2. Quality noninferiority and critical slice gates: no unacceptable task, safety, rare user or long context regression. 3. Operational SLO and recovery gates: TTFT/TPOT, success, full stream completion, N−1 readiness and restore path. 4. Among feasible candidates , compare dollars per validated SLO outcome, risk margin, operational complexity, capacity availability and energy if relevant. 5. Escalate trade offs to accountable owners only where a soft objective can legitimately be relaxed; never silently relax a hard requirement. A Pareto frontier contains candidates where no other candidate is at least as good on all selected objectives and strictly better on one. This helps narrow the search but does not remove hard floors or choose between policy incomparable alternatives automatically. 5. Design a trustworthy evaluation protocol Start with a versioned, held out quality corpus containing representative code/RAG/chat/structured output tasks and critical failure cases. Test candidate model/quantization changes using paired examples , preserving relevant decoding settings and assessing stochastic variability. Blind manual review can complement deterministic scoring. Benchmark with a realistic open loop request arrival process across multiple offered load levels; include bursts, long prompts, varying output lengths, cancellations and mixed tenant traffic. Keep warm cache and cold cache conditions explicit. Prefix reuse can make a repeated benchmark look better than a new production request distribution. Evaluate p95/p99 TTFT, request level TPOT, joint SLO goodput, all request failure rate , GPU hours and KV preemptions. In vLLM, ITL gaps and per request TPOT are not identical under multi token emission, such as speculative decoding. Test failure scenarios: one replica lost, a GPU node restart, model cold start, decode/prefill worker loss and regional fallback where required. Run repeated trials or controlled bootstraps to estimate uncertainty, and state minimum acceptable per slice label/request counts. A difference of two failures in 40 requests is not convincing evidence of a true 5 percentage point change. Compare the same version configuration across repeats; document source revision, hardware, environment, offered load and sample counts to make results reproducible. Goodput caution: vLLM's built in goodput checks TTFT/TPOT/E2E constraints where configured, but it does not automatically evaluate your application’s factual correctness or safety . Combine quality and SLO results using an authorized application level evaluator; report the overlap (joint success) rather than summing independent marginals. 6. Cost modeling must include idle and failure overhead For each candidate, estimate: \\\text total effective cost = billed GPU hours + storage + network + control plane + reserved or warm standby cost + retries and recomputation cost + operational overhead \\\ Then: \cost per good request = total effective cost / joint successful original requests\. Keep provider discounts and committed GPU hours in the actual chosen billing cost basis ; shutting down a committed instance may not save the assumed $/hour immediately. Inability to acquire a scarce GPU when needed is also a reliability risk. Cost sensitivity test: vary projected request rate, prompt length distribution, output length, cache reuse, spot interruption rate, model startup and GPU price. A winner that saves only 3% in the median case but loses 40% under a long context burst may be fragile. 7. Compare model and serving configuration together Common controls and their trade offs: Model size/swap: smaller models can reduce memory and latency, but require task specific accuracy and safety acceptance. Weight quantization: shrinks weights and changes kernels; may improve throughput or reduce GPU count, but can cause precision sensitive quality degradation. KV quantization: increases possible cache capacity, but may affect quality and transfer/attention behavior; check long context and rare tasks. Batch size/token budget: improves occupancy until queueing, preemption or decode interference damage user experience. Chunked prefill: can reduce interference from long prompts but affects batch scheduling and TTFT/ITL trade offs. Tensor parallelism: fits large models and increases compute resources but adds collective communication and deployment cost. Speculative decoding: can accelerate accepted token generation but introduces draft model work, memory demand and verification overhead. It is not always better. Caching: accelerates repeated prefixes but changes KV memory pressure and benchmark interpretation. GPU SKU and replica count: faster GPU does not guarantee lower cost per good request; price, topology, headroom and demand shape matter. Spot/reservations: reduce or stabilize certain cost/capacity risks, but interruption and standby requirements still apply. Change only a small, measured set of variables per experiment where possible. Use controlled configuration sweeps when interactions matter; avoid attributing a compound improvement to one setting without evidence. 8. A release gate with concrete decision states An effective evaluator emits: PASS: all hard constraints met with adequate evidence; cost improvement verified and operational risk acceptable. FAIL: confirmed breach of quality, security, SLO, reliability or mandatory cohort requirement. INCONCLUSIVE: insufficient labels/requests, contaminated benchmark, unstable telemetry, or confidence interval too wide. EXCEPTION REQUIRED: a documented soft trade off needs business/engineering owner signoff; hard safety/residency violations remain prohibited. Example pseudocode: \\\python def decide(candidate, policy): if not candidate.provenance complete: return "INCONCLUSIVE" if not candidate.region allowed or not candidate.safety pass: return "FAIL" if candidate.critical slice failure: return "FAIL" if candidate.quality lower confidence bound < policy.min quality: return "INCONCLUSIVE OR FAIL" if candidate.p95 ttft ms policy.max ttft ms: return "FAIL" if candidate.p95 tpot ms policy.max tpot ms: return "FAIL" if candidate.original success rate < policy.min success: return "FAIL" if not candidate.n minus one pass: return "FAIL" return "PASS FEASIBILITY" \\\ Note: \INCONCLUSIVE OR FAIL\ needs an explicit predeclared statistical decision policy. This is simplified illustrative pseudocode, not a ready to deploy classifier. A deterministic threshold breach can fail immediately, while uncertainty at a noninferiority boundary may require more samples. 9. Production incident: cheap quantization wins a benchmark, loses production Incident (hypothetical): A team selects an FP8 quantized deployment because offline tests show 30% better output tokens/s and a 20% lower GPU hour rate . After release, long context RAG users experience more citation mistakes, peak hour TTFT rises, and retries increase the number of backend attempts per original request. Investigation: 1. The quality suite contained too few long context and citation groundedness examples; only aggregate answer quality was evaluated. 2. The benchmark used a fixed prompt set with high prefix cache reuse and lower offered concurrency than the actual peak. 3. The proposed configuration had less usable KV headroom and preempted more long requests. 4. The cost spreadsheet compared GPU price rather than dollars per SLO compliant, validated original request , excluding retries and lost answers. Fix: stop promotion/roll back to the approved baseline; add long context citation and structured output slice gates; replay realistic traffic with matched cache conditions; compare GPU+retry+standby cost and joint success. Prevention: require model quality and systems performance evaluation together, a reproducible configuration manifest, and canary release gates that enforce both hard safety thresholds and measured workload specific goodput. 10. Rollout, verification and ownership Begin with offline functional and quality gates ; ensure critical tasks and safety policies pass. Run matched open loop serving benchmarks under normal, burst, peak and failure envelope traffic. Use shadow requests for side effect free production path validation; candidate output must never trigger external effects during shadow testing. Start a small canary , compare stable/candidate by model, workload cohort and tenant; monitor quick operational SLIs and slower quality evidence. Stop/rollback for predeclared hard regression triggers. Do not promote just because short window p95 and cost look good when delayed labels or rare group coverage are missing. After promotion, keep continuous benchmarking and SLO dashboards , and re run evaluations on engine upgrade, new GPU SKU, model or tokenizer changes. Define accountable owners: ML evaluation owner for quality, platform/SRE for SLO/recovery, security for policy, FinOps for cost basis and a final approver for cross domain trade offs. 11. Common mistakes and limitations Comparing GPU hour price alone rather than effective end to end service cost. Averaging percentiles across worker pools or concluding per request joint success from separate TTFT/TPOT percentiles. Counting only HTTP completed requests and ignoring cancellations/rejected work. Assuming a uniform model wide quality score protects rare high risk customer tasks. Treating an LLM as judge score as a definitive truth without calibration and human review of important decisions. Measuring an artificially warm prefix cache and extrapolating the result to cold production traffic. Choosing the greatest GPU utilization even when it worsens latency isolation or failover reserve. Combining every criterion into a scalar weighted score that hides a hard violation. Claiming that a 1% gain is real without adequate trials and statistical uncertainty. Treating successful canary traffic as proof of full failure zone or multi region recovery. Allowing a cost bot to silently change the model/quantization/runtime after approval; the approved configuration fingerprint is the contract. 12. Interview ready answer “I'd enumerate feasible model/GPU/engine configurations, define per workload hard quality, safety, reliability and tail latency gates , then replay matched realistic traffic with correct cache and arrival conditions. I'd measure joint SLO goodput and quality alongside total cost per successful original request, including standby and retry work. I'd reject infeasible candidates before ranking costs, use Pareto analysis for the remaining trade offs, assess statistical uncertainty and critical slices, and promote through shadow/canary gates with a versioned configuration and rollback plan.” Research references: vLLM — Serving Benchmark and Joint Goodput vLLM — Benchmark Metrics and Prefix Cache Interpretation NVIDIA — Benchmarking LLM Inference at Scale with AIPerf (2026) MLCommons — MLPerf Inference Benchmarks and Accuracy/Latency vLLM — Continuous Benchmarking Dashboard