How do you optimize LLM inference cost using GPU reservations, spot capacity, and workload-aware scheduling without violating SLOs?

Cost-aware LLM inference scheduling places requests on the cheapest *eligible and sufficiently reliable* GPU capacity while maintaining each workload's latency, correctness, data-residency and availability objectives. The foundation is a protected serving baseline for latency-sensitive requests, with flexible surplus capacity for deferrable or interruption-tolerant workloads. Buying the cheapest GPU-hour is not necessarily cheapest: interrupted generations, failed SLOs, cold-start delays and wasted prompt tokens can make the cost per successfully completed SLO-compliant request worse.

This lesson follows Question 260 (GPU sizing and unit economics), Question 251 (tenant fairness), Question 261 (autoscaling), Question 263 (stream draining), and Question 272 (regional capacity planning). Here the new skill is combining purchasing models with workload-class placement, interruption risk, and explicit economic/SLO constraints.

1. Four distinct cloud purchasing concepts

• On-demand: Pay for instances as needed; useful for elastic baseline/overflow when available. On-demand is not by itself a guarantee that scarce GPU instances can be launched instantly.
• Capacity Reservations: Reserve specified eligible compute capacity in a defined zone, helping ensure the necessary instances can be launched. This is a capacity assurance mechanism, not inherently a discount; unused reserved capacity may still be billable. Actual product options and terms vary.
• Savings Plans / Reserved Instances / commitments: Reduce eligible compute billing under a usage/configuration commitment. A Savings Plan does not by itself reserve GPU capacity. Understand term, region/family flexibility, underutilization, and the effects of commitments when architecture changes.
• Spot / preemptible capacity: Discounted spare capacity that the provider can reclaim. Availability and interruption behavior are variable. Excellent for checkpointable, repeatable batch work; risky as the sole pool for long, real-time streaming sessions.

AWS specifics: AWS Spot normally provides a two-minute interruption notice; an earlier rebalance recommendation may be available, but no two-minute window is a promise that all generations will finish. AWS separately documents Capacity Reservations and Savings Plans. Google Cloud differs: Spot VMs can be configured for a 120-second advance preemption notice, while the default is 0-second dedicated notice followed by a best-effort shutdown of up to 30 seconds. Do not apply the AWS two-minute rule to every provider.

2. Classify requests before choosing a GPU pool

Tier A — interactive with strict SLO

• User-facing chat, tool-using assistants and interactive coding need low TTFT/TPOT, predictable success and stable streaming.
• Prefer adequately sized, warmed noninterruptible/contracted or reserved capacity for the protected baseline. Allow Spot only when failure is explicitly tolerable and reliable fallback/readmission is tested.

Tier B — elastic but still user-facing

• Lower-priority chat or asynchronous APIs may tolerate longer deadlines or model fallback.
• Route to cost-efficient GPUs only when measured queue and token workload indicate sufficient headroom. If a risky pool becomes unhealthy, reject/defer or fall back within the stated policy.

Tier C — deferrable, restartable batch

• Offline summarization, embeddings/backfills where applicable, evaluation sweeps or bulk document generation can often tolerate interruption.
• Favor Spot/preemptible resources with checkpointing, idempotent job IDs, retry budgets and a deadline-aware fallback to on-demand.

Tier D — highly sensitive or constrained

• Tenant isolation, strict residency, model licensing, regulated workflows or minimum-accuracy requirements may prohibit some pools, regions or fallback models regardless of cost.

The scheduler should decide based on workload eligibility + predicted token work + SLO deadline + actual ready capacity, not on cloud price alone.

3. Beginner-friendly numerical example: two capacity choices

Assume a production workload receives 20 interactive requests/s and 8 batch requests/s. A compatible one-GPU serving replica safely handles 5 requests/s *for this particular measured mix and SLO*, after planned per-replica headroom. Illustrative GPU prices:

• On-demand GPU: $3.00/hour.
• Spot GPU: $1.00/hour.
• All quantities and prices are educational assumptions, not current cloud quotes.

Plan A — all on-demand:

• \ceil((20 + 8)/5) = 6 GPUs\.
• Hourly GPU cost = \6 × $3 = $18/hour\.

Plan B — workload-separated pools:

• Interactive baseline: \ceil(20/5) = 4\ on-demand GPUs.
• Batch pool: \ceil(8/5) = 2\ Spot GPUs.
• Hourly GPU cost = \4 × $3 + 2 × $1 = $14/hour\.
• Nominal savings = \($18 − $14)/$18 ≈ 22.2%\.

Important assumptions: the model and SLO-safe rate really are comparable across pools; jobs can use dedicated capacity without unreasonable imbalance; Spot GPUs are continuously available; the four on-demand GPUs have enough protected serving headroom; and storage/network costs are excluded. In reality, long batch generations can have very different token work. Independently benchmark each tier instead of extrapolating the same 5-RPS capacity to unlike workloads.

4. Why Spot interruption can erase a discount

Continue the example: batch pool contains two Spot GPUs costing $2/hour, but preemption makes only 80% of attempted batch work yield useful successfully finished output (after accounting for wasted work and before any retry). If throughput is otherwise the same, the *naive effective GPU cost per successful unit* scales roughly as: \$2 / 0.80 = $2.50\ per equivalent hour's successful work, not $2.

The formula is a simplified normalization; true cost also includes the time of retries, checkpoint writes, cold starts, duplicate tokens, quality verification and fallback capacity. A real system should compute:

\effective unit cost = (all billed GPU-hours + retry/checkpoint/storage/network + operational overhead) / successfully completed eligible work\

For an online SLO, replace the denominator with successfully finished SLO-compliant requests. Work rejected, interrupted or timed out must not be silently excluded from availability reporting.

5. Capacity floor and burst protection

Use measured goodput-per-replica from realistic load tests:

1. Let \D_interactive\ = forecast peak interactive offered RPS (or a calibrated token-work rate).
2. Let \c_protected\ = sustainable SLO-compliant rate per eligible protected replica.
3. Minimum protected replicas \R = ceil(D_interactive / c_protected)\ with additional requirements for N−1 failures, zoning and burst headroom.
4. Use discounted/interruption-tolerant capacity only for demand that can be deferred or safely displaced.
5. Allow overflow to protected capacity only if interactive headroom remains; never let a cheap batch surge consume interactive queue/KV budgets.
6. Reserve or prewarm capacity as required by actual model-loading time. A GPU available on paper but still loading weights is not ready service capacity.

Failure case: if the four on-demand replicas in the worked example each handle five RPS, interactive traffic of 20 RPS fully consumes their planned safe rate. Therefore there is no guaranteed protected slack for a lost Spot batch pool or a simultaneous replica failure. The plan is financially attractive only if batch work can wait and the strict interactive tier's additional reliability policies are addressed.

6. Workload-aware scheduling policy

A practical scheduler may use:

• Hard eligibility filters: approved region, GPU memory/model fit, hardware topology, tokenizer/adapter/version, encryption, tenant policy and tool safety.
• Request classification: online/async, deadline, priority, estimated prompt and output tokens, streaming flag, expected prefix reuse and cancellation semantics.
• Capacity checks: available ready model replicas, queue token work, active sequences, KV blocks, recent TTFT/ITL, goodput and health.
• Economics: effective $/GPU-hour, remaining commitments, interruption risk, data egress, cold-start cost and estimated retries.
• Routing: online Tier A to protected low-latency pools; batch Tier C to cheaper interruptible pools subject to deadline and progress checkpoints.
• Escalation: nearing deadline → promote eligible batch requests to reliable capacity only if budget and protected workload headroom permit; otherwise return an explicit deferral/failure status.

Never rely on an uncalibrated prediction of output length as a hard admission fact. Output tokens are uncertain; use conservative estimates, maximum generation limits and ongoing queue signals.

7. Kubernetes placement example

Create appropriately labeled, separately managed protected and spot GPU node pools. Enforce the intended selection using node affinity/selector plus resource requests and, where appropriate, taints/tolerations.

Illustrative batch Job template fragment: \\\`yaml spec: template: spec: restartPolicy: Never nodeSelector: capacity-class: spot tolerations:

• key: "capacity-class"

operator: "Equal" value: "spot" effect: "NoSchedule" containers:

• name: batch-inference

image: registry.example.com/approved-llm-job:PINNED_DIGEST resources: limits: nvidia.com/gpu: 1 \\\`

This is a conceptual manifest, not a complete deployment: your cluster must actually apply the label/taint, provide GPU scheduling/device plugins, use an authentic digest and configure the full Job, checkpoint storage and security. A toleration only allows scheduling; it does not force placement. The node selector provides the strict class match in this example. Do not schedule critical control-plane components exclusively on interruptible nodes.

8. Interruption handler and state safety

1. Monitor the correct provider-specific preemption signals and capacity-risk recommendations.
2. Immediately stop new assignments to a threatened worker.
3. For a long batch generation, write recoverable progress and a durable idempotency/ownership record at a safe application boundary, if the job supports it.
4. Attempt to drain active streams within the *actual* remaining termination period; classify unfinished streams as incomplete rather than successful.
5. Requeue only jobs whose effects are idempotent or explicitly reconciled. An LLM decoder's KV state on a lost GPU normally does not survive preemption.
6. Restart from an approved checkpoint or from earlier application-level state. Exact token-for-token continuation is not guaranteed, especially with stochastic generation.
7. Back off retries and avoid stampeding the model object store after a large Spot reclamation.

The interruption warning may be absent, delayed or too short for long generations. Never design correctness around guaranteed notice delivery.

9. Cost-aware autoscaling and purchasing

• Baseline commitments: use historical stable usage and forecast confidence to decide what fraction of GPU hours warrants long-term discounted commitments. Excess commitments still cost money when traffic shrinks or models change.
• Capacity reservations: consider when GPU scarcity threatens a strict RTO/SLO; separately evaluate whether discount instruments apply.
• Autoscale batch pools: use job queue depth and estimated queued token work rather than interactive GPU utilization. Scale down empty interruptible capacity promptly after safe checkpoint/termination.
• Model-aware packing: schedule compatible inference jobs to avoid loading too many copies of an enormous checkpoint, while protecting tenant isolation and avoiding KV/OOM contention.
• Heterogeneous hardware: benchmark equivalent model quality and SLO success on each GPU SKU; the lowest $/GPU-hour may have the highest $/successful request.
• Multi-region: the cheapest geographic pool may violate data residency or add enough latency/egress cost to fail the workload's objective.

10. Production incident: batch jobs evict live chat capacity

Scenario: A service puts both user-facing chat and offline summarization on a shared Spot GPU pool. A Spot interruption terminates two workers during a burst. Chat completions stop midstream, the remaining workers receive retries, queue wait spikes, and p95 TTFT increases from 0.8 to 4 seconds. The lower hourly price does not compensate for failed customer SLOs.

Root cause: no protected Tier A capacity; the scheduler ignored workload deadline/restartability and treated Spot GPUs as equivalent to stable warmed replicas.

Fix:

• Move the minimum interactive serving footprint to protected, warm GPUs with explicit N−1 headroom and admission budgets.
• Run only interruption-safe batch work on Spot; isolate queues and preemption handling.
• Implement provider interruption detection, bounded retry and job checkpoint/idempotency.
• Calculate $ per SLO-compliant user request and $ per verified completed batch job, not aggregate GPU-rate discounts.

Prevention: run synthetic full-Spot-pool reclamation drills while interactive traffic stays at peak; verify no critical streaming tier depends on ephemeral capacity, and confirm the job queue recovers without duplicate side effects.

11. Monitoring and common mistakes

Track separately by capacity class:

• Billed GPU-hours, unused commitments/reservations, actual Spot prices, interruptions/hour, job retries and lost compute work.
• Ready serving replicas, time to re-acquire GPUs, checkpoint restore/load time, queued token work and batch job age.
• Original-request p95/p99 TTFT, TPOT, error/rejection/stream truncation, joint SLO goodput and cost per good request.
• Per-tier quality and model version, tenant isolation and data-residency policy violations.

Avoid:

• Assuming Savings Plans imply a guaranteed GPU supply.
• Counting merely requested or still-warming Spot GPU instances as ready capacity.
• Treating every batch request as safely restartable without durable progress and idempotency.
• Making Spot the only production tier for sub-second streaming SLOs.
• Mixing user-facing and batch request classes without scheduling/fairness isolation.
• Treating instance-hour discounts as proven unit-cost savings without lost work, cold starts and failure costs.
• Depending on an AWS-specific two-minute notice across all cloud providers.

12. Interview-ready answer

“I'd benchmark real SLO goodput and create a protected GPU baseline for interactive traffic, using reservations where capacity availability warrants it and commitments only when the expected usage is steady. I'd schedule checkpointable batch inference on Spot/preemptible GPUs with provider-specific interruption handlers, idempotent retries and separate queue budgets. The optimizer should consider actual ready capacity, token-work cost, deadlines, quality and residency rather than hourly price alone. I'd validate savings as total dollars divided by successfully completed SLO-conforming requests and run interruption drills before production.”

Research references:

• AWS — EC2 Purchasing Options
• AWS — Spot Instances and Interruption Notices
• AWS — Capacity Reservations
• Google Cloud — Spot VM Preemption
• Kubernetes — Taints and Tolerations
• GKE — Spot VMs with GPU Node Pools