How do you optimize GPU utilization across mixed interactive and batch LLM inference workloads while preserving latency isolation?

Mixed-workload LLM inference scheduling increases useful GPU work by combining latency-sensitive interactive requests with throughput-oriented batch requests—without letting batch prefill, long decodes, or KV-cache pressure violate interactive service-level objectives (SLOs). The challenge is to reclaim genuinely idle GPU capacity while providing a measurable, enforceable isolation boundary. A high aggregate GPU-utilization percentage is not success if chat users experience worse time to first token (TTFT), inter-token latency (ITL/TPOT), errors or truncated streams.

This lesson is a production design continuation of Question 220 (why chatbot and batch GPU demands differ), Question 251 (tenant priority/fairness), Question 273 (Spot/reserved GPU purchasing) and Question 274 (batch deadlines/checkpointing). Here the specific focus is GPU time and memory isolation, token-aware co-scheduling, borrowing idle capacity, eviction semantics and measurable safe utilization.

1. Why interactive and batch work interfere

• Interactive chat: often short or bursty prompts, unpredictable arrivals, streaming responses and strict TTFT/ITL targets. It needs quick admission and reliable KV memory headroom.
• Batch summarization/research: may generate many tokens from large prompts, tolerate higher queue delay, and prefer large batches or sustained GPU occupation to maximize tokens per GPU-hour.
• Prefill versus decode: prompt prefill tends to be compute-intensive; decode frequently depends on memory-bandwidth/KV-cache access. Mixing phases intelligently can help device efficiency, but very large prefills can delay already-streaming users.
• Long-context requests: consume more KV blocks and may cause preemption/recomputation even if the immediate GPU utilization looks moderate.
• Shared network/CPU/storage: two pools on different GPU allocations can still interfere through checkpoint downloads, tokenizer CPU, router queues or KV-transfer links.
• Output-length uncertainty: estimated tokens are planning signals, not guarantees. Enforce explicit request budgets and monitor actual progress.

2. Choose the isolation architecture

Option A — separate GPU serving pools (strong baseline)

• Maintain an interactive pool sized for forecast SLO demand, burst headroom and required replica failure tolerance.
• Keep a separately admission-controlled batch pool, potentially on cheaper or interruptible GPUs where jobs are restartable.
• Router sends each class only to approved compatible model/version/region pools. This is easiest to reason about and protects chat from batch-related OOM and scheduling pauses.
• Cost: interactive GPUs may sit partly idle when chat demand is low.

Option B — shared serving engine with priority-aware token scheduling (controlled borrowing)

• Use a serving engine supporting per-request priorities, maximum token budgets, preemption and KV-aware admission.
• Let batch work borrow surplus engine capacity only while observed interactive latency and queue age remain healthy.
• Protect interactive decode steps, limit long-prefill chunk size and cap active batch sequences and KV-block use.
• Risk: logical scheduling does not guarantee strict GPU memory/compute isolation; an oversized batch may already hold KV state or occupy kernels.

Option C — hardware-partitioned GPU resources

• On supported NVIDIA GPUs, MIG can partition hardware into GPU instances with isolated resources, but each slice has limited HBM and compute; many large LLMs will not fit.
• GPU time-slicing may improve access for smaller models/jobs, but NVIDIA explicitly documents no memory isolation and no throughput guarantee with time-slicing. It is not a hard SLO boundary.
• Multi-Process Service (MPS) is another sharing approach with its own support and isolation constraints. Benchmark the exact configuration and ensure tenant-security requirements.
• For strict isolation of large models, whole-GPU or whole-replica pools are usually a safer starting point.

3. Beginner-friendly numerical example: utilization versus SLO

Assume four equivalent GPU replicas are deployed:

• Baseline interactive demand uses 40% of the fleet's GPU capacity during quiet periods.
• Batch backfill can use idle capacity but has no user-facing TTFT requirement.
• Interactive SLO: p95 TTFT ≤ 1 second, p95 ITL ≤ 50 milliseconds and request success ≥ 99%.
• The results below are illustrative controlled measurements, not published GPU benchmark claims.

A. No batch borrowing:

• GPU active utilization = 40%.
• Interactive p95 TTFT = 500 ms; p95 ITL = 35 ms; success = 99.9%.

B. Unrestricted batch borrowing:

• GPU active utilization = 90%.
• Interactive p95 TTFT = 1,800 ms; p95 ITL = 75 ms; success = 98%.
• Conclusion: the fleet is busier, but the interactive service violates *all three* objectives; calling this an optimization would be misleading.

C. Token-aware, limited borrowing:

• GPU active utilization = 72%.
• Interactive p95 TTFT = 700 ms; p95 ITL = 42 ms; success = 99.5%.
• This configuration satisfies all three example goals, while increasing GPU utilization by 32 percentage points versus the quiet baseline (a relative 80% increase, \(72−40)/40 = 0.8\).

Do not choose 72% as a universal target. Safe GPU occupancy changes with model, token length distribution, active sequences, KV memory, quantization, concurrent tenants and underlying hardware. A setting that passed at 40 interactive requests/s may fail during peak demand.

4. A simple capacity-borrowing policy

Define:

• \L_i\: observed interactive offered token-work rate and active/request backlog.
• \C_safe\: load-tested safe capacity under the real interactive mix (not theoretical GPU FLOPS).
• \H\: reserved headroom for interactive bursts, model warmup and worker failures.
• \B_budget\: permitted batch token, queue and KV budgets at the current condition.

A conceptual controller:

1. If interactive p95 TTFT and ITL have healthy margin, queue age is low, KV occupancy below the tested pressure point, and spare capacity exists: allow limited batch tokens/requests.
2. If interactive queue-token pressure rises or SLO burn increases: immediately stop new batch admissions, reduce/stop future batch prefill chunks and drain or preempt *eligible* batch work.
3. If KV pressure approaches unsafe levels: reclaim batch capacity only using the serving engine's supported preemption/cancellation mechanisms. A reclaimed batch KV cache may require recomputation; never assume free/instant migration.
4. When interactive conditions recover for a sustained quiet interval: slowly reopen batch borrowing with hysteresis and a cap.
5. Always enforce per-tier maximums: batch concurrency, pending token work, allowed output length, maximum memory/KV allocation and deadline/retry budget.

This is a conceptual policy, not a documented built-in vLLM autoscaler. GPU utilization alone should not drive the decisions.

5. Understand vLLM scheduling controls

Current vLLM scheduler documentation describes:

• FCFS and priority policies: request priority is a numeric field; lower numeric values receive higher priority when the priority policy is configured. Priorities have semantics within the engine, not automatically across every router or model replica.
• \`max_num_batched_tokens\`: caps a token budget per scheduling step. It influences how many prefills can mix with ongoing decodes and can materially change TTFT versus ITL.
• \`max_num_seqs\`: bounds active sequences. More concurrency can improve throughput but also increases KV pressure and may harm tail latency.
• Chunked prefill: splits oversized prefills into smaller pieces to interleave with ongoing generations, improving the chance to protect decode responsiveness.
• Preemption: when KV cache is constrained, the scheduler may preempt work; in vLLM V1 this commonly requires recomputation. Even if a batch request is preemptible, its lost GPU work is a cost.
• The vLLM V1 scheduler is token-budget-based and does not require a strict global alternation between separate prefill and decode stages. Avoid presenting prefill/decode batching as permanently independent fixed phases.

Version caveat: vLLM scheduling defaults and optional flags evolve. Verify the deployed version, supported request priority field and model-serving endpoint behavior before copying flags.

6. Protect latency with two layers

Layer 1 — routing and pool-level safeguards

• Assign a minimum ready interactive replica pool and enforce per-class admission before the GPU scheduler.
• Reserve an interactive share of compatible GPUs or token work; stop scheduling new batch tasks when forecast peak load approaches.
• Separate queue budgets, auth/tenant policies and timeouts. Batch deadlines should not override a customer's latency-critical request.
• Prefer quick-start, warm interactive capacity so scaling and routing changes do not create cold-start latency.

Layer 2 — per-engine safeguards

• Keep ongoing interactive decode requests responsive with token-budget and supported priority policies.
• Break long-prefill batch requests into bounded chunks; tune chunk sizes against representative short and long prompts.
• Set batch concurrency/KV thresholds based on observed preemptions and memory.
• When the engine cannot offer sufficient class isolation, move batch work to another replica/pool rather than assuming priorities are enough.

Priority is not the same as GPU kernel preemption: a high-priority request cannot necessarily interrupt an already-running CUDA kernel or immediately reclaim live KV state.

7. Production incident: 95% GPU utilization, poor chatbot experience

Scenario: A team colocates document summarization jobs with an interactive chatbot because the GPUs appear underutilized at night. When batch requests with 32k-token prompts start, GPU usage reaches 95%. Streaming chat p95 ITL worsens from 40 ms to 110 ms and p95 TTFT from 600 ms to 2.4 seconds.

Root-cause debugging:

1. Slice TTFT, ITL and queue wait by interactive versus batch, request length and worker.
2. Trace large batch prefills through scheduler steps; they fill token budgets and delay interactive prefill admission.
3. Inspect KV-cache occupancy and preemption/recomputation counts; long batch generations leave less headroom for chat.
4. Verify whether the router admits batch work onto the very replicas required for critical user streams.
5. Compare a no-batch control replay to progressive borrowing rates.

Fix: partition an interactive baseline; bound batch prefill chunks and active KV work; apply engine-aware priorities/queues; stop batch admission when interactive queue/SLO pressure increases; ensure unfinished batch work is checkpointable at shard boundaries if moved to other workers.

Prevention: run mixed-class load tests at normal and peak interactive demand, and enforce per-class SLO gates and safe-borrowing limits before enabling production co-location.

8. Evaluate efficiency correctly

Track three groups of metrics:

• Customer protection: interactive p95/p99 TTFT, ITL, successful stream completion, error rate and SLO-compliant goodput.
• Batch usefulness: completed and validated batch items/hour, checkpoint redo work, job lateness, preemptions, effective GPU cost per completed batch item.
• Resource efficiency: GPU utilization, prompt/output tokens per second by workload class, KV-cache occupancy, memory pressure, preempted/recomputed tokens, CPU/network bottlenecks, total GPU-hours and cost per accepted/successful outcome.

Report outcomes under equivalent *offered* interactive traffic, with bursty arrivals and realistic prompt/output length distributions. Do not compare “utilization before” and “utilization after” if chat load differs.

An appropriate optimization objective is: \maximize validated batch throughput per incremental GPU-hour\ subject to interactive TTFT/ITL/success SLOs, tenant safety and batch deadline constraints.

Batch speedups that cause user-visible SLO failures are not feasible solutions, regardless of aggregate throughput gains.

9. An engineering rollout checklist

1. Establish the interactive-only p95/p99 baseline and forecast peak plus failure headroom.
2. Load-test dedicated pools to estimate class-specific resource consumption.
3. Define hard isolation requirements (tenant, model, region, GPU memory, KV boundaries and support for shared hardware).
4. Start with dedicated pools; add controlled borrowing on a small portion of eligible replicas.
5. Increment batch concurrency or token budget in measured steps, retaining enough sample volume for tail latency.
6. During each step, inspect interactive TTFT/ITL, queue-token age, KV occupancy and preemptions.
7. Require a fast batch-admission stop switch; validate that stopping new batch work actually restores interactive SLOs within the required time.
8. Test an interactive burst and one-GPU/replica failure while batch runs.
9. If batch jobs are interrupted, use durable shard checkpoints and idempotency from Question 274.
10. Continuously report incremental batch completion versus its impact on interactive SLO-compliant goodput and total cost.

Limitations: queue priorities do not guarantee strict compute isolation; hardware sharing may be unavailable on the chosen GPU profile; heavy KV reuse and large dynamic context lengths complicate fairness; and global route-level SLO controls require integration beyond one local engine.

10. Interview-ready answer

“I'd protect a ready interactive GPU baseline first and benchmark real p95 TTFT/ITL and goodput under peak traffic. Batch inference can use separate cheaper pools by default, then borrow tested surplus capacity via token-aware queues, bounded chunked prefills, priority scheduling and per-class KV/concurrency limits. If chat queue age or SLO burn rises, batch admissions stop and the system safely drains/preempts eligible batch work. I'd avoid using GPU time-slicing as a hard isolation guarantee and optimize verified batch completions per GPU-hour subject to user-facing latency and reliability targets.”

Research references:

• vLLM — Scheduler Configuration and Priority Policy
• vLLM — Architecture and Token-Budget Scheduler
• vLLM — Agentic Mixed Traffic Scheduling (September 2026)
• NVIDIA — GPU Time Slicing and Its Isolation Limits
• NVIDIA — MIG Partitioning
• NVIDIA Dynamo — Inference Scheduler and GPU Fleet