How do you use SLO-aware routing and latency prediction to choose an LLM inference replica?

SLO-aware routing selects a healthy, compatible model replica based on whether it is predicted to satisfy the request's time to first token (TTFT) and time per output token (TPOT) objectives—not simply which GPU has the fewest requests. An online latency predictor estimates the delays on each candidate from prompt size, prefix-cache reuse, queueing, active decode load, and KV-memory pressure. The router favors endpoints with positive SLO headroom and avoids wasting time on a replica likely to miss the deadline.

Unlike Question 241 (which introduced cache-plus-load-aware routing) and Question 255 (health-aware failover), this lesson focuses on predictive latency models, per-request SLO headroom, uncertainty, routing decision overhead and validating predicted versus actual SLO attainment.

1. Why naive balancing falls short

• Round-robin ignores expensive prompts and uneven active generations.
• Fewest active requests treats a 100-token prompt like a 20,000-token prompt.
• Prefix affinity alone can send a request to an overloaded replica just because the replica has reusable KV.
• Current GPU utilization is a lagging and incomplete proxy for TTFT/TPOT; a busy GPU can still meet SLOs, while a queued workload on another may not.
• A predictor can use richer request/server signals to estimate what the next request will experience on each candidate endpoint, including a hypothetical placement.

2. Inputs to a latency predictor

For each eligible replica collect:

1. Input prompt tokens and an estimated output-token length or current generated-token count where relevant.
2. Prefix-cache match fraction and uncached prefill tokens.
3. Queue depth and input tokens in flight, not only queued request count.
4. Running sequences, GPU concurrency, and remaining decode work if estimable.
5. KV-cache utilization, evictions/preemptions and memory headroom.
6. Hardware/model/adapter compatibility; network/KV-transfer delay for disaggregated deployments.

llm-d documents an online-trained predictor using recent request-level observations, with TTFT features including input length, queue depth, active requests, KV use and prefix match. Its TPOT features also include generated tokens. Current implementations can train an XGBoost-based model online. The prediction's accuracy depends on telemetry quality, traffic coverage and how recently it was updated.

3. Beginner-friendly numerical routing example

Suppose the application requires TTFT ≤ 800 ms and TPOT ≤ 50 ms per output token. The predictor evaluates three compatible endpoints:

• Replica A: predicted TTFT 500 ms, TPOT 65 ms. It has TTFT headroom 300 ms but violates the TPOT target by 15 ms.
• Replica B: predicted TTFT 650 ms, TPOT 35 ms. Headroom is 150 ms for TTFT and 15 ms for TPOT.
• Replica C: predicted TTFT 900 ms, TPOT 25 ms. It misses TTFT by 100 ms, despite excellent predicted TPOT.

Decision: choose B if the product requires *both* targets. A greedy minimum-TTFT router might choose A and harm token streaming; a greedy minimum-TPOT router might choose C and delay the first token.

Important: These are illustrative point predictions, not guarantees. A robust production policy should include predictor error margins and consider p95 probabilities or a measured confidence budget. For example, a 150-ms predicted TTFT headroom does not protect the SLO if recent forecast errors frequently exceed 200 ms.

4. One workable routing algorithm

1. Authorize and filter: require matching model/version, adapter, context, tools, region, security boundary, and readiness.
2. Classify SLOs: determine tenant/workload-specific TTFT/TPOT objectives from trusted policy, with sensible defaults.
3. Predict both metrics for each compatible candidate.
4. Calculate headroom: H_ttft = SLO_ttft − predicted_TTFT; H_tpot = SLO_tpot − predicted_TPOT.
5. Prefer candidates with nonnegative headroom for both. Rank with headroom, predicted violation probability and resource efficiency rather than solely one latency number.
6. If all candidates violate an SLO, use a documented fallback policy: least-bad available candidate when useful, queue/defer within deadline, or reject quickly; do not represent the target as guaranteed.
7. Record actual outcomes and compare predictions against measured TTFT/TPOT to update calibration.

llm-d's latency scoring and headroom filters follow this broad design. The specific score is implementation-dependent; not every router optimizes a simple sum of the two metrics.

5. Worked production incident

Incident: Interactive chat p95 TTFT increases from 650 ms to 1.8 seconds after nightly long-context summarization starts. Round-robin spreads requests evenly by *count*, but one worker handles a high share of 16k-token prefills and another holds many long decode sequences.

Diagnosis:

• Split observed TTFT/TPOT by replica, prompt length, prefix-match rate, queue delay and KV occupancy.
• Compare *predicted versus actual* latencies to see whether the model captures long prompts and decode saturation.
• Check that routing did not concentrate requests on a cache-affine but already busy replica.

Fix: Route based on predicted TTFT+TPOT feasibility, incorporate token workload/KV telemetry, keep short interactive requests within a suitable SLO tier, and admit only when capacity makes the target plausible. Preserve an ordinary load-aware fallback while prediction is unavailable or cold.

Prevention: Test changing prompt/output distributions, replica restarts, sudden batch bursts, prediction model cold starts and mixed tenants. Recalibrate as hardware, inference engine or attention kernels change.

6. Trade-offs and failure modes

• Cold-start model: early predictions may be uninformative until real traffic provides training data; use load-aware routing until enough evidence accumulates.
• Distribution drift: new prompt lengths, model releases, hardware or batch patterns can break learned latency relationships.
• Prediction overhead: scoring every candidate adds CPU work and network calls. With R requests/s and N eligible replicas, prediction demand can be roughly R × N per-request-candidate predictions. For 200 RPS and 8 replicas, that is 1,600 candidate predictions/s before batching/caching and other costs.
• Tail risks: an accurate average model can still underestimate rare queue spikes. Track p95/p99 target attainment and conservative uncertainty margins.
• Feedback loops: routing changes endpoint load, thereby changing future latency. Monitor oscillation and add hysteresis or bounded score updates.
• Goodput versus throughput: maximize *responses meeting both SLOs*, not just generated tokens/s. More replicas may be expensive; route and autoscale together.

7. What to monitor

• Forecast error: p50/p95 absolute TTFT/TPOT error, calibration by request length, tenant and replica.
• User SLOs: actual p50/p95/p99 TTFT and TPOT, violation rate for each objective and percentage meeting both.
• Router overhead: prediction latency, routing decision time, predictor availability and fallback frequency.
• Worker state: queued tokens, active sequences, KV occupancy, prefix-cache hits, preemptions and per-replica load.
• Business efficiency: SLO-compliant goodput, cost per successful response and any tenant fairness regression.

Interview-ready answer: “I would filter endpoints for model and tenant compatibility, predict TTFT and TPOT on every candidate from token counts, prefix reuse, queue state and KV load, and prefer replicas with positive headroom against both SLOs. I would audit predicted-versus-observed latency, limit prediction overhead and fall back to load-aware routing when the predictor is stale. The goal is goodput meeting real user SLOs—not merely balanced request counts.”

References:

• llm-d — Latency Predictor
• llm-d — Predicted-Latency Based Scheduling for LLMs
• Google Cloud — Predicted latency-based routing with GKE Inference Gateway
• llm-d latency scorer