How do you size and autoscale prefill and decode worker pools in disaggregated LLM serving?

Size prefill and decode pools separately using measured token workloads, service-time capacity and latency SLOs. Prefill is mainly prompt-processing compute; decode generates tokens while retaining KV state and may be limited by memory bandwidth and active KV capacity. Scale whichever phase is the actual bottleneck instead of using GPU utilization alone.

1. What do the two pools need?

• Prefill pool (P): prompt tokens per second, prompt-length distribution, prefill queue waiting time and TTFT. Longer prompts consume more work than shorter ones.
• Decode pool (D): generated tokens per second, active generations, average output length, inter-token latency (ITL/TPOT), remaining KV capacity and request preemptions.
• Between them: KV-transfer time and network bandwidth. Faster prefill is not useful if KV handoff or decoding is saturated.
• Global goal: meet p95/p99 TTFT and inter-token latency targets at minimum sustainable GPU cost, with spare capacity for bursts and failures.

2. Beginner-friendly sizing example

Suppose a service receives 20 requests per second in steady state. Each request averages 2,000 input tokens and 400 output tokens.

1. Prefill demand = 20 × 2,000 = 40,000 prompt tokens/second.
2. Decode demand = 20 × 400 = 8,000 generated tokens/second.
3. Load testing shows each configured prefill worker can sustain 10,000 prompt tokens/second and each decode worker can sustain 2,000 generated tokens/second while maintaining the desired latency targets.
4. Nominal requirement: 4 prefill workers and 4 decode workers.
5. With a planning target of 70% utilization: ceil(40,000/(10,000×0.7)) = 6 prefill workers; ceil(8,000/(2,000×0.7)) = 6 decode workers.

Important: These are illustrative, workload-conditioned rates, not universal GPU benchmarks. Prefill cost is not perfectly linear in length, decode throughput depends on batching and KV occupancy, and heterogeneous workers need individual calibration.

3. Why the ratio changes

• Many long prompts and short replies: proportionally more prefill capacity.
• Short prompts and long generated answers: more decode capacity, especially as live KV state grows.
• Heavy prefix-cache reuse reduces fresh prefill work and may change the required P:D ratio.
• A bursty chatbot and an offline batch job should be modeled separately; their p95 latency targets differ.
• Scale complete tensor-parallel worker groups, not arbitrary fractions of a shard.

4. Autoscaling signals by phase

• Prefill: in-flight or queued *input tokens*, estimated time to drain the queue, prefill running requests and measured p95 TTFT.
• Decode: active sequence count, output token backlog when estimable, KV occupancy, preemptions and p95 ITL/TPOT.
• End to end: admission/transfer queue depth, KV-transfer latency and goodput meeting both user-facing SLOs.
• Avoid GPU utilization as the sole metric: continuous batching may keep it near 100% at very different levels of customer-visible saturation.

5. Production controller

1. Run representative load tests to estimate per-role throughput at acceptable TTFT/ITL.
2. Export per-role request/token backlog, KV occupancy and latency to a monitoring system.
3. Use KEDA/HPA or an equivalent controller to raise a role's replica count when projected queueing or SLO pressure exceeds thresholds.
4. Include headroom, cooldown/hysteresis and startup time to avoid oscillation; prewarm new workers because model load times can be significant.
5. Coordinate prefill and decode growth: a larger prefill pool can flood a smaller decode pool. The router should stop over-admitting when decode has insufficient capacity.
6. Scale down carefully: allow decode workers to drain active sessions and preserve correct KV transfers. Avoid terminating a replica carrying long generations.
7. Validate under long-context bursts, changing prefix-cache hit rates, network saturation, mixed tenants and worker failures.

6. Practical incident

Symptom: TTFT rises from 250 ms to 900 ms while decode ITL remains near 30 ms. Metrics show the prefill token queue growing, but decode KV occupancy is only 45%.

• Diagnosis: insufficient prefill capacity. Adding decode GPUs alone is unlikely to improve TTFT.
• Action: scale prefill workers or reduce prefill load through permitted prompt-prefix reuse and request admission.
• Counterexample: low TTFT but decode KV occupancy above 90%, frequent preemptions and higher ITL suggest a decode-capacity bottleneck.

7. Limitations and trade-offs

• A static capacity equation approximates a complex queueing system; burstiness, prompt-length tails and per-request parallelism cause nonlinear p95 effects.
• Remote KV transfers may become the bottleneck even when both GPU pools look healthy.
• More replicas increase cost and may reduce cache locality or increase coordination.
• Never claim that disaggregation intrinsically improves throughput; compare against aggregated serving and chunked prefill on the same workload.
• If using llm-d, consult the installed release's documentation; newer autoscaling guides support token-aware prefill backlog estimates and decode KV occupancy, while autoscaling strategies and component maturity can vary by release.

Interview-ready answer: “I'd estimate prefill and decode token demand separately, benchmark per-worker service capacity under TTFT and ITL targets, and size each pool with headroom. Autoscaling should follow prompt-token backlog and prefill queueing for P, plus KV occupancy, active generations and ITL for D. I'd coordinate scaling across both pools and the KV-transfer network and judge success by SLO-compliant goodput and cost.”

References: llm-d autoscaling guide; llm-d disaggregated serving; NVIDIA: disaggregated inference on Kubernetes.