What is continuous batching in LLM inference?

Continuous batching (also called iteration-level batching) is an LLM serving technique that dynamically groups requests at each generation step, rather than keeping the same requests in a fixed batch until all of them finish.

The main benefit is higher GPU utilization and serving throughput when many users send requests of different lengths.

Why is ordinary batching inefficient?

Suppose three requests arrive together:

• A: Generate 10 tokens.
• B: Generate 100 tokens.
• C: Generate 30 tokens.

In a static batch, the serving system may wait for the whole group to finish before admitting another batch. Once A and C finish, their slots may be wasted while B continues.

With continuous batching, A leaves the running batch as soon as it finishes, and a new request, D, can join while B and C continue.

Step 1:   [A, B, C]
Step 10:  A finishes → admit D
Step 11:  [D, B, C]
Step 30:  C finishes → admit E
Step 31:  [D, B, E]

This is a simplified illustration: real schedulers also manage prompt processing, token budgets, and memory availability.

How does continuous batching work?

1. Queue requests: Incoming prompts wait for capacity.
2. Schedule a step: The server chooses active requests for the next model iteration.
3. Run the model: Each decoding request typically advances by one token in that iteration.
4. Update state: Completed requests leave; unfinished requests retain their own KV-cache state.
5. Admit new work: Waiting requests join when capacity permits, and the next iteration begins.

The scheduler can therefore change the active batch without restarting unfinished generations.

Why does it improve inference performance?

• Less idle GPU capacity: A short request finishing does not force the system to wait for longer requests.
• Higher aggregate throughput: More requests or output tokens may be processed over time.
• Better handling of variable-length outputs: Requests with different generation lengths can share hardware efficiently.
• Potentially lower queueing delay: New requests need not wait for the slowest member of a previous static batch.

These are serving-level improvements; continuous batching does not automatically reduce the compute required for a single token or guarantee lower latency for every individual request.

How does it relate to KV cache and prefill?

• KV cache: Each active sequence retains attention keys and values so it can resume decoding across iterations.
• Prefill: Newly admitted prompts must first be processed to build their KV caches.
• Chunked prefill: A *separate but complementary* technique that divides long prompt-processing work into smaller pieces, allowing the scheduler to interleave prefill and decoding rather than blocking decoding with one very long prompt.

What are the trade-offs?

Continuous batching needs a scheduler that respects GPU memory and compute limits. Too many active sequences can exhaust KV-cache capacity, increase per-token latency, or create unfair waiting times. Systems use limits on active sequences, batched tokens, queue depth, and admission to balance throughput, responsiveness, and memory.

Serving engines such as vLLM and Hugging Face Transformers serving use dynamic request scheduling, with implementation-specific policies.

What should you monitor in production?

Track requests per second, output tokens per second, time to first token (TTFT), time per output token (TPOT), queueing delay, and GPU/KV-cache utilization. Increasing throughput is useful only if user-facing latency remains acceptable.

In short: continuous batching keeps a changing group of LLM requests running together, adding and removing sequences at each generation step to use accelerator capacity more efficiently.

References: Hugging Face: Continuous batching, ORCA: A Distributed Serving System for Transformer-Based Generative Models, and vLLM scheduler configuration.