What is inference latency in LLMs?

Inference latency in LLMs is the time a system takes to process an inference request and produce a response. For text generation, latency may mean the wait until the first output token appears or the time until the entire response arrives. It is important to state which one is being measured.

Main latency metrics

• Time to first token (TTFT): Time from sending the request until the first output token is received. It includes costs such as network transit, queueing, and processing the input prompt (prefill).
• Time per output token (TPOT): Average time spent producing each additional output token after the first, commonly calculated as (total latency − TTFT) / (output tokens − 1).
• End-to-end (E2E) latency: Time from request submission until the final output is received, including queueing, prefill, decoding, and communication overhead.

Important: Benchmarking tools can define inter-token latency and TPOT slightly differently, especially when streamed chunks contain multiple tokens. Compare the formulas and measurement points, not only their names.

Practical example

Assume a chatbot request returns 50 output tokens with:

• TTFT: 0.6 seconds
• Average TPOT: 0.04 seconds per subsequent token

The estimated end-to-end latency is:

E2E latency ≈ TTFT + (output tokens − 1) × TPOT
            ≈ 0.6 + 49 × 0.04
            ≈ 2.56 seconds

The user sees the first token after about 0.6 seconds, but receives the complete answer after about 2.56 seconds.

What increases inference latency?

• Long input prompts: More tokens require additional prefill work.
• Long responses: Autoregressive models usually generate tokens sequentially, adding decoding time.
• Model and hardware characteristics: Compute throughput, model size, precision, and memory bandwidth all matter.
• Server load and queueing: Concurrent requests can increase waiting time.
• Networking and application overhead: These affect latency measured by the client, even if model execution is fast.

How can latency be reduced?

• Avoid unnecessary prompt and output tokens.
• Use KV caching and suitable serving optimizations.
• Apply quantization or choose a smaller model when accuracy requirements permit.
• Tune batching and scheduling for the intended traffic pattern.
• Stream the response to improve perceived responsiveness; streaming alone does not guarantee lower total generation time.

Latency vs. throughput

• Latency describes how long one request takes.
• Throughput describes how many requests or tokens a system handles per unit of time.

A configuration that improves total throughput may not minimize each user's latency.

For related concepts, see “What happens during an LLM inference request?” and “How does LLM inference work?”

In short: LLM inference latency measures response speed. Monitor TTFT, TPOT, and end-to-end latency together to understand both the user experience and the model-serving bottlenecks.

References: NVIDIA NIM benchmarking metrics and vLLM inference metrics.