What is time per output token in LLM inference?

Time Per Output Token (TPOT) measures the average time required to produce each additional output token after the first during an LLM inference request. It describes the speed of the decoding phase and is usually reported in milliseconds per token (ms/token).

How is TPOT calculated?

A widely used client-side definition, also used in vLLM serving benchmarks, is:

TPOT = (End-to-end latency − TTFT) / (Output tokens − 1)

• TTFT (Time to First Token): Time until the first generated token arrives.
• End-to-end latency: Time until the final generated token arrives.
• Output tokens: Total number of generated tokens.

The first token is excluded because the time spent waiting for it is already counted in TTFT, which includes work such as queueing and prefill.

Practical example

Suppose a request has:

• TTFT: 400 ms
• End-to-end latency: 2,000 ms
• Generated output: 41 tokens

TPOT = (2,000 − 400) / (41 − 1)
     = 1,600 / 40
     = 40 ms/token

So, after the first token, this request delivers output at an average rate of about 25 tokens per second (1,000 / 40).

What affects TPOT?

• Model size and hardware: GPU compute, memory bandwidth, and model architecture affect decoding speed.
• Active context and KV cache: Attention work and memory use can change as the sequence grows.
• Concurrent requests: Scheduling and contention can increase per-request time between tokens.
• Serving optimizations: Efficient batching, quantization, and specialized decoding strategies may improve TPOT, depending on workload.

TPOT vs. TTFT vs. throughput

• TTFT: How long the user waits for the first token.
• TPOT: How quickly subsequent tokens arrive *on average* for one request.
• Throughput: Total requests or tokens served per second across the system.

Low TTFT makes the response start quickly; low TPOT makes the response continue smoothly. A server can improve aggregate throughput through batching while individual requests experience higher TPOT.

Is TPOT the same as inter-token latency (ITL)?

Not always. ITL often measures gaps between consecutive *streamed output events*. In ordinary one-token-per-event streaming, ITL and TPOT are similar. With speculative decoding or chunked output, one event may contain several tokens, so their measured values can differ.

Measurement notes: TPOT is undefined under the formula above for responses containing zero or one output token; benchmarking tools may omit those requests or record a placeholder. Compare benchmark definitions before comparing systems.

For related concepts, see “What is time to first token (TTFT)?”, “What is inference latency in LLMs?”, and “What is LLM throughput?”

In short: TPOT is the average time per generated token after the first, helping measure the responsiveness of ongoing LLM output.

References: NVIDIA NIM inference metrics and vLLM benchmarking metric definitions.