How does LLM inference work?

LLM inference works by running a trained model on an input prompt, computing next-token probabilities, and—when generating text—repeatedly selecting and appending tokens until a stopping condition is met. The model's weights normally stay fixed throughout this process.

Step 1: Prepare the request

The application assembles the effective prompt, which may include instructions, conversation history, and the current user message. A tokenizer converts the text into token IDs that the model can process.

Step 2: Process the prompt (prefill)

The model processes the prompt tokens through its Transformer layers.

• Token IDs are converted into vector representations.
• Attention and feed-forward layers compute contextual representations using pretrained weights.
• The final hidden state is projected into logits, or scores, for possible next tokens.
• An inference engine may build a KV cache, storing attention keys and values from the prompt for reuse.

This stage is called prefill because it processes the provided context before incremental generation begins.

Step 3: Select the first output token

The model's next-token scores are transformed into a token choice using the configured decoding strategy.

For example:

• Greedy decoding: Choose the highest-scoring token.
• Sampling: Draw a token using model probabilities, potentially modified by temperature or top-p.

The selected token is appended to the existing sequence.

Step 4: Generate subsequent tokens (decode)

For an autoregressive LLM, the next output tokens are generated one by one.

Prompt → Tokenize → Transformer prefill
                         ↓
                  Next-token logits
                         ↓
                    Select token
                         ↓
                Append to the sequence
                         ↓
                 Decode next token
                         ↺ repeat

With KV caching, the model can reuse previously computed keys and values rather than recomputing attention representations for all earlier tokens. It still performs calculations for every new token.

Step 5: Stop and return the response

Generation stops when the model emits an ending token, a configured stop sequence is matched, an output-token limit is reached, or the application interrupts the request.

The output token IDs are decoded back into text. A serving system may stream tokens as they are generated rather than wait for the entire response.

What makes inference fast or slow?

• Prompt length: Longer prompts generally increase prefill computation.
• Generated token count: More output tokens require more sequential decoding steps.
• Model size and precision: These determine substantial portions of compute and memory requirements.
• Serving optimizations: KV caching, batching, quantization, and hardware choice can improve efficiency.

Two common metrics are time to first token (TTFT) and time per output token; these help distinguish prefill and decoding performance.

In short: LLM inference consists of preparing tokens, running a prefill pass through frozen model weights, generating tokens autoregressively (often with a KV cache), and stopping when a termination condition is reached.