What is decoding in an LLM?

Decoding in an LLM is the process of choosing which token to generate next from the probability distribution produced by the model.

The model predicts probabilities for many possible next tokens. Decoding decides how one of those tokens is selected.

Where does decoding happen?

Suppose the current text is:

The capital of France is

The model may assign probabilities such as:

Paris   → high probability
Lyon    → lower probability
London  → very low probability

Decoding determines which token is actually chosen.

Why is decoding needed?

The model does not directly output a finished sentence.

At every generation step it produces scores for tokens in its vocabulary.

Those scores are converted into probabilities, and a decoding strategy chooses the next token.

The process is:

current context
      ↓
next-token probabilities
      ↓
decoding strategy
      ↓
chosen token
      ↓
append to context and repeat

What are common decoding strategies?

Common approaches include:

• Greedy decoding — choose the highest-probability token.
• Sampling — randomly choose according to the probability distribution.
• Top-k sampling — sample only from the k most likely tokens.
• Top-p sampling — sample from the smallest group of tokens whose cumulative probability reaches a threshold.
• Beam search — keep multiple promising token sequences and compare them as generation continues.

Each strategy creates different output behavior.

How does decoding affect the response?

Decoding influences properties such as:

• Determinism
• Creativity
• Diversity
• Repetition
• Fluency
• Response stability

For example, greedy decoding tends to be more deterministic, while sampling can produce more varied responses.

What role does temperature play?

Temperature modifies the token probability distribution before sampling.

A lower temperature makes high-probability tokens more dominant.

A higher temperature spreads probability across more possibilities.

So temperature is not itself the model's knowledge. It changes how aggressively decoding favors likely tokens.

Is decoding the same as next-token prediction?

No.

Next-token prediction produces the probability distribution.

Decoding determines how a token is chosen from that distribution.

Conceptually:

LLM prediction:
"What tokens are likely?"

Decoding:
"Which token should we actually choose?"

Does decoding happen only once?

No.

It happens repeatedly during autoregressive generation.

After each selected token is appended to the sequence, the model produces a new probability distribution and the decoding process runs again.

Can decoding parameters change the answer without changing the model?

Yes.

The same LLM and prompt can produce different responses when settings such as:

• Temperature
• Top-k
• Top-p
• Repetition penalties

are changed.

The model parameters remain the same; only the generation policy changes.

For the different approaches in more detail, see “What are decoding strategies in LLMs?”

In short: decoding is the step that converts an LLM's next-token probability distribution into an actual token choice. The decoding strategy strongly affects how deterministic, diverse, or creative the generated response becomes.