How does an LLM predict the next token?

An LLM predicts the next token by turning the current text into internal numerical representations, processing those representations through Transformer layers, and then assigning a probability to every token in its vocabulary.

Suppose the input is:

The sky is

The model might produce probabilities such as:

blue     0.62
clear    0.14
dark     0.08
bright   0.05
...

The model then selects one token according to the decoding strategy being used.

1. The input is tokenized

The text is first converted into token IDs.

For a dedicated explanation, see “What is tokenization in LLMs?”

2. Tokens are converted into vectors

Each token ID is mapped to an embedding. Positional information is also added so the model knows where each token appears in the sequence.

3. Transformer layers process the context

The embeddings pass through many Transformer layers.

Using self-attention, the model determines which earlier tokens are relevant to the current prediction and builds a context-aware representation.

For example, in:

The cat drank the milk because it was thirsty.

the representation of later tokens depends on relationships with earlier ones.

4. The final hidden state is converted into logits

For the position where the next token must be predicted, the model produces a vector of scores called logits.

There is typically one logit for each token in the model's vocabulary.

For example:

blue     8.2
clear    6.7
dark     6.1
...

A higher logit means the model considers that token more suitable in the current context.

5. Softmax converts logits into probabilities

The logits are transformed into probabilities using the softmax function.

Conceptually:

logits → softmax → token probabilities

The probabilities across the vocabulary sum to 1.

6. A token is selected

The application can choose the highest-probability token or sample from the probability distribution using decoding settings such as temperature, top-k, or top-p.

The selected token is appended to the context, and the model repeats the same process to predict the following token.

In short: an LLM predicts the next token by processing the current context through Transformer layers, producing a logit for every vocabulary token, converting those logits into probabilities, and selecting the next token from that distribution.