How does a decoder-only LLM generate text?

A decoder-only LLM generates text one token at a time by repeatedly predicting the most likely next token from the tokens already available in its context.

It uses the same Transformer stack for both the original prompt and the tokens generated so far.

1. The prompt is converted into tokens

Suppose the prompt is:

The capital of France is

The tokenizer converts this text into token IDs, which are then mapped to embeddings and passed through the model.

2. Causal self-attention processes the sequence

A decoder-only LLM uses causal self-attention.

This means each position can use information from earlier tokens, but not from future tokens.

Conceptually:

The → capital → of → France → is
                              ↓
                     predict next token

This causal structure matches the way text is generated from left to right.

3. The model produces next-token scores

After processing the current sequence, the model produces a score for every token in its vocabulary.

These scores are usually called logits.

They are converted into probabilities representing how likely each token is to come next.

For example:

"Paris"   → high probability
"London"  → lower probability
"banana"  → very low probability

4. A decoding strategy selects the next token

The next token can be chosen using a decoding strategy such as:

• Greedy decoding
• Temperature-based sampling
• Top-k sampling
• Top-p sampling

The selected token is appended to the sequence.

For example:

The capital of France is
→ Paris

5. The process repeats

Now the model processes:

The capital of France is Paris

and predicts the next token again.

Conceptually:

prompt
  ↓
predict next token
  ↓
append token
  ↓
predict next token
  ↓
append token
  ↓
repeat

This autoregressive loop continues until generation stops.

6. What makes generation stop?

Generation can stop when:

• The model produces an end-of-sequence token
• A configured stop sequence is reached
• A maximum output-token limit is reached
• Application-specific stopping logic is triggered

Does the model recompute everything for every token?

Not usually.

During inference, implementations commonly use a KV cache to reuse attention information from previously processed tokens.

This makes generation much more efficient than recalculating the entire sequence from scratch for every new token.

For the architecture itself, see “What is a decoder-only LLM?”

In short: a decoder-only LLM generates text by processing the prompt with causal self-attention, predicting probabilities for the next token, selecting one token, appending it to the context, and repeating the process until a stopping condition is reached.