How does greedy decoding work?

Greedy decoding works by selecting the single highest-probability token at every generation step.

The model does not sample randomly or keep multiple candidate paths.

Step 1: Predict next-token probabilities

Suppose the current context is:

The capital of France is

The model may predict:

Paris   → 0.72
Lyon    → 0.12
London  → 0.06
Rome    → 0.04
...

Greedy decoding chooses:

Paris

because it has the highest probability.

Step 2: Append the selected token

The sequence becomes:

The capital of France is Paris

That updated sequence is passed back to the model.

Step 3: Predict again

The model now calculates probabilities for the next token.

For example:

.       → 0.61
and     → 0.14
which   → 0.08

Greedy decoding again chooses the highest-probability option.

Step 4: Repeat until generation stops

The full process is:

context
  ↓
predict probabilities
  ↓
choose argmax token
  ↓
append token
  ↓
repeat

Generation stops when an end token, stop sequence, or output-length limit is reached.

Why is greedy decoding deterministic?

There is no random sampling step.

If the same model and context produce the same probability distribution, greedy decoding always chooses the same highest-probability token.

Why can greedy decoding still produce a weak answer?

The best token right now is not always part of the best complete sequence.

For example, one token may have the highest immediate probability but lead to a repetitive or less coherent continuation later.

Greedy decoding does not reconsider previous choices.

How is it different from beam search?

Greedy decoding keeps only one path:

best token → continue

Beam search keeps several promising candidate sequences and compares them as generation continues.

How is it different from sampling?

Sampling allows multiple tokens to have a chance of being selected.

Greedy decoding always chooses:

argmax(probability)

so it favors reproducibility over diversity.

For the broader concept, see “What is greedy decoding?”

In short: greedy decoding repeatedly selects the highest-probability next token, appends it to the context, and continues until a stopping condition is reached. It is simple and deterministic, but its locally optimal choices do not always produce the best overall response.