What is top-k sampling?

Top-k sampling is a decoding strategy where an LLM limits the next-token choice to the k most probable tokens and then samples from that reduced set.

Instead of sampling from the entire vocabulary, the model ignores all tokens outside the top-k candidates.

How does top-k sampling work?

Suppose the model predicts:

Token A → 0.40
Token B → 0.25
Token C → 0.15
Token D → 0.08
Token E → 0.05
...

If:

top-k = 3

only these remain eligible:

Token A
Token B
Token C

The probabilities of those remaining tokens are renormalized, and the next token is sampled from them.

Why use top-k sampling?

An LLM vocabulary can contain tens of thousands of tokens.

Many of those tokens may have extremely small probabilities for the current context.

Sampling from the full vocabulary can occasionally select an unlikely token and produce strange output.

Top-k reduces this risk by restricting sampling to the most likely candidates.

What happens with a small k?

A small value makes generation more conservative.

For example:

top-k = 1

means only the single most probable token is allowed.

That effectively behaves like greedy decoding.

A slightly larger value allows some variation while remaining tightly controlled.

What happens with a large k?

A larger k allows more candidate tokens into the sampling pool.

That usually increases:

• Diversity
• Creativity
• Variation between responses

But if k is too large, lower-quality or less relevant tokens may also become eligible.

Is top-k itself random?

Top-k filtering is not the random part.

It first decides which tokens are allowed.

Sampling then chooses among those allowed tokens according to their probabilities.

Conceptually:

all vocabulary tokens
        ↓
keep k most probable
        ↓
renormalize probabilities
        ↓
sample one token

How does top-k interact with temperature?

Temperature modifies the relative probabilities of the candidate tokens.

Top-k controls how many tokens are allowed.

They can be used together:

model logits
   ↓
temperature adjustment
   ↓
top-k filtering
   ↓
sampling

The exact order can vary by implementation, but both influence the final token selection.

How is top-k different from top-p?

Top-k always keeps a fixed number of tokens.

Top-p keeps a variable number of tokens based on cumulative probability.

For example:

• Top-k = 50 always considers up to 50 tokens.
• Top-p = 0.9 considers however many tokens are needed to cover roughly 90% of the probability mass.

This makes top-p adapt to how confident the model is at each step.

When is top-k useful?

Top-k can be useful when you want:

• More diversity than greedy decoding
• Less randomness than unrestricted sampling
• Protection against extremely unlikely token choices
• A simple way to control the candidate set

In short: top-k sampling keeps only the k most likely next tokens and samples from that smaller group. It provides a simple balance between deterministic generation and unrestricted randomness.