What is top-p sampling?

Top-p sampling, also called nucleus sampling, is an LLM decoding strategy that samples the next token from the smallest set of high-probability tokens whose cumulative probability is at least p.

Unlike top-k sampling, which keeps a fixed number of tokens, top-p adapts the candidate set to the model's confidence at each generation step.

Simple example

Suppose the LLM predicts these next-token probabilities, sorted from highest to lowest:

Token A → 0.52  (cumulative: 0.52)
Token B → 0.23  (cumulative: 0.75)
Token C → 0.13  (cumulative: 0.88)
Token D → 0.07  (cumulative: 0.95)
Token E → 0.05  (cumulative: 1.00)

With top-p = 0.90, the model keeps A, B, C, and D. Their cumulative probability is 0.95, which is the first point where the total reaches or exceeds 0.90.

Token E is excluded. The probabilities of the retained tokens are renormalized, and one is sampled.

Why is top-p useful?

The model's next-token distribution changes with the context:

• Confident prediction: A few tokens may already cover most of the probability mass, so top-p keeps a small candidate set.
• Uncertain prediction: Probability is spread across many plausible tokens, so top-p can keep more candidates.

This adaptive filtering helps balance natural variation with protection against very unlikely token choices.

What happens when p changes?

• Lower p: Keeps a smaller, more conservative group of candidates, generally reducing variety.
• Higher p: Allows more candidates and usually increases diversity.
• p = 1: Does not filter tokens by cumulative probability.

Top-p is a sampling filter, not a guarantee of factual accuracy. Raising p can allow weaker continuations, while lowering p can still produce incorrect answers.

Top-p vs. top-k

• Top-k: Keep exactly the k most probable tokens (when at least k are available).
• Top-p: Keep however many highest-probability tokens are needed to meet the probability threshold.

Top-p is commonly combined with temperature, which reshapes probabilities before sampling. The exact behavior also depends on the inference implementation.

For the fixed-size alternative, see “What is top-k sampling?”

In short: top-p sampling dynamically selects a nucleus of likely next tokens whose combined probability reaches a specified threshold, then samples from that set.