Why do LLMs sometimes repeat themselves?

LLMs sometimes repeat themselves because they generate text one token at a time, using their own previous output as part of the context for the next prediction. A repeated phrase can make that same phrase increasingly likely to appear again, creating a feedback loop.

What causes repetition?

• Self-reinforcing predictions: Once a phrase repeats, the growing context may make another repetition seem like a probable continuation.
• Decoding strategy: Greedy decoding and beam search favor high-probability continuations. For open-ended writing, these can get stuck in repetitive loops rather than explore alternatives.
• Repetitive context: A prompt containing the same wording or examples many times may encourage the model to continue that pattern.
• Long generation or weak stopping cues: Without a clear endpoint, a model may circle back to earlier ideas instead of finishing.

Simple example

The solution is to check the logs.
The solution is to check the logs.
The solution is to check the logs.

After the phrase repeats once, the repeated wording becomes part of the context, potentially making another copy more likely.

How can repetition be reduced?

• Use suitable decoding settings: Top-p sampling can introduce alternative continuations instead of always choosing the most likely token. Temperature also affects variation.
• Apply a modest repetition penalty: This lowers the preference for tokens already seen. See “What is a repetition penalty in LLM generation?”
• Use repetition constraints when appropriate: Some inference systems support no_repeat_ngram_size, which blocks repeated token sequences of a specified length.
• Give focused instructions and stopping conditions: Request a specific length or structure and avoid unnecessarily long output limits.

Can repetition ever be useful?

Yes. Repeating a technical term, variable name, or key fact can improve clarity. Excessive penalties may make language unnatural or interfere with legitimate repetition.

Research on neural text degeneration has shown that decoding choices can strongly affect repetitive generation, even when the underlying language model stays unchanged.

In short: repetition is often a feedback effect of autoregressive generation, amplified by certain decoding settings or repetitive context. Better decoding and clear stopping instructions can reduce it, though they cannot guarantee its elimination.