What is context truncation?

Context truncation is the process of removing part of an LLM's input context so the remaining token sequence fits within the model's context limit.

Unlike context compression, truncation usually removes tokens rather than rewriting or summarizing them.

Why is truncation needed?

An LLM can process only a limited number of tokens at one time.

If a request becomes too long, an application may need to shorten it before sending it to the model.

Conceptually:

context is too long
      ↓
remove some tokens
      ↓
context fits the limit
      ↓
send to LLM

What parts of the context can be truncated?

An application may remove:

• Older conversation turns
• The beginning of a document
• The end of a document
• Less important retrieved passages
• Repeated or low-priority content

Which part is removed depends on the application's truncation strategy.

What are left and right truncation?

Two common approaches are:

• Left truncation — remove tokens from the beginning of the sequence.
• Right truncation — remove tokens from the end of the sequence.

For example:

Original:
A B C D E F G

Left truncation:
D E F G

Right truncation:
A B C D

In conversation systems, left truncation may be used to remove older messages while preserving recent turns.

What is the risk of truncation?

Truncation can remove information that is important for the current task.

If a critical instruction or fact is removed, the model cannot directly use it in that inference request.

This can lead to:

• Missing details
• Inconsistent answers
• Forgotten earlier instructions
• Incorrect conclusions based on incomplete context

Context truncation vs. context compression

They solve a similar problem but in different ways.

Truncation removes part of the context.

Compression tries to preserve important information in fewer tokens, for example by summarizing or extracting key facts.

Conceptually:

Truncation:
remove information

Compression:
rewrite or select information more compactly

For the related concept, see “What is context compression in LLMs?”

Is truncation always automatic?

No.

Some APIs reject an oversized request instead of truncating it automatically.

In other systems, the application decides what to remove before making the request.

Developers therefore need to understand the model's token limit and the behavior of the specific API or framework they use.

In short: context truncation removes tokens from an oversized prompt or conversation so it fits within an LLM's context window. It is simple and efficient, but it can discard important information if the truncation strategy is not chosen carefully.