What is context compression in LLMs?

Context compression in LLMs is the process of reducing the number of tokens in the information given to a model while trying to preserve the details that are most useful for the task.

It is commonly used when prompts, conversation histories, or retrieved documents are too large or unnecessarily verbose.

Why is context compression used?

LLMs have limited context windows, and longer contexts usually require more memory, computation, and cost.

Context compression tries to turn:

large amount of context
        ↓
smaller, more relevant context
        ↓
LLM

The goal is to keep important information while removing material that contributes little to the answer.

How can context be compressed?

Common approaches include:

• Summarizing older conversation turns
• Removing duplicate or repetitive text
• Extracting only important facts
• Selecting only relevant document sections
• Removing boilerplate or formatting that is not useful
• Rewriting verbose text into a shorter form

In retrieval-based systems, returning only the most relevant passages can also reduce the amount of context sent to the model.

Example

Suppose a conversation history contains 20,000 tokens, but most of the earlier discussion can be summarized as:

The user selected PostgreSQL, requires multi-tenancy,
and wants API tests generated from requirement documents.

Instead of sending the entire old conversation, an application might send that shorter summary plus the most recent messages.

Is context compression lossless?

Usually not.

Many context-compression methods are lossy, meaning some information is removed.

That creates an important tradeoff:

more compression
→ fewer tokens
→ lower cost and context usage
→ greater risk of losing useful details

A good compression strategy therefore tries to preserve information that is likely to matter for the current task.

Context compression vs. truncation

They are not exactly the same.

Context compression tries to reduce size while preserving important information.

Context truncation simply removes part of the context, often based on position or a token limit.

For that distinction, see “What is context truncation?”

Does context compression change the model itself?

No.

Context compression usually happens before the prompt is sent to the LLM. It changes the information provided to the model, not the model's parameters.

In short: context compression reduces the token size of prompts, histories, or documents while attempting to preserve the most relevant information, helping applications fit useful context within an LLM's limited context window.