How does an LLM context window work?

An LLM context window works by giving the model a limited sequence of tokens that it can use together during a single inference request.

Those tokens may contain the current prompt, previous conversation turns, retrieved information, instructions, and other supplied content.

1. The input is converted into tokens

Before the model processes a request, the tokenizer converts the text into token IDs.

Conceptually:

Prompt + history + documents
          ↓
       Tokenizer
          ↓
      Token sequence

The total sequence must fit within the model's supported context limit.

2. Tokens are processed together

Inside a Transformer-based LLM, attention mechanisms allow token representations to use information from other relevant positions in the available sequence.

For a causal language model, each generated position can attend to permitted earlier positions, but not to future tokens.

This lets the model use earlier instructions, facts, examples, and conversation turns while predicting the next token.

3. The model predicts the next token

After processing the current context, the model produces probabilities for the next token.

One token is selected according to the decoding strategy and added to the sequence.

Conceptually:

Current context
      ↓
LLM predicts next token
      ↓
New token is added
      ↓
Repeat

This process continues token by token until generation stops.

4. Generated tokens also use context capacity

As the model generates a response, those new tokens become part of the active sequence used for subsequent predictions.

That means the available context budget may need to accommodate both input and output tokens, depending on the model and API.

For example:

context capacity
= input tokens + generated tokens

The exact accounting rules are model-specific.

5. What happens when the context gets too large?

The model cannot directly process an unlimited number of tokens.

If the request exceeds the supported limit, an application may need to:

• Remove older messages
• Truncate part of the input
• Summarize earlier information
• Retrieve only relevant passages
• Split the task into multiple requests

What happens when the limit is exceeded is covered separately in “What happens when an LLM exceeds its context window?”

Does everything inside the window receive equal attention?

No.

Being inside the context window means information is available to the model, but it does not guarantee that every token will influence the answer equally.

Relevance, position, prompt structure, model behavior, and the amount of competing information can all affect how effectively the model uses the supplied context.

For the basic definition, see “What is a context window in an LLM?”

In short: an LLM context window works by holding a limited token sequence that the model can reference while predicting each next token. As input and generated tokens accumulate, they consume that finite context capacity.