Large Language Models

Large language models are deep learning models, usually transformer-based, trained to predict tokens from large corpora. After pretraining, they are often instruction-tuned and aligned so they can follow prompts and assist users.

What is a context window in an LLM?

A context window is the maximum amount of tokenized information an LLM can consider at one time while processing a request. It is measured in tokens , not words or characters. What can fit inside the context window? Depending on the model and application, the context window can include: System instructions User messages Previous conversation turns Retrieved documents Tool outputs Code or other supplied text Tokens generated as part of the current response, depending on the model/API design Conceptually: The exact accounting rules vary by model and API. Why does the context window matter? An LLM can directly use only information that is available within the context it receives for the current inference request. If important information is outside that available context, the model cannot directly attend to it unless the application brings that information back into the prompt, for example through retrieval or summarization. Example Suppose a model supports a context window of: A request containing instructions, conversation history, and documents must fit within that limit together with whatever token budget is reserved for the model's response. If the combined content exceeds the supported limit, the application may need to: Remove older content Truncate documents Summarize previous information Retrieve only relevant sections Split the task into multiple requests Is a larger context window always better? Not automatically. A larger window allows more information to be supplied at once, but long context processing can require more memory and computation. Also, simply placing more text into the context does not guarantee that the model will use every part equally well. The quality and relevance of the supplied context still matter. Context window vs. model knowledge The context window is different from knowledge stored in the model's learned parameters. Parameter based knowledge comes from training. The context window contains information supplied for the current request. So: Does a chat application's visible history equal the context window? Not necessarily. An application may show a long conversation history while sending only part of it to the model, or it may summarize older messages before including them. The actual context is the token sequence supplied to the model for that inference request. In short: a context window is the maximum token based working context an LLM can process at once. It determines how much prompt, conversation history, retrieved information, and response content can participate in a single inference request.

How does an LLM context window work?

An LLM context window works by giving the model a limited sequence of tokens that it can use together during a single inference request. Those tokens may contain the current prompt, previous conversation turns, retrieved information, instructions, and other supplied content. 1. The input is converted into tokens Before the model processes a request, the tokenizer converts the text into token IDs. Conceptually: The total sequence must fit within the model's supported context limit. 2. Tokens are processed together Inside a Transformer based LLM, attention mechanisms allow token representations to use information from other relevant positions in the available sequence. For a causal language model, each generated position can attend to permitted earlier positions, but not to future tokens. This lets the model use earlier instructions, facts, examples, and conversation turns while predicting the next token. 3. The model predicts the next token After processing the current context, the model produces probabilities for the next token. One token is selected according to the decoding strategy and added to the sequence. Conceptually: This process continues token by token until generation stops. 4. Generated tokens also use context capacity As the model generates a response, those new tokens become part of the active sequence used for subsequent predictions. That means the available context budget may need to accommodate both input and output tokens, depending on the model and API. For example: The exact accounting rules are model specific. 5. What happens when the context gets too large? The model cannot directly process an unlimited number of tokens. If the request exceeds the supported limit, an application may need to: Remove older messages Truncate part of the input Summarize earlier information Retrieve only relevant passages Split the task into multiple requests What happens when the limit is exceeded is covered separately in “What happens when an LLM exceeds its context window?” Does everything inside the window receive equal attention? No. Being inside the context window means information is available to the model, but it does not guarantee that every token will influence the answer equally. Relevance, position, prompt structure, model behavior, and the amount of competing information can all affect how effectively the model uses the supplied context. For the basic definition, see “What is a context window in an LLM?” In short: an LLM context window works by holding a limited token sequence that the model can reference while predicting each next token. As input and generated tokens accumulate, they consume that finite context capacity.

What determines an LLM’s context window size?

An LLM’s context window size is determined by the model’s architecture, training configuration, positional representation, and the practical compute and memory limits of running the model. It is not simply an arbitrary number chosen at inference time. 1. The model is trained for a certain sequence length During training, the model sees token sequences up to a configured length. That training length strongly influences how well the model can use information across long sequences. A model trained only on relatively short sequences cannot automatically become a reliable long context model just by allowing a larger input. 2. Positional information matters Transformers need a way to represent where tokens appear in a sequence. Methods such as: Learned positional embeddings Rotary Position Embeddings (RoPE) ALiBi affect how far the model can represent positions and how well it can generalize to longer sequences. Some positional methods make context extension easier than others, but extending the numerical limit does not guarantee equally good performance across the entire longer context. 3. Attention architecture affects scalability Standard self attention compares token positions across the sequence. As context becomes longer, attention can require much more computation and memory. Models may therefore use techniques such as: Sliding window attention Sparse attention Grouped or optimized attention implementations to make longer contexts more practical. These approaches are covered separately in their dedicated articles. 4. Memory and compute impose practical limits Longer contexts require more resources during both training and inference. They can increase: GPU memory usage Attention computation KV cache memory Inference latency So context window size is partly an engineering tradeoff between capability and cost. 5. The model configuration defines the supported limit A model usually has a configured maximum sequence or context length. The tokenizer and serving system use that model configuration to determine how many tokens can be accepted in one request. APIs may also impose their own limits for input and output tokens. Can a model’s context window be extended later? Sometimes. Techniques such as additional long context training, positional scaling, or other architectural adjustments can extend the usable context of some models. However: A model may technically accept a longer sequence while still becoming less reliable at using information across the full range. For the basic concept, see “What is a context window in an LLM?” In short: an LLM’s context window size is determined by its training sequence length, positional representation, attention design, model configuration, and the memory and compute available to process long token sequences.

What happens when an LLM exceeds its context window?

When an LLM request exceeds its context window , the full token sequence can no longer fit within the model’s supported context length. What happens next depends on the model API or application using it. 1. The request may be rejected Some APIs return an error when the combined token count is larger than the supported limit. Conceptually: In that case, the request must be shortened before it can be processed. 2. Part of the input may be truncated Some applications automatically remove part of the supplied context. For example, they may drop: Older conversation turns Earlier document sections Less important retrieved passages Other low priority context If important information is removed, the model can no longer directly use it for that request. 3. The available output budget may shrink In systems where input and generated output share the same total context capacity, a very large prompt can leave fewer tokens available for the response. For example: The exact accounting rules depend on the model and API. 4. Applications may compress the context Instead of simply deleting information, an application can reduce the amount of text by: Summarizing older conversation history Retrieving only relevant document sections Compressing repeated information Splitting a large task into multiple requests These techniques help keep the most useful information within the available token budget. Does the LLM automatically remember removed content? No. If information is no longer included in the active context, the model cannot directly attend to it during that inference request. An application must provide the information again, retrieve it from an external source, or include a summary of it. Why is this important in long conversations? A chat interface may appear to contain a very long history, but the application may not send every previous message to the model forever. As the conversation grows, older information may be truncated, summarized, or selectively included to stay within the context limit. For the basic concept, see “What is a context window in an LLM?” In short: when an LLM exceeds its context window, the request must be reduced somehow. Depending on the system, it may be rejected, truncated, summarized, selectively retrieved, or given a smaller output budget.

Why do LLMs have context limits?

LLMs have context limits because processing longer token sequences requires more memory, computation, and model capacity. A context window therefore places a practical limit on how many tokens the model can consider at one time. 1. Attention becomes expensive as sequences grow Transformer models use attention to relate tokens to one another. In standard self attention, increasing the sequence length can increase computation and memory very quickly because many token positions interact with many others. So: This is one of the main reasons context length cannot grow without cost. 2. Longer contexts require more memory During inference, the model stores intermediate information for tokens in the active sequence. For autoregressive LLMs, the KV cache stores key and value representations from previous tokens so they do not need to be recomputed for every new token. As the context becomes longer, the KV cache grows and consumes more memory. 3. Models are trained for specific sequence lengths An LLM is trained using sequences of certain lengths and positional representations designed for a particular range. Simply allowing more tokens at inference time does not guarantee that the model will understand very long sequences reliably. Long context capability normally requires suitable architecture, positional methods, and training. 4. Positional representation has limits Transformers need information about token order and position. Methods such as positional embeddings or Rotary Position Embeddings (RoPE) are designed around particular sequence ranges. Extending those ranges may require scaling techniques or additional training. 5. Longer context increases serving cost and latency Supporting larger contexts can increase: GPU memory requirements Inference latency Compute consumption KV cache storage Infrastructure cost Model providers therefore choose context limits that balance capability with practical serving constraints. Can context limits be made larger? Yes. Modern models use techniques such as: More efficient attention mechanisms Sliding window attention Sparse attention Improved positional representations Long context training KV cache optimizations These techniques can make much larger context windows practical. However, a larger advertised context limit does not automatically mean the model uses every part of that context equally well. For the factors that set the limit, see “What determines an LLM’s context window size?” In short: LLMs have context limits because longer sequences require more attention computation, memory, KV cache capacity, suitable positional representations, and long context training. The limit is a tradeoff between how much information the model can process at once and the cost of doing so.

What is context length in an LLM?

Context length in an LLM is the number of tokens currently included in the sequence the model is processing. It is measured in tokens and can include the prompt, conversation history, retrieved information, system instructions, and generated tokens that are still part of the active sequence. Context length vs. context window These two terms are related but not identical. Context length is how many tokens are currently being processed. Context window is the maximum number of tokens the model can process at once. For example: The current request is using only part of the model's available context capacity. What contributes to context length? Depending on the model and application, context length may include: System instructions User prompts Previous conversation turns Retrieved documents Tool results Special tokens Generated response tokens The exact accounting depends on the model and API. Why does context length matter? As context length increases: More of the context window is consumed Memory usage can increase Inference can become slower Attention computation may increase Less room may remain for additional input or output A longer context can provide more information, but only when that information is relevant and the model can use it effectively. Is context length measured in words? No. Context length is normally measured in tokens . Because different tokenizers split text differently, the same number of words can produce different context lengths across different LLMs. For example: Does context length stay fixed during generation? No. During autoregressive generation, each newly generated token becomes part of the sequence used to predict the next token. So the active context length usually grows as the response is generated. For the maximum supported capacity, see “What is a context window in an LLM?” In short: context length is the number of tokens currently being processed by an LLM, while the context window is the maximum number of tokens the model can handle at one time.

What is the difference between input tokens and output tokens?

Input tokens are the tokens sent to an LLM as part of a request, while output tokens are the tokens the model generates in its response. Both are created and counted by the model's tokenizer. What are input tokens? Input tokens come from the information provided to the model before generation starts. Depending on the application, they can include: System instructions User prompts Previous conversation turns Retrieved documents Tool results Special formatting or chat tokens For example: These tokens form the context the model uses when deciding what to generate next. What are output tokens? Output tokens are the new tokens generated by the LLM. An autoregressive LLM produces them one at a time: Each generated token becomes part of the context used to predict the following token. How do they relate to the context window? In many LLM systems, input and output tokens share the model's total context capacity. Conceptually: The exact limits and accounting rules depend on the model and API. A very large input can therefore leave less room for the model's response. Do input and output tokens cost the same? Not necessarily. Many hosted LLM APIs price input and output tokens separately, and output tokens may have a different price from input tokens. The exact pricing depends on the provider and model. Simple example Suppose a request contains: The model processed 2,000 tokens of supplied context and generated 500 new tokens. The request therefore involved both input processing and output generation. Are tokens the same as words? No. A token may be: A complete word Part of a word Punctuation Whitespace A character or byte based piece So 1,000 words do not necessarily equal 1,000 tokens. For the underlying unit, see “What is a token in an LLM?” In short: input tokens are the tokenized information supplied to an LLM, while output tokens are the new tokens generated by the model. Both affect context usage, computation, and often API cost.

What is maximum sequence length in an LLM?

Maximum sequence length in an LLM is the largest number of tokens the model is configured to process in a single sequence. It places an upper bound on how long the model’s token sequence can be during training or inference. What does it include? Depending on the model and API, the sequence can contain: System instructions User input Conversation history Retrieved documents Special tokens Generated output tokens Conceptually: Is maximum sequence length the same as context window? They are closely related and are often used interchangeably, but the wording can emphasize different things. Maximum sequence length usually refers to the technical maximum number of token positions the model can process. Context window refers to the usable token capacity available as context for a request. In many practical LLM discussions, both refer to the same overall token limit. Example Suppose a model has a maximum sequence length of: A request might use: if that model and API share the same total sequence capacity between input and output. The exact accounting rules vary by model and provider. What determines this limit? Maximum sequence length is influenced by: Model architecture Positional representation Training sequence length Attention design Memory and compute constraints Model configuration A model cannot reliably support an arbitrarily long sequence simply because an application sends more tokens. What happens if the limit is exceeded? The system may: Reject the request Truncate part of the input Reduce the available output length Require the application to shorten or split the content The exact behavior depends on the serving system. Does supporting a larger maximum sequence length guarantee better long context performance? No. A model may technically accept a very long sequence but still struggle to use information effectively across the entire context. The supported limit describes how much text can fit, not necessarily how well every token will be used. For the related concept, see “What is a context window in an LLM?” In short: maximum sequence length is the largest token sequence an LLM can process at once. It defines the technical upper bound for the combined token sequence used during training or inference.

How does context length affect LLM performance?

Context length affects LLM performance because increasing the number of tokens gives the model access to more information, but it also makes the task of finding and using the most relevant information harder. A longer context can help when important details are spread across a large document or conversation, but simply adding more tokens does not guarantee better answers. 1. Longer context can provide more useful information A larger context can let the model consider: Longer documents More conversation history More retrieved passages Larger codebases More examples or instructions This can improve performance when the additional information is relevant to the task. 2. Too much irrelevant context can hurt If the context contains a large amount of unrelated or repetitive information, the model has more competing information to process. This can make it harder to identify the details that actually matter. So: The quality of the context is often as important as its length. 3. Models may not use every position equally well An LLM may technically support a very large context window without using information at every position equally effectively. Information placed far from the current generation point, or buried among many other tokens, can sometimes have less influence on the answer. This is one reason long context evaluation measures more than whether the input simply fits inside the window. 4. Long contexts can expose retrieval problems A model may need to locate one important fact among thousands of tokens. As the context becomes longer, tasks such as: Finding a specific detail Connecting distant facts Following instructions from earlier sections Comparing information across a document can become more difficult. The lost in the middle problem is one example and is covered separately. 5. Longer context can increase inference cost and latency Processing more tokens generally requires more computation. As context length increases, the model may take longer to process the prompt and consume more serving resources. The memory impact is covered separately in “How does context length affect LLM memory usage?” Does shorter context always perform better? No. A context that is too short may omit important information. The goal is not to minimize context length at all costs, but to provide enough relevant information without overwhelming the model with unnecessary text. A useful principle is: For the basic definition, see “What is context length in an LLM?” In short: longer context can improve LLM performance by providing more useful information, but very long or noisy contexts can make retrieval, reasoning, latency, and effective information use more difficult.

How does context length affect LLM memory usage?

Longer context length increases LLM memory usage because the model must keep more information about the tokens in the active sequence. The exact memory cost depends on whether the model is training or generating text, but longer sequences generally require more memory in both cases. 1. More tokens create more intermediate data Each token produces hidden representations as it moves through the model's layers. With more tokens in the sequence, the model must process and store more of these intermediate values. Conceptually: 2. KV cache grows during inference During autoregressive generation, Transformer LLMs commonly use a KV cache . The cache stores key and value representations for previously processed tokens so the model does not need to recompute them every time it generates the next token. As context length grows: The amount of KV cache memory also depends on factors such as the number of layers, attention heads, hidden dimensions, numeric precision, and attention design. 3. Attention itself can require substantial memory In standard self attention, tokens interact with many other token positions. For a sequence containing (n) tokens, a full attention matrix can contain roughly: relationships for each attention head. This is why memory and compute can grow quickly as sequences become longer, especially during training or the prompt processing stage. 4. Training usually needs even more memory During training, the model must keep additional intermediate values so gradients can be calculated during backpropagation. Longer training sequences therefore increase memory requirements for: Activations Attention calculations Gradients Temporary tensors Techniques such as activation checkpointing can reduce memory usage, but they usually trade memory savings for additional computation. 5. Numeric precision also matters Storing model states in lower precision formats can reduce memory consumption. For example, using fewer bits per cached value can reduce KV cache memory, although the exact implementation and quality tradeoffs depend on the system. Does memory usage grow only because of context length? No. Memory usage also depends on: Model size Batch size Number of layers Hidden dimensions Attention architecture Numeric precision KV cache implementation Context length is one important factor among several. For the broader context topic, see “What is context length in an LLM?” In short: longer context increases LLM memory usage because more token representations must be processed and stored. During inference, the KV cache grows with the number of active tokens, while training also needs memory for activations and gradient related computation.