Large Language Models
Large language models are deep learning models, usually transformer-based, trained to predict tokens from large corpora. After pretraining, they are often instruction-tuned and aligned so they can follow prompts and assist users.
Why is long-context processing computationally expensive?
Long context processing is computationally expensive because an LLM must process and keep track of many more token positions at the same time. As the context grows, attention work, memory usage, KV cache size, and latency all increase. 1. Self attention becomes more expensive In standard full self attention, each token can interact with many other tokens in the sequence. If the sequence length is (n), the number of pairwise attention relationships grows roughly like: So doubling the context length can increase the attention work by much more than two times during full sequence processing. 2. The model must process more token representations Every additional token passes through the model's layers. Longer sequences therefore require more computation for: Attention Feed forward layers Normalization Other per token operations Even operations that scale roughly linearly with sequence length become more expensive when millions of token positions are involved across many layers. 3. KV cache memory grows with context length During autoregressive generation, LLMs commonly store key and value representations for previous tokens in a KV cache . Conceptually: A larger cache also increases the amount of data the hardware must read while generating later tokens. 4. Prompt processing becomes slower Before generation begins, the model must process the input context. A very long prompt therefore increases the prefill workload before the first output token can be generated. This can increase time to first token. 5. Generating each new token can also become more expensive During decoding, a newly generated token attends to previously cached context. As the context becomes longer, there are more past positions to consider. So long contexts can increase the work required for each subsequent output token as well. 6. Hardware bandwidth becomes important Long context inference is not only about arithmetic. Large KV caches and intermediate tensors must be moved through GPU memory. Memory bandwidth and available GPU memory can therefore become major bottlenecks. How do LLMs reduce this cost? Modern systems may use techniques such as: FlashAttention style optimized attention Sliding window attention Sparse attention KV cache compression or quantization Grouped query or multi query attention These techniques reduce memory or computation, but long contexts still require more resources than short ones. For the memory side specifically, see “How does context length affect LLM memory usage?” In short: long context processing is expensive because more tokens create more attention work, larger intermediate states, bigger KV caches, greater memory traffic, and higher prompt processing and generation latency.
What is long-context modeling?
Long context modeling is the design and training of language models so they can process and use information across very long token sequences. The goal is not only to let the model accept more tokens, but also to help it use information that may be far apart within the context. What does “long context” mean? A long context can include: Large documents Long conversations Multiple retrieved passages Large code files Many examples or instructions For example, instead of processing only a few thousand tokens, a long context model may be designed to work with tens of thousands or more. Why is long context modeling useful? Many real tasks require information that does not fit into a short prompt. Examples include: Summarizing long reports Analyzing books or research papers Reviewing large codebases Working with long chat histories Comparing information across many documents Reasoning over distant parts of the same text Long context modeling allows more of this information to be supplied in one inference request. What makes long context modeling difficult? A larger context window creates several challenges. The model must: Represent token positions over much longer ranges Attend to relevant information among many more tokens Maintain reasonable memory usage Control attention and KV cache costs Avoid losing important information in the middle of long sequences So: A model may technically accept a long sequence but still struggle to retrieve or use information effectively across the entire sequence. How do models support longer contexts? Long context models may use techniques such as: Long sequence training Improved positional representations Rotary Position Embedding scaling Sliding window attention Sparse attention More efficient attention implementations KV cache optimizations Different models combine these techniques in different ways. Is long context modeling the same as RAG? No. Long context modeling increases how much information the model can process directly in one context. Retrieval Augmented Generation (RAG) selects relevant information from an external knowledge source and adds only that information to the prompt. The two approaches can also be combined. For example: Does a longer context remove the need for retrieval? Not necessarily. Even a model with a very large context window may benefit from retrieval because selecting only relevant information can reduce cost, latency, and distraction from irrelevant text. For the computational tradeoff, see “Why is long context processing computationally expensive?” In short: long context modeling focuses on making LLMs capable of processing and effectively using information across very long token sequences, using suitable training, positional methods, attention strategies, and memory optimizations.
What is context compression in LLMs?
Context compression in LLMs is the process of reducing the number of tokens in the information given to a model while trying to preserve the details that are most useful for the task. It is commonly used when prompts, conversation histories, or retrieved documents are too large or unnecessarily verbose. Why is context compression used? LLMs have limited context windows, and longer contexts usually require more memory, computation, and cost. Context compression tries to turn: The goal is to keep important information while removing material that contributes little to the answer. How can context be compressed? Common approaches include: Summarizing older conversation turns Removing duplicate or repetitive text Extracting only important facts Selecting only relevant document sections Removing boilerplate or formatting that is not useful Rewriting verbose text into a shorter form In retrieval based systems, returning only the most relevant passages can also reduce the amount of context sent to the model. Example Suppose a conversation history contains 20,000 tokens, but most of the earlier discussion can be summarized as: Instead of sending the entire old conversation, an application might send that shorter summary plus the most recent messages. Is context compression lossless? Usually not. Many context compression methods are lossy , meaning some information is removed. That creates an important tradeoff: A good compression strategy therefore tries to preserve information that is likely to matter for the current task. Context compression vs. truncation They are not exactly the same. Context compression tries to reduce size while preserving important information. Context truncation simply removes part of the context, often based on position or a token limit. For that distinction, see “What is context truncation?” Does context compression change the model itself? No. Context compression usually happens before the prompt is sent to the LLM. It changes the information provided to the model, not the model's parameters. In short: context compression reduces the token size of prompts, histories, or documents while attempting to preserve the most relevant information, helping applications fit useful context within an LLM's limited context window.
What is context truncation?
Context truncation is the process of removing part of an LLM's input context so the remaining token sequence fits within the model's context limit. Unlike context compression, truncation usually removes tokens rather than rewriting or summarizing them. Why is truncation needed? An LLM can process only a limited number of tokens at one time. If a request becomes too long, an application may need to shorten it before sending it to the model. Conceptually: What parts of the context can be truncated? An application may remove: Older conversation turns The beginning of a document The end of a document Less important retrieved passages Repeated or low priority content Which part is removed depends on the application's truncation strategy. What are left and right truncation? Two common approaches are: Left truncation — remove tokens from the beginning of the sequence. Right truncation — remove tokens from the end of the sequence. For example: In conversation systems, left truncation may be used to remove older messages while preserving recent turns. What is the risk of truncation? Truncation can remove information that is important for the current task. If a critical instruction or fact is removed, the model cannot directly use it in that inference request. This can lead to: Missing details Inconsistent answers Forgotten earlier instructions Incorrect conclusions based on incomplete context Context truncation vs. context compression They solve a similar problem but in different ways. Truncation removes part of the context. Compression tries to preserve important information in fewer tokens, for example by summarizing or extracting key facts. Conceptually: For the related concept, see “What is context compression in LLMs?” Is truncation always automatic? No. Some APIs reject an oversized request instead of truncating it automatically. In other systems, the application decides what to remove before making the request. Developers therefore need to understand the model's token limit and the behavior of the specific API or framework they use. In short: context truncation removes tokens from an oversized prompt or conversation so it fits within an LLM's context window. It is simple and efficient, but it can discard important information if the truncation strategy is not chosen carefully.
How does an LLM handle very long text?
An LLM handles very long text by fitting as much relevant information as possible inside its context window and using application level techniques when the text is larger than that limit. The exact approach depends on the model, its context size, and the system built around it. 1. If the text fits, the model processes it directly When the full text is within the supported context window, it can be tokenized and sent to the model in one request. The model can then use attention to relate information across the available sequence. 2. If the text is too long, it must be reduced or divided When the text exceeds the model's context limit, an application may: Split it into smaller chunks Truncate less important sections Summarize parts of the text Retrieve only the sections relevant to the current question Process different sections in separate requests So: 3. Chunking can process a document in parts A long document can be divided into smaller sections that fit comfortably within the model's context window. Each chunk can then be: Summarized independently Analyzed separately Indexed for later retrieval Combined into a higher level summary For example, a long report might be processed chapter by chapter before the individual summaries are combined. 4. Retrieval can select only relevant sections Instead of sending the entire document, a system can search it for passages related to the user's question and place only those passages into the prompt. This is common in Retrieval Augmented Generation systems. It can reduce token usage while keeping task relevant information available to the model. 5. Summarization can compress earlier information Applications can convert long passages or conversation histories into shorter summaries. This preserves some important facts while reducing context length. However, summarization is usually lossy, so details that are omitted from the summary may no longer be available to the model. 6. Long context models can accept more text directly Some LLMs are specifically designed and trained for much larger context windows. They can process more text in a single request, but larger context still increases memory, computation, and latency. Also, fitting information inside the window does not guarantee that the model will use every part equally well. Does the LLM automatically remember the whole document? No. The model can directly use only the information supplied in its current context. If earlier sections were removed, summarized, or processed in another request, the application must bring the relevant information back when needed. For the underlying limit, see “What is a context window in an LLM?” In short: an LLM handles very long text by processing it directly when it fits, or by using chunking, retrieval, summarization, truncation, and multiple requests when it exceeds the available context window.
What is the lost-in-the-middle problem in LLMs?
The lost in the middle problem is a long context behavior where an LLM may use information near the beginning or end of a long prompt more reliably than equally important information placed in the middle. In other words, information can technically be inside the context window but still have less influence on the model's answer because of where it appears. Simple example Suppose a long prompt contains many facts: If the question depends on Fact B , the model may sometimes overlook it even though it is present in the prompt. Why can this happen? Several factors can contribute: The model may not attend equally effectively to all positions in a very long sequence. Important information can be buried among large amounts of unrelated text. Long context training may not make the model equally strong at retrieving information from every position. Competing instructions or facts can reduce the influence of the relevant passage. The exact behavior depends on the model, task, and prompt structure. Does this mean the model cannot access the middle? No. The information is still inside the context window. The problem is that availability does not guarantee effective use . A model may successfully use middle position information in one task and struggle with it in another. Why is this important? The lost in the middle problem matters when working with: Long documents Large conversation histories Large retrieved context Multi document prompts Long code files Simply placing all available information into a very large context window may not produce the best result. How can applications reduce the problem? Useful approaches include: Retrieve only the most relevant passages Remove unnecessary context Put critical instructions in prominent positions Summarize long sections Split very large tasks into smaller focused requests Repeat or restate essential information when appropriate The goal is to make important information easier for the model to identify and use. Is this the same as exceeding the context window? No. When the context window is exceeded, some information may not fit at all. With the lost in the middle problem, the information does fit inside the context , but the model may still fail to use it effectively. For the broader issue, see “Why can LLMs struggle with information in long contexts?” In short: the lost in the middle problem occurs when an LLM is less effective at using important information buried in the middle of a long context, even though that information is still within the model's supported context window.
Why can LLMs struggle with information in long contexts?
LLMs can struggle with information in long contexts because having more tokens available does not guarantee that the model will identify, retain, and use every relevant detail equally well. As the context grows, important information has to compete with much more surrounding text. 1. Relevant information can become harder to find A long context may contain thousands of tokens, but only a small part may matter for the current question. The model must effectively locate that information among: Repeated details Unrelated passages Similar looking facts Multiple instructions Large amounts of background text This makes retrieval from the prompt more difficult. 2. Position can affect how well information is used Information placed at different positions in a long context may not influence the model equally. A known example is the lost in the middle problem , where important information buried in the middle of a long sequence can sometimes be used less reliably than information near the beginning or end. For that specific behavior, see “What is the lost in the middle problem in LLMs?” 3. More context can create more competing signals Long prompts often contain multiple facts, examples, or instructions. If some of them are irrelevant, outdated, duplicated, or contradictory, the model must determine which information should guide the response. More tokens can therefore add useful evidence, but they can also add noise. 4. Connecting distant information can be difficult Some tasks require the model to combine facts that appear far apart in the sequence. For example: This can be more difficult than reasoning over information that appears close together. 5. Long context capability depends on training A model may support a large context window technically, but its ability to use that full range depends on how it was trained and evaluated. A larger supported context does not automatically mean equally strong performance at every context length. How can this be improved? Applications can make long contexts easier to use by: Retrieving only relevant passages Removing duplicated or irrelevant text Structuring prompts clearly Placing important instructions prominently Summarizing older information Splitting complex tasks into smaller focused steps The goal is to provide the model with relevant context , not simply the maximum possible amount of context. In short: LLMs can struggle with long contexts because important information becomes harder to locate, position can matter, irrelevant text creates competition, and distant facts can be difficult to connect reliably.
Can an LLM remember information outside its context window?
An LLM cannot directly use information that has fallen outside its active context window unless that information is supplied again or recovered through an external mechanism. The context window acts like the model's working input for the current inference request. What happens when earlier information leaves the context? Suppose a long conversation originally contained: If that message is later removed from the active context, the model no longer has direct access to that specific conversation detail during the current request. It may appear to have "forgotten" it because the information is no longer being provided. Does the model have any memory at all? An LLM has information stored in its learned parameters from training, but that is different from remembering a specific earlier conversation. So there are two different ideas: The model may already know a general fact from training, but it cannot recover a private conversation detail that is no longer present just from its parameters. How can applications provide longer term memory? An application can store information outside the model and bring it back when needed. Common approaches include: Conversation summaries Databases Retrieval systems User profile stores Vector databases Structured application state Conceptually: The LLM can then use the retrieved information because it has been returned to the active context. Is external memory the same as a larger context window? No. A larger context window allows more tokens to remain directly available at one time. External memory stores information outside the active context and selectively retrieves it later. These approaches can be combined. What about chat applications that seem to remember old details? The application may be: Re sending older messages Maintaining a summary Retrieving stored user information Injecting saved state into the prompt In those cases, the apparent memory is partly provided by the surrounding application rather than by the LLM's context window alone. Can the model infer missing information? Sometimes the model may guess or reconstruct something from clues that remain in the current context. That is not the same as actually remembering the missing information and can be wrong. In short: an LLM cannot directly access conversation information that is outside its active context window. Longer term memory requires the application to store, summarize, retrieve, or reintroduce that information into the model's current context.
What is effective context length?
Effective context length is the amount of context an LLM can use reliably for a task, which may be smaller than its advertised or maximum context window. A model may technically accept a very long sequence while still becoming less accurate at finding or using information across the entire sequence. Maximum context vs. effective context Suppose a model supports: That means the model can accept a sequence up to that size. But if performance begins to drop when important information is spread across very long inputs, its effective context length for that task may be lower. Conceptually: Why can effective context be smaller? Several factors can reduce effective context length: Important facts may be buried among irrelevant text. Information near some positions may be harder to use than others. The model may struggle to connect details that are far apart. Long prompts may contain competing instructions or repeated information. The model may not have been trained strongly enough on very long sequences. This is why a larger context window does not automatically guarantee equally strong performance at every token position. Is effective context length fixed? No. It depends on the task. A model might handle a long summarization task well but struggle with a question that requires finding one small fact hidden deep inside the same amount of text. It can also depend on: Prompt structure Position of relevant information Amount of irrelevant context Model architecture Long context training Retrieval difficulty How is it evaluated? Effective context length is usually studied by testing whether the model can still retrieve, combine, or reason over information as the input becomes longer. For example, an evaluation might place an important fact at different positions in a long prompt and check whether the model can still answer correctly. The goal is to measure useful long context behavior rather than just the maximum number of tokens the model accepts. Why does it matter in applications? Developers should not assume that filling the entire advertised context window is always optimal. A shorter prompt containing only relevant information may sometimes perform better than a much larger prompt full of unnecessary material. This is why retrieval, summarization, and context compression can still be valuable even with long context models. For the related concept, see “What is context utilization in LLMs?” In short: effective context length is the amount of context an LLM can use reliably in practice. It can be smaller than the model's maximum context window and depends on the task, prompt structure, information position, and model capability.
What is context utilization in LLMs?
Context utilization in LLMs describes how effectively a model uses the information that is available inside its context window. A model may be able to accept a very long prompt, but that does not mean every part of the prompt contributes equally to the answer. What does good context utilization mean? Good context utilization means the model can: Find relevant information in a long prompt Follow instructions given earlier in the context Connect facts that appear far apart Ignore irrelevant or distracting text Use retrieved documents accurately Maintain important details across a long conversation The key idea is not just whether information fits inside the context window, but whether the model actually uses it when needed. Example Suppose a prompt contains 50,000 tokens and one important fact appears near the middle: If the model correctly uses that fact when answering a later question, it is demonstrating effective context utilization. If it overlooks the fact even though it is still inside the context window, context utilization is weaker. What affects context utilization? Several factors can influence it: Position of important information Total context length Amount of irrelevant text Repeated or conflicting information Prompt organization Model architecture Long context training The difficulty of the reasoning task This is why two prompts with the same token count can produce very different results. How is it different from context length? Context length tells us how many tokens are currently present. Context utilization tells us how effectively the model makes use of those tokens. Conceptually: A model can therefore have a large context window but poor utilization of some parts of it. How can applications improve context utilization? Useful approaches include: Remove irrelevant text Retrieve only relevant passages Organize context clearly Put critical instructions in prominent locations Summarize repetitive history Separate different sources or sections clearly Keep related information close together when possible These techniques reduce the amount of competition for the model's attention. How is this related to effective context length? Effective context length depends partly on context utilization. If a model technically accepts 128,000 tokens but reliably uses only a smaller portion for a particular task, its practical or effective context may be lower. For that concept, see “What is effective context length?” In short: context utilization is how effectively an LLM finds, connects, and uses relevant information within its available context. A large context window is useful only when the model can make good use of the information placed inside it.