Large Language Models

Large language models are deep learning models, usually transformer-based, trained to predict tokens from large corpora. After pretraining, they are often instruction-tuned and aligned so they can follow prompts and assist users.

What is ALiBi in Large Language Models?

ALiBi stands for Attention with Linear Biases . It is a positional method used in Transformer models to give attention information about how far apart tokens are in a sequence. Instead of adding positional embeddings to token representations, ALiBi modifies the attention scores directly . What is the main idea? In ALiBi, attention scores receive a distance based bias. Tokens that are farther apart receive a larger penalty. Conceptually: This gives the model information about relative token distance without adding a separate positional embedding. How does the bias work? Suppose one token is attending to earlier tokens. The farther an earlier token is from the current position, the more its attention score is reduced. Conceptually: The penalty grows approximately linearly with distance. Does every attention head use the same bias? No. Different attention heads use different slopes. One head may penalize distant tokens strongly, while another may penalize them more gradually. This allows different heads to operate over different effective distance ranges. How is ALiBi different from positional embeddings? Traditional positional embeddings add position information to token representations. ALiBi instead changes the attention score itself: This keeps the positional mechanism relatively simple. How is ALiBi different from RoPE? Both give Transformers positional information, but they do it differently. RoPE rotates query and key representations according to position. ALiBi adds a distance dependent bias to attention scores. So RoPE changes the representations used to calculate attention, while ALiBi changes the resulting attention scores. Why is ALiBi useful for long sequences? Because ALiBi is based on relative distance rather than a fixed learned embedding for each absolute position, it can generalize more naturally to sequence lengths beyond those used during training. However, this does not mean a model can handle arbitrarily long context without quality or compute limitations. Does ALiBi replace self attention? No. ALiBi is an addition to the attention calculation. The model still performs self attention; ALiBi simply introduces a positional bias into the attention scores. In short: ALiBi provides positional information by adding head specific linear distance penalties to attention scores. Nearby tokens receive smaller penalties, distant tokens receive larger ones, and the method avoids using separate positional embeddings.

What is grouped-query attention (GQA)?

Grouped query attention (GQA) is an attention design where multiple query heads are divided into groups, and the heads within each group share the same key and value projections. It sits between standard multi head attention (MHA) and multi query attention (MQA) . Why is GQA used? In standard multi head attention, each attention head usually has its own: Query projection Key projection Value projection During autoregressive inference, the model stores key and value states for previously processed tokens in the KV cache . If every query head also has its own key and value head, the KV cache can become large. GQA reduces this cost by sharing key and value heads across groups of query heads. How does GQA work? Suppose a model has several query heads. Instead of giving every query head a separate key/value pair, GQA groups them: Each query head still computes its own query representation, but several query heads reuse the same key and value representations. How is GQA different from multi head attention? With standard multi head attention: With GQA: This means there are fewer key/value heads than query heads. How is GQA different from multi query attention? Multi query attention takes sharing even further. In MQA, all query heads share a single set of key and value projections. Conceptually: GQA therefore provides a middle ground between the flexibility of MHA and the efficiency of MQA. Why can GQA improve inference efficiency? The KV cache stores keys and values for earlier tokens. Using fewer key/value heads means: Less KV cache memory Less memory bandwidth during decoding More efficient serving for long sequences or large batches These savings become especially important as context length grows. Does GQA always improve model quality? Not automatically. The number of query heads, groups, and key/value heads is an architectural choice. Too much sharing can reduce representational flexibility, while too little sharing gives smaller efficiency gains. GQA is popular because it can preserve much of the quality of multi head attention while reducing inference cost. For the related design, see “What is multi query attention (MQA)?” In short: grouped query attention lets several query heads share the same key and value heads. This reduces KV cache size and memory bandwidth while retaining more attention diversity than sharing one key/value head across all queries.

What is multi-query attention (MQA)?

Multi query attention (MQA) is an attention design where many query heads share a single set of key and value projections. It is designed to make Transformer inference more memory efficient, especially during autoregressive generation. How does MQA differ from standard multi head attention? In standard multi head attention, each head usually has its own: Query projection Key projection Value projection Conceptually: In MQA, the query heads remain separate, but they all share the same keys and values: Why share keys and values? During autoregressive inference, an LLM stores key and value states from earlier tokens in the KV cache . If every attention head has its own keys and values, this cache can consume a large amount of memory. MQA reduces the number of key/value heads to one, which can greatly reduce: KV cache memory usage Memory bandwidth Data movement during decoding What happens to the query heads? The model still keeps multiple query heads. Each query head can ask for different information, but all of them attend over the same shared keys and values. This preserves some diversity in attention while reducing the amount of cached data. How is MQA different from grouped query attention? Grouped query attention (GQA) is a middle ground. Conceptually: MQA provides the most aggressive key/value sharing of the three. Why is MQA useful for LLM serving? Token generation is often limited by how quickly the system can read the KV cache from memory. Reducing KV cache size can therefore improve inference efficiency, especially with: Long contexts Large batch sizes Many concurrent users Large models Does MQA always preserve model quality? Not necessarily. Sharing one set of keys and values reduces representational flexibility compared with standard multi head attention. In some architectures, GQA is used instead because it can provide a compromise between model quality and inference efficiency. For the middle ground approach, see “What is grouped query attention (GQA)?” In short: multi query attention keeps multiple query heads but makes all of them share one key head and one value head, reducing KV cache memory and memory bandwidth during LLM inference.

What is sliding-window attention?

Sliding window attention is an attention pattern where each token attends only to a limited neighborhood of nearby tokens instead of attending to every token in the full sequence. It is commonly used to reduce the memory and computation required for long context processing. How does it work? Suppose the attention window allows a token to look at only a few nearby positions. Conceptually: Tokens outside that local window are not directly attended to in that layer. In a causal language model, the window usually includes only a limited number of previous tokens. For example: where W is the window size. Why use a sliding window? With full self attention, every token may compare itself with every other token in the sequence. As sequence length grows, this becomes expensive. Sliding window attention limits the number of token to token comparisons. For a fixed window size, this can reduce attention cost from roughly quadratic growth with sequence length to roughly linear growth with sequence length times the window size. Does a token lose all information outside its window? Not necessarily. Although one layer sees only a local region, information can move across the sequence through multiple Transformer layers. Conceptually: So the model's effective receptive field can become larger than the window of a single layer. What is the trade off? The main advantage is efficiency. Benefits can include: Lower attention computation Lower memory usage Better scalability to long sequences The trade off is that distant tokens cannot directly interact within every attention layer. This can make some long range relationships harder to model if the architecture relies only on local windows. Can sliding window attention be combined with global attention? Yes. Some architectures combine local sliding window attention with mechanisms such as: Selected global tokens Occasional full attention layers Other sparse attention patterns This can preserve efficient local processing while still allowing important long distance interactions. Is sliding window attention the same as sparse attention? Sliding window attention is one type of sparse attention . Instead of allowing every possible token to token connection, it keeps only a structured local subset. Sparse attention is the broader idea and can use many different patterns. For that broader concept, see “What is sparse attention in LLMs?” In short: sliding window attention restricts each token to attending to a nearby region of the sequence. This reduces the cost of attention for long contexts while trading some direct long range connectivity for greater efficiency.

What is sparse attention in LLMs?

Sparse attention is an attention approach where each token attends to only a selected subset of other tokens instead of attending to every token in the sequence. The goal is to reduce the computation and memory required by full self attention, especially for long contexts. Why is sparse attention needed? In full self attention, every token can interact with every other allowed token. For a sequence with many tokens, the number of attention comparisons grows rapidly as the sequence becomes longer. Sparse attention reduces those comparisons by keeping only selected connections. Conceptually: How are the selected connections chosen? Different sparse attention designs use different patterns. Common examples include: Local or sliding window attention — tokens attend mainly to nearby tokens Global tokens — selected tokens can attend broadly across the sequence Strided or dilated patterns — attention skips across positions at regular intervals Block sparse attention — attention is allowed only between selected blocks of tokens Hybrid patterns — combine local and long range connections The pattern is usually defined by the model architecture rather than chosen independently for every token at runtime. How does sparse attention improve efficiency? Because fewer query key comparisons are calculated, sparse attention can reduce: Attention computation Memory usage Attention score storage The exact savings depend on the sparsity pattern. For example, a fixed local window can make attention cost grow roughly with sequence length times the window size instead of with all token pairs. What is the trade off? The main trade off is reduced direct connectivity. If two important tokens are far apart and the attention pattern does not connect them directly, information may need to travel through: Multiple Transformer layers Global tokens Other long range sparse connections A poorly designed sparse pattern can therefore make some long range dependencies harder to capture. Is sparse attention the same as sliding window attention? No. Sliding window attention is one specific type of sparse attention. Sparse attention is the broader category. For example: Does sparse attention mean the model ignores most of the context? Not necessarily. A token may not directly attend to every other token in every layer, but information can still propagate through multiple layers or through specially connected positions. The effectiveness depends on how the sparse pattern is designed. Is sparse attention always better than full attention? No. Full attention provides direct interaction between all allowed token pairs, which can be useful for complex long range relationships. Sparse attention trades some of that flexibility for better efficiency. For the local version, see “What is sliding window attention?” In short: sparse attention reduces the number of token to token attention connections by allowing only selected interactions. This can make long context processing more efficient, but the attention pattern must be designed carefully so important information can still flow across the sequence.

What is LLM pretraining?

LLM pretraining is the initial large scale training stage in which a language model learns general patterns of language from a very large collection of text and other training data. During pretraining, the model is not usually taught one specific application such as customer support, coding assistance, or summarization. Instead, it learns broad statistical patterns that can later support many tasks. What happens during pretraining? The model repeatedly receives token sequences and tries to solve a training objective. For a decoder only LLM, a common objective is next token prediction . For example: The model might be trained to predict: Its prediction is compared with the actual next token, and the model's parameters are adjusted to reduce the error. This process is repeated across enormous amounts of training data. What does the model learn? Through repeated prediction, the model can learn patterns related to: Grammar and syntax Word meanings and relationships Writing styles Common facts and associations Document structure Reasoning patterns present in the data Programming language patterns, when code is included The model does not store the training corpus as a normal searchable database. Instead, training changes the numerical parameters of the neural network. Why is it called pretraining? It is called pretraining because it happens before later stages that adapt the model for particular behaviors or use cases. A simplified lifecycle is: The exact post training pipeline differs across models. Is pretraining supervised? Most LLM pretraining is largely self supervised . The training signal can be created directly from the data itself. For next token prediction, the earlier tokens become the input and the actual following token becomes the target. This allows models to learn from huge datasets without requiring humans to manually label every example. Does pretraining create a chat assistant? Not necessarily. A pretrained base model primarily learns to continue or model text according to its training objective. Additional post training is commonly used to make the model better at: Following instructions Maintaining conversations Producing preferred response styles Following safety and behavioral constraints These stages are covered separately in the articles on fine tuning and post training. Is pretraining the same as fine tuning? No. Pretraining builds broad language capabilities from a very large general corpus. Fine tuning starts from an already pretrained model and trains it further on a smaller or more targeted dataset. For that distinction, see “What is the difference between pretraining and fine tuning an LLM?” In short: LLM pretraining is the large scale initial training stage where a model learns general language and data patterns by repeatedly solving objectives such as next token prediction across a massive corpus.

How are Large Language Models pretrained?

Large Language Models are pretrained by exposing them to very large datasets and repeatedly training them to solve a language modeling objective. For many modern generative LLMs, that objective is next token prediction . 1. Build a large training corpus Pretraining begins with a large collection of data. Depending on the model, the corpus may include: Web pages Books Articles Reference material Code Forums or discussions Other text sources The exact mixture varies by model. 2. Clean and prepare the data Raw data usually needs preprocessing before training. Typical steps can include: Removing low quality or malformed content Deduplicating repeated documents Filtering unwanted material Normalizing text formats Splitting data into training examples These steps help improve the quality of the training signal. 3. Tokenize the text The text is converted into tokens using the model's tokenizer. Conceptually: Those token IDs become the input sequences used during training. 4. Create the training objective For a decoder only LLM, the model commonly learns to predict the next token. For example: The same sequence provides many training targets because each position can be trained to predict the following token. 5. Run the forward pass The token sequence passes through the Transformer. The model produces probability distributions over the vocabulary for the tokens it must predict. Its predictions are then compared with the correct target tokens. 6. Calculate the loss A loss function measures how different the model's predictions are from the correct targets. If the correct token receives low probability, the loss is higher. If the model assigns higher probability to the correct token, the loss becomes lower. 7. Update the model parameters Using backpropagation and an optimization algorithm, the training system calculates how the model's parameters should change to reduce the loss. Conceptually: This process is repeated across enormous numbers of token sequences. 8. Repeat across the dataset Training continues over many batches of data. Over time, the model gradually learns statistical patterns related to: Language structure Meaning and associations Facts present in the data Writing styles Code patterns Common forms of reasoning found in the corpus What happens after pretraining? The result is usually a base model . It can then go through additional stages such as: Supervised fine tuning Instruction tuning Preference optimization Other post training methods These later stages adapt the pretrained model toward specific behaviors. In short: Large Language Models are pretrained by tokenizing enormous datasets, repeatedly predicting training targets such as the next token, measuring prediction error, and updating billions of model parameters through backpropagation until the model learns broad language patterns.

What is a pretraining corpus?

A pretraining corpus is the large collection of data used to train a language model during its initial pretraining stage. It provides the examples from which the model learns language patterns, facts, writing styles, code structures, and other statistical relationships. What can a pretraining corpus contain? Depending on the model, a corpus may include data such as: Web pages Books News and articles Reference documents Code Forums and discussions Academic or technical text Other publicly available or licensed text sources Some models may also include other modalities, but for a text LLM the corpus is primarily tokenized text. Why is the corpus important? The model can only learn patterns that are represented in its training data. The corpus therefore strongly influences: What topics the model has seen Which languages it handles well What writing styles it recognizes How much domain specific knowledge it can learn The biases and limitations it may inherit A larger corpus is useful only if the data is also sufficiently high quality and diverse. Is the corpus used directly by the model after training? No. During pretraining, the corpus is converted into token sequences and used to update the model's parameters. Conceptually: After training, the model does not normally query the original corpus like a search engine or database. Does a pretraining corpus need labels? Usually not for standard self supervised LLM pretraining. For next token prediction, the training targets come directly from the text itself. For example: This allows very large amounts of unlabeled text to be used. Is every document in the corpus equally useful? No. Low quality, duplicated, corrupted, or irrelevant data can reduce training efficiency or harm model quality. That is why pretraining pipelines commonly include: Data cleaning Deduplication Filtering Language detection Quality scoring Dataset balancing These topics are covered separately in the articles on training data cleaning and deduplication. Is a pretraining corpus the same as a dataset? A pretraining corpus is a type of training dataset, but the term corpus usually emphasizes a very large collection of text or language data assembled for model training. It may itself be built from many individual datasets and sources. In short: a pretraining corpus is the large, prepared collection of text and other training data that an LLM learns from during pretraining. Its quality, diversity, coverage, and composition strongly influence what the model ultimately learns.

How much data is needed to pretrain an LLM?

There is no single fixed amount of data required to pretrain an LLM. The amount depends on the model size, training objective, available compute, data quality, and the performance target. Why does model size matter? Larger models generally need more training data to use their capacity effectively. A model with relatively few parameters can reach diminishing returns sooner, while a much larger model may continue improving with substantially more tokens. Conceptually: The three need to be reasonably balanced. How is training data measured? For LLM pretraining, data is usually discussed in tokens , not simply documents or gigabytes. For example: A modern large scale model may be trained on hundreds of billions or even trillions of token occurrences, depending on its size and training strategy. Is more data always better? No. Adding more data helps only when the additional data contributes useful training signal. A smaller collection of high quality, diverse data can sometimes be more valuable than a much larger collection containing: Duplicates Spam Corrupted text Very low quality content Excessively repetitive material So data quality and diversity matter alongside raw quantity. Can the same data be used more than once? Yes. A training pipeline may expose the model to some examples multiple times. However, excessive repetition can lead to memorization or overfitting to particular samples. The ideal number of passes depends on the dataset size, model size, and training plan. What determines the required amount in practice? Important factors include: Number of model parameters Compute budget Training objective Dataset quality Dataset diversity Number of training steps Desired languages and domains Whether the model is general purpose or specialized For example, a narrow domain specific model may need far less data than a general purpose model expected to handle many languages, topics, and tasks. Is there an exact formula? There are empirical scaling laws that help researchers estimate good trade offs between model size, training tokens, and compute. But there is no universal rule such as: The best ratio depends on the architecture, data, budget, and training setup. What happens if there is too little data? A model may: Underuse its parameter capacity Generalize poorly Memorize training examples Learn narrow or biased patterns What if there is too much data? More data can still help, but eventually the additional training may provide smaller gains relative to its computational cost. At that point, developers must decide whether the extra improvement is worth the additional training resources. In short: the amount of data needed to pretrain an LLM depends on model size, compute, data quality, and training goals. Large general purpose LLMs commonly require extremely large token corpora, but simply adding more low quality data is not enough—the quantity, quality, and diversity of the training data must be balanced.

What types of data are used to train LLMs?

LLMs are trained on large mixtures of data rather than a single type of source. The exact dataset differs by model, but common categories include general text, code, reference material, and domain specific content. 1. Web text Large web datasets can provide broad coverage of: Everyday language Articles Blogs Forums Public documentation General knowledge Because raw web data can be noisy, it usually requires substantial filtering and cleaning before training. 2. Books and long form text Books and other long form documents can help models learn: Coherent writing Long range structure Narrative patterns Explanations Formal language Long form text is useful because it exposes the model to language beyond short web snippets. 3. Reference and educational material Training mixtures may include content such as: Encyclopedic text Technical documentation Tutorials Academic material Educational resources These sources can contribute structured factual and explanatory patterns. 4. Code Many modern LLMs are trained on source code in addition to natural language. Code data can include: Programming languages Configuration files Documentation Comments Examples This helps models learn syntax, APIs, programming patterns, and relationships between code and natural language instructions. 5. Conversational and discussion data Some datasets include dialogue like or discussion oriented text. This can help expose the model to: Questions and answers Informal language Different viewpoints Conversational structure However, noisy or low quality discussion data usually needs careful filtering. 6. Multilingual data Models intended to support multiple languages need training data from those languages. The amount and quality of data available for each language strongly influence multilingual performance. If one language is heavily represented while another has very little data, model capability may be uneven. 7. Domain specific data Some models include specialized material from areas such as: Science Law Finance Medicine Mathematics Engineering Specialized data can improve the model's exposure to domain terminology and patterns, although high stakes domains still require careful evaluation. 8. Synthetic data Training pipelines may also use synthetic data generated by models or other automated systems. Synthetic data can be useful for: Expanding underrepresented task types Creating structured examples Producing instruction response pairs Targeting specific capabilities Its quality must be controlled because errors in synthetic data can also be learned. Does every model use all of these data types? No. The exact training mixture depends on the model's goals, available data, licensing constraints, languages, domains, and training budget. A general purpose LLM may use a broad mixture, while a specialized model may emphasize a narrower set of sources. Is the raw data used directly? Usually not. Before training, data typically goes through steps such as: Cleaning Deduplication Filtering Quality scoring Language detection Formatting Tokenization Those preparation steps are covered separately in “How is training data cleaned for LLMs?” In short: LLMs are commonly trained on a mixture of web text, books, reference material, code, discussions, multilingual content, domain specific sources, and sometimes synthetic data. The quality and balance of that mixture strongly influence what the model learns.