Large Language Models
Large language models are deep learning models, usually transformer-based, trained to predict tokens from large corpora. After pretraining, they are often instruction-tuned and aligned so they can follow prompts and assist users.
What is an attention mask in an LLM?
An attention mask is a mechanism that tells an LLM which token positions are allowed to participate in attention and which positions should be ignored or blocked. It is applied during the attention calculation so the model does not treat every token position as equally available. Why is an attention mask needed? Different tasks require different attention rules. For example, an attention mask can be used to: Prevent a decoder only model from seeing future tokens Ignore padding tokens in a batch Restrict attention to certain parts of a sequence Support specialized attention patterns Conceptually: How does masking work? Self attention first produces compatibility scores between token positions. Before those scores are converted into attention weights, the mask modifies the disallowed positions so that they receive effectively zero attention after softmax. Conceptually: The exact implementation can vary, but the result is that masked positions do not contribute to the attention output. What is a padding attention mask? When sequences of different lengths are processed together, shorter sequences may contain padding tokens. For example: A padding mask tells the model not to treat those <PAD positions as meaningful context. What is a causal attention mask? Decoder only LLMs use a causal mask to prevent each position from attending to future positions. For example: Future tokens are masked. This preserves autoregressive next token prediction. The causal version is covered separately in “What is a causal attention mask?” Is an attention mask itself a token? No. An attention mask is usually an additional tensor or set of rules used during the attention calculation. It is different from special tokens such as <PAD or <EOS , although masks are often used together with those tokens. Does every model use the same mask format? No. The exact representation depends on the architecture and framework. Some systems use binary masks such as: while others internally convert masks into large negative values before softmax. The principle is the same: control which positions can influence attention. In short: an attention mask controls which token positions an LLM is allowed to attend to. It is commonly used to block future tokens, ignore padding, or enforce other attention patterns.
What is a causal attention mask?
A causal attention mask is a mask used in autoregressive language models to prevent each token from attending to tokens that come later in the sequence. It enforces the rule: Why is it needed? During training, the complete sequence is available to the model. Without a causal mask, a token could directly access future tokens, including information it is supposed to predict. That would break next token prediction. The causal mask ensures the model learns under the same information constraint used during generation. What does the mask look like? For a sequence of four tokens, the allowed attention pattern is conceptually: Positions to the right are blocked. This creates a triangular attention pattern. How is the mask applied? Self attention first computes scores between token positions. The causal mask then marks future positions as disallowed before softmax is applied. Conceptually: In many implementations, blocked positions are assigned a very large negative value so their probability becomes effectively zero after softmax. How does this support text generation? Suppose the sequence so far is: When predicting the next token, the model can use: but it cannot use tokens that have not yet been generated. After a new token is produced, it is added to the sequence and becomes available for the next prediction. Is a causal mask the same as a padding mask? No. A causal mask blocks future positions . A padding mask blocks padding tokens that should not contribute useful information. A model may use both kinds of masking at the same time. Is a causal attention mask the same as causal self attention? They are closely related but not identical concepts. Causal attention mask is the masking rule that blocks future positions. Causal self attention is self attention operating under that causal restriction. For the broader mechanism, see “What is causal self attention?” In short: a causal attention mask prevents tokens from seeing future positions, creating the left to right information flow required for autoregressive next token prediction.
What are query, key, and value representations in LLM attention?
In LLM attention, query, key, and value representations are three learned versions of each token representation that help the model decide which other tokens are relevant and what information to take from them. They are commonly abbreviated as: Q = Query K = Key V = Value What does the query represent? The query represents what the current token is looking for. For example, if the model is processing a pronoun such as: its query can help identify which earlier token is most relevant to that pronoun. What does the key represent? The key represents information used to determine how relevant a token is to another token's query. The model compares a query with the keys of allowed tokens. A stronger match generally produces a higher attention score. Conceptually: What does the value represent? The value contains the information that can actually be passed forward if that token receives attention. After the model calculates attention weights from the query key comparisons, it uses those weights to combine the corresponding value representations. So: Where do Q, K, and V come from? Each token starts with an internal representation. The model applies learned linear transformations to create separate query, key, and value vectors: These transformations are learned during training. Why use three different representations? Using separate Q, K, and V representations gives the model different roles for: Asking what information is relevant Describing what information a token can match against Carrying the information that will be combined This makes attention flexible because the same token can be useful in different ways depending on the surrounding context. Simple example Consider: When processing: its query may strongly match the key associated with: That token can therefore receive a higher attention weight, allowing its value representation to contribute more strongly to the representation of she. Are Q, K, and V the same across all attention heads? No. In standard multi head attention, different heads usually learn their own query, key, and value projections. This allows different heads to focus on different relationships in the same sequence. That idea is covered separately in “What is multi head attention in an LLM?” In short: queries determine what a token is looking for, keys help measure which tokens are relevant, and values carry the information that is combined according to the resulting attention weights.
What is multi-head attention in an LLM?
Multi head attention is an attention mechanism where an LLM performs several attention calculations in parallel instead of relying on a single attention pattern. Each attention calculation is called an attention head . Why use multiple heads? Different heads can learn to focus on different kinds of relationships in the same sequence. For example, one head may focus more on: Nearby grammatical relationships Long distance dependencies References between words Important instruction tokens Other useful token interactions The model does not manually assign these roles. They are learned during training. How does it work? The token representations are projected into separate query, key, and value representations for each head. Conceptually: Each head computes its own attention weights and produces its own output. The outputs from all heads are then combined and transformed before being passed to the next part of the Transformer layer. What does each head see? In standard multi head attention, every head typically receives information derived from the same token sequence, but each head uses its own learned projections. That means two heads can assign very different attention weights to the same tokens. Does more attention heads always mean a better model? No. The number of heads is an architectural choice. Using more heads can increase the number of distinct attention patterns the model can represent, but model quality also depends on factors such as: Hidden dimension Number of layers Training data Training objective Overall architecture Compute budget More heads alone do not guarantee better performance. How is this related to query, key, and value? Each attention head works with query, key, and value representations. In standard multi head attention, different heads usually have separate learned Q, K, and V projections. For the underlying representations, see “What are query, key, and value representations in LLM attention?” Are there alternatives? Yes. Modern LLMs may use alternatives such as: Multi query attention (MQA) Grouped query attention (GQA) These approaches reduce some of the memory and computation associated with standard multi head attention, especially during inference. They are covered separately in their dedicated articles. In short: multi head attention lets an LLM compute several attention patterns in parallel, allowing different heads to capture different relationships between tokens before their outputs are combined.
Why do LLMs use multiple attention heads?
LLMs use multiple attention heads so the model can learn several different attention patterns at the same time instead of relying on a single way of relating tokens. Each head performs its own attention calculation using learned projections. Why is one attention head not enough? A sentence can contain many kinds of relationships at once. For example, the model may need to recognize: Which noun a pronoun refers to Which words form a phrase Which earlier instruction is relevant Which distant token provides important context Which nearby tokens matter for grammar A single attention pattern would have to represent all of these relationships at once. Multiple heads give the model more flexibility. How does this work? Each head receives the same sequence, but usually uses its own learned query, key, and value projections. Conceptually: The outputs from the heads are combined and passed forward through the Transformer layer. What can different heads learn? Different heads may become sensitive to different relationships. For example, one head might strongly connect: while another might focus on the relationship between a verb and its object. However, these roles are not manually assigned, and individual heads are not always easy to interpret. Does every head always learn something unique? No. Some heads may learn overlapping behavior, and some may be more important than others. The benefit of multi head attention is not that every head has a perfectly separate human readable job. The benefit is that the model has multiple learned subspaces in which to compute attention. Why is this useful for LLMs? Multiple heads help the model represent several contextual relationships simultaneously. This improves its ability to build rich token representations across complex sequences. It is especially useful when the same token needs to participate in several relationships at once. Does adding more heads always improve the model? No. The number of attention heads must fit the model's hidden dimension, architecture, memory budget, and compute budget. More heads increase architectural capacity, but they do not automatically improve quality. Modern models may also use alternatives such as grouped query attention or multi query attention to reduce inference cost. For the mechanism itself, see “What is multi head attention in an LLM?” In short: LLMs use multiple attention heads so they can learn and combine several different token to token attention patterns in parallel, giving the model a richer representation of context than a single attention calculation would provide.
What is positional information in an LLM?
Positional information is the information an LLM uses to understand where each token appears in a sequence. Self attention can compare tokens with one another, but by itself it does not inherently know their order. Positional information gives the model a sense of sequence. Why is position important? Word order can completely change meaning. For example: The same words appear in both sentences, but their positions are different. An LLM needs positional information to distinguish these sequences. How is positional information added? Different models use different methods. Common approaches include: Positional embeddings Rotary Position Embeddings (RoPE) ALiBi Other learned or relative position methods These methods give the model information about where tokens occur or how far apart they are. Where does positional information enter the model? Conceptually: In some architectures, positional information is added directly to token embeddings. In others, such as RoPE, position is incorporated into the attention calculation itself. What does positional information help the model learn? It helps the model reason about things such as: Which token came first Which words are close together Which words are far apart Sentence structure Ordering of instructions Relationships across a sequence Without positional information, two sequences containing the same tokens in different orders could appear much more similar to the attention mechanism than they should. Is position always represented as an absolute token number? No. Some methods represent absolute positions, while others focus more on relative relationships between positions. For example: can sometimes be more useful than knowing only that a token is at position 127. Does positional information affect context length? Yes. The positional method used by a model can influence how well it handles longer sequences and whether it can generalize beyond lengths seen during training. That is one reason modern LLM architectures pay significant attention to positional encoding design. Dedicated articles cover “What are positional embeddings in LLMs?” , “What is Rotary Position Embedding (RoPE)?” , and “What is ALiBi in Large Language Models?” In short: positional information tells an LLM where tokens occur and how they relate in sequence order, allowing the Transformer to distinguish differently ordered text and model relationships across the context.
Why do LLMs need positional information?
LLMs need positional information because the Transformer’s attention mechanism does not inherently know the order of tokens in a sequence. Without position information, the model would have much more difficulty distinguishing between sequences that contain the same tokens in different orders. Why does token order matter? Consider: Both sentences contain almost the same words, but the meaning changes because the order changes. The model therefore needs information about where each token appears. What problem does positional information solve? Self attention compares tokens with one another, but it mainly operates on their learned representations. Positional information adds clues such as: Which token came before another Which token came after another How far apart two tokens are Where a token appears in the sequence This helps the model preserve sequence structure. Why is this important for language? Language depends heavily on order. Positional information helps the model capture: Grammar Word order Sentence structure Instruction order Long distance relationships Sequence dependent meaning For example: The placement of only changes the meaning. Why can’t token embeddings handle this alone? A token embedding mainly represents information about the token itself. If the same token appears in different positions, its base token embedding may be the same. Positional information gives the model an additional signal that helps distinguish those occurrences. Conceptually: How do LLMs represent position? Different architectures use different methods, including: Positional embeddings Rotary Position Embeddings (RoPE) Relative position methods ALiBi These approaches represent position in different ways, but they all help the model reason about sequence order. Does positional information matter for long contexts? Yes. The positional method can affect how effectively a model handles long sequences and positions beyond those commonly seen during training. This is one reason positional techniques are an important part of long context LLM design. For the underlying concept, see “What is positional information in an LLM?” In short: LLMs need positional information because attention alone does not provide a strong notion of token order. Position signals allow the model to distinguish differently ordered sequences and understand relationships that depend on where tokens appear.
What are positional embeddings in LLMs?
Positional embeddings are representations that give an LLM information about where each token appears in a sequence. Because Transformer attention does not inherently encode token order, positional embeddings provide an additional signal that helps the model distinguish between different positions. Why are positional embeddings needed? Consider: The words are similar, but their order changes the meaning. The model therefore needs information about token position in addition to token identity. How do positional embeddings work? Each position is associated with a position representation. Conceptually: This combined representation tells the model both: What the token is Where the token appears Are positional embeddings learned? They can be. One common approach is to create a trainable embedding for each supported position. For example: These vectors are optimized during training along with the rest of the model. Other Transformer designs use fixed mathematical position encodings instead of learned embeddings. Are positional embeddings the only way to represent position? No. Modern LLMs may use other positional techniques, such as: Rotary Position Embeddings (RoPE) Relative position methods ALiBi These approaches may incorporate positional information directly into attention rather than simply adding a position vector to the token embedding. What is the difference between positional information and positional embeddings? Positional information is the general concept of telling the model about token order and relative position. Positional embeddings are one specific method for representing that information. So: Do positional embeddings affect context length? They can. If a model uses learned embeddings for a fixed range of positions, the supported sequence length may be tied to how those position representations were designed and trained. Other positional approaches are often chosen partly because they can behave differently when sequence lengths become large. For related concepts, see “What is positional information in an LLM?” and “What is Rotary Position Embedding (RoPE)?” In short: positional embeddings attach a learned or predefined representation to token positions so a Transformer can distinguish token order and use sequence structure during attention.
What is Rotary Position Embedding (RoPE)?
Rotary Position Embedding (RoPE) is a method used in Transformer models to give attention information about token positions. Instead of adding a separate position vector to each token embedding, RoPE incorporates position directly into the query and key representations used by self attention. What problem does RoPE solve? Self attention does not inherently know the order of tokens. RoPE gives the model information about: Where tokens appear in the sequence How far apart tokens are The relative position between tokens This helps the model distinguish sequences whose meaning depends on word order. What is the core idea? RoPE applies position dependent rotations to parts of the query and key vectors. Conceptually: Because different positions receive different rotations, the attention calculation can reflect relative positional relationships between tokens. Why is RoPE useful? RoPE has several practical advantages: It integrates position directly into attention. It naturally represents relative positional relationships. It does not require adding a separate learned position embedding to every token representation. It works efficiently with Transformer attention. It has become a common positional method in modern generative LLMs. Is RoPE the same as positional embeddings? No. Traditional positional embeddings may be added directly to token embeddings. RoPE instead modifies the query and key representations used in attention. So: Does RoPE remove context length limits? No. RoPE helps represent positions, but it does not make an LLM's context window unlimited. How well a RoPE based model works at very long sequence lengths depends on factors such as its training length, architecture, RoPE configuration, and any position scaling techniques used. Is RoPE used by every LLM? No. Different models can use different positional methods, including learned positional embeddings, relative position techniques, RoPE, or ALiBi. For the mathematical mechanism behind the rotations, see “How does RoPE work in LLMs?” In short: RoPE is a positional technique that rotates attention query and key representations according to token position, allowing Transformer attention to incorporate both token content and relative positional relationships.
How does RoPE work in LLMs?
RoPE works by applying a position dependent rotation to the query and key vectors used in Transformer attention. Instead of adding a separate position vector to a token embedding, RoPE changes the orientation of parts of the query and key vectors according to the token's position. Step 1: Create query and key vectors Each token representation is projected into a query and key: These vectors contain information about the token and its context. Step 2: Split dimensions into pairs RoPE treats dimensions in pairs. For a small example: can be viewed as: Each pair can then be rotated in a two dimensional plane. Step 3: Rotate each pair according to position The rotation angle depends on the token's position. Conceptually: Different dimension pairs use different rotation frequencies, allowing the representation to encode positional patterns at multiple scales. Step 4: Use the rotated vectors in attention Attention compares the rotated query of one token with the rotated keys of other tokens. A useful property of these rotations is that the resulting query key relationship naturally depends on the relative distance between positions . For example, attention can distinguish whether two tokens are: rather than relying only on their absolute position numbers. Why do rotations capture relative position? If two vectors are rotated according to positions m and n, their interaction contains information related to the difference: This gives attention a natural way to represent relative positional relationships. Does RoPE rotate the value vectors too? In the standard RoPE approach, position dependent rotation is applied to the query and key representations used to compute attention scores. The value vectors are typically not rotated in the same way. Why use different rotation frequencies? Different frequencies allow different dimension pairs to respond to position at different scales. Some dimensions change more rapidly with position, while others change more slowly. This gives the model a richer positional signal across short and long token distances. Does RoPE make long context automatically reliable? No. RoPE provides positional structure, but long context performance still depends on training, model architecture, context window design, and any RoPE scaling or extension technique used. For the high level concept, see “What is Rotary Position Embedding (RoPE)?” In short: RoPE works by rotating pairs of query and key dimensions according to token position. Because attention compares these rotated vectors, the resulting scores naturally carry information about the relative positions of tokens.