Large Language Models
Large language models are deep learning models, usually transformer-based, trained to predict tokens from large corpora. After pretraining, they are often instruction-tuned and aligned so they can follow prompts and assist users.
What architecture is commonly used in Large Language Models?
The architecture most commonly used in Large Language Models is the Transformer architecture. Transformers became the foundation of modern language models because they can process token relationships efficiently using attention mechanisms rather than relying on recurrent processing. What are the main parts of a Transformer? A Transformer based language model is built from repeated layers containing components such as: Self attention — lets tokens use information from other relevant tokens in the sequence. Feed forward networks — transform the representation of each token after attention. Residual connections — help information flow through many layers. Normalization layers — help stabilize training. Positional information — tells the model where tokens appear in the sequence. Conceptually: What Transformer architecture do LLMs use? There are three common Transformer arrangements: Encoder only Decoder only Encoder decoder Many modern generative LLMs use a decoder only Transformer because this architecture is naturally suited to predicting the next token repeatedly. Encoder only and encoder decoder architectures are also important, but they are commonly used for somewhat different language modeling tasks. These architectures are covered separately in: “What is a decoder only LLM?” “What is an encoder only language model?” “What is an encoder decoder language model?” Why are Transformers well suited to LLMs? Transformers are useful because they can: Model relationships between tokens across a sequence Scale to very large parameter counts Train efficiently on modern parallel hardware Work with long sequences Learn rich contextual representations Support autoregressive text generation Their ability to scale with large datasets and large amounts of compute is one reason they became the dominant architecture for LLMs. Is every LLM exactly the same Transformer? No. Models can differ in: Attention design Number of layers Hidden dimensions Positional representation Feed forward structure Normalization strategy Context window design Dense vs. mixture of experts layers So two LLMs can both be Transformer based while having very different internal designs. Are Transformers the only possible architecture? No. Researchers continue to explore alternatives and hybrid architectures that aim to reduce memory or computation, especially for long sequences. However, the Transformer remains the standard architecture behind most widely used LLMs. In short: Large Language Models are most commonly built using the Transformer architecture. Transformers combine self attention, feed forward layers, positional information, residual connections, and normalization to learn contextual representations and generate language effectively.
What is a decoder-only LLM?
A decoder only LLM is a Transformer based language model built primarily from decoder style blocks and trained to predict the next token in a sequence. It reads the tokens that have already appeared and generates new tokens one at a time. How does it work? Suppose the input is: The model uses the available earlier tokens to predict what should come next: After generating blue, that token becomes part of the context: The model then predicts the next token again. This process continues until generation stops. Why is it called decoder only? The original Transformer architecture contains separate encoder and decoder components. A decoder only LLM does not use a separate encoder stack. Instead, the input prompt and generated output are processed through one stack of decoder style Transformer layers. Conceptually: Modern decoder only LLMs typically use causal self attention and do not require the encoder decoder cross attention used in the original Transformer decoder. What is causal attention doing? Decoder only models generally use causal self attention . This prevents a token from looking at future tokens during training. For example, when predicting token 4: This matches how the model will generate text during inference: one token at a time using only the tokens already available. Why are decoder only models common for LLMs? They are well suited to open ended generation because the same next token prediction objective can be used for many tasks. With suitable prompting or post training, a decoder only LLM can perform tasks such as: Question answering Summarization Translation Code generation Conversation Text completion Is a decoder only LLM limited to text completion? No. Although its underlying mechanism is next token prediction, instructions, examples, conversation history, retrieved information, and other context can all be placed before the generated answer. The model then continues the sequence in a way that produces the requested response. Decoder only vs. other Transformer architectures Other common Transformer arrangements include: Encoder only models , which are commonly used to build contextual representations of input text. Encoder decoder models , which separately encode an input sequence and generate an output sequence. These are covered in their dedicated articles. In short: a decoder only LLM uses a single stack of causal Transformer layers to repeatedly predict the next token from the tokens already available, making it especially suitable for generative language tasks.
What is an encoder-only language model?
An encoder only language model is a Transformer based model that uses only the encoder part of the Transformer architecture to build contextual representations of input text. Instead of generating text one token at a time, an encoder only model is mainly designed to understand or represent the entire input sequence . How does it work? Suppose the input is: The model processes all tokens together and creates a contextual representation for each token. The representation of the word: is influenced by the surrounding words, helping the model understand that it refers to a financial institution in this sentence. Conceptually: What kind of attention does it use? Encoder only models usually use bidirectional self attention . This means a token can use information from tokens that appear both before and after it. For example, when representing one token: This differs from causal decoder only models, where a token is prevented from attending to future positions during autoregressive generation. What are encoder only models used for? They are well suited to tasks where the main goal is understanding the input rather than generating a long continuation. Common uses include: Text classification Sentiment analysis Named entity recognition Semantic similarity Search and retrieval embeddings Extractive question answering Document representation How are encoder only models commonly trained? A common training approach is masked language modeling . Some tokens are hidden, and the model learns to predict them using context from both sides. For example: The model may learn that a word such as: fits the surrounding context. Masked language modeling is covered separately in “What is masked language modeling during pretraining?” Are encoder only models good for open ended generation? They are generally not the standard choice for autoregressive text generation. Because they are designed to encode the full input rather than repeatedly predict the next token, decoder only or encoder decoder architectures are usually more suitable for generative tasks. Example BERT is a well known encoder only Transformer model. It produces contextual representations that can be adapted to many language understanding tasks. For the contrasting architecture, see “What is a decoder only LLM?” In short: an encoder only language model uses bidirectional Transformer encoder layers to build rich contextual representations of an entire input sequence, making it especially useful for understanding, classification, retrieval, and extraction tasks rather than open ended next token generation.
What is an encoder-decoder language model?
An encoder decoder language model is a Transformer based model that uses two separate components: an encoder to understand the input and a decoder to generate the output. It is especially useful when one sequence must be transformed into another. How does it work? Suppose the input is: The model works in two stages. First, the encoder processes the complete input and creates contextual representations of it. Then, the decoder uses those encoder representations while generating the output token by token: What does the encoder do? The encoder usually uses bidirectional self attention, allowing each input token to use information from tokens on both sides. Its job is to build useful representations of the input sequence. For example, in translation, it helps represent the meaning and relationships in the source sentence before generation begins. What does the decoder do? The decoder generates the output autoregressively. It typically uses: Causal self attention to look at previously generated output tokens Cross attention to use information produced by the encoder Conceptually: This lets the decoder generate a response while continuously referring back to the encoded input. What tasks are encoder decoder models useful for? They are well suited to sequence to sequence tasks such as: Translation Summarization Text rewriting Question answering Structured text generation Converting one text format into another The architecture is useful when there is a clear input sequence and a separate output sequence. How is this different from a decoder only LLM? A decoder only model processes the prompt and generated continuation through one causal Transformer stack. An encoder decoder model separates those roles: The decoder can directly attend to the encoder's representations through cross attention. Examples Well known encoder decoder Transformer families include T5 and BART . Their exact training methods differ, but both use the general encoder decoder structure. For the other major architecture types, see “What is an encoder only language model?” and “What is a decoder only LLM?” In short: an encoder decoder language model first encodes the input into contextual representations and then uses a separate decoder to generate the output token by token, making the architecture particularly suitable for sequence to sequence tasks.
What is the difference between encoder-only and decoder-only language models?
Encoder only and decoder only language models are both based on the Transformer architecture, but they are designed for different kinds of language tasks. The main difference is how they process context and what they are optimized to produce. Encoder only models Encoder only models process the input sequence as a whole. They typically use bidirectional self attention , which allows each token to use information from tokens that appear both before and after it. This makes them well suited to understanding and representing text. Common tasks include: Text classification Sentiment analysis Named entity recognition Semantic search Embedding generation Extractive question answering A well known example is BERT . Decoder only models Decoder only models are designed primarily for autoregressive generation. They use causal self attention , which prevents a token from seeing future tokens while predicting the next token. Conceptually: Decoder only models are commonly used for: Chat Text generation Code generation Summarization Question answering Instruction following How do their training objectives differ? Encoder only models are commonly associated with objectives such as masked language modeling , where selected tokens are hidden and predicted using surrounding context. Decoder only models are commonly trained with causal language modeling , where the model predicts the next token from the tokens that came before it. These training objectives influence the kinds of tasks each architecture handles naturally. How do their outputs differ? An encoder only model normally produces contextual representations of the input tokens. Those representations can then be used by another layer for tasks such as classification or retrieval. A decoder only model produces probabilities for the next token and repeatedly generates new tokens. So: Does this mean decoder only models cannot understand text? No. Decoder only LLMs can perform many language understanding tasks because the prompt, instructions, and relevant context are all provided before the answer is generated. The distinction is architectural and training related, not a statement that one architecture can only “understand” and the other can only “generate.” Which architecture is better? Neither is universally better. The appropriate architecture depends on the task. Encoder only models can be efficient for representation and classification tasks, while decoder only models are especially flexible for open ended generation and instruction following systems. For the individual architectures, see “What is an encoder only language model?” and “What is a decoder only LLM?” In short: encoder only models use bidirectional attention to build rich representations of the full input, while decoder only models use causal attention to predict and generate tokens sequentially.
Why do many modern LLMs use decoder-only architectures?
Many modern LLMs use decoder only architectures because they are a simple and scalable fit for autoregressive next token prediction, which is the core training objective behind many general purpose generative models. A decoder only model uses one Transformer stack to process the prompt and repeatedly predict the next token. 1. The architecture matches next token prediction directly During training, the model learns: Causal self attention prevents each position from seeing future tokens, so the training setup closely matches how the model generates text during inference. 2. One architecture can support many tasks A decoder only model does not need a separate architecture for each task. Different tasks can be expressed as text in the prompt: The model then generates the appropriate continuation. This makes decoder only models flexible for general purpose assistants and language generation systems. 3. The design is comparatively simple Encoder decoder models contain separate encoder and decoder stacks, with cross attention connecting them. Decoder only models use a single main stack. Conceptually: A simpler architecture can make large scale training and serving easier to design and optimize. 4. They scale well with data and compute Decoder only models can be pretrained on enormous collections of text using the same next token prediction objective across the dataset. The model can learn language, facts, code patterns, reasoning patterns, and task structures from the same basic objective. This consistency makes the approach practical for scaling to very large datasets and model sizes. 5. Prompting fits naturally Instructions, examples, retrieved information, conversation history, and the requested answer can all be represented as one token sequence. For example: The model simply continues the sequence. This works naturally with few shot prompting, chat formatting, and in context learning. Does decoder only mean it is always the best architecture? No. Encoder only models can be efficient for tasks such as classification and embeddings, while encoder decoder models remain useful for sequence to sequence tasks. Decoder only architectures are popular mainly because they combine simple training, flexible generation, and strong scalability in one general purpose model. For the architecture itself, see “What is a decoder only LLM?” In short: many modern LLMs use decoder only architectures because causal next token prediction is simple to train at scale, supports many tasks through prompting, and provides a flexible foundation for general purpose text and code generation.
How does a decoder-only LLM generate text?
A decoder only LLM generates text one token at a time by repeatedly predicting the most likely next token from the tokens already available in its context. It uses the same Transformer stack for both the original prompt and the tokens generated so far. 1. The prompt is converted into tokens Suppose the prompt is: The tokenizer converts this text into token IDs, which are then mapped to embeddings and passed through the model. 2. Causal self attention processes the sequence A decoder only LLM uses causal self attention . This means each position can use information from earlier tokens, but not from future tokens. Conceptually: This causal structure matches the way text is generated from left to right. 3. The model produces next token scores After processing the current sequence, the model produces a score for every token in its vocabulary. These scores are usually called logits . They are converted into probabilities representing how likely each token is to come next. For example: 4. A decoding strategy selects the next token The next token can be chosen using a decoding strategy such as: Greedy decoding Temperature based sampling Top k sampling Top p sampling The selected token is appended to the sequence. For example: 5. The process repeats Now the model processes: and predicts the next token again. Conceptually: This autoregressive loop continues until generation stops. 6. What makes generation stop? Generation can stop when: The model produces an end of sequence token A configured stop sequence is reached A maximum output token limit is reached Application specific stopping logic is triggered Does the model recompute everything for every token? Not usually. During inference, implementations commonly use a KV cache to reuse attention information from previously processed tokens. This makes generation much more efficient than recalculating the entire sequence from scratch for every new token. For the architecture itself, see “What is a decoder only LLM?” In short: a decoder only LLM generates text by processing the prompt with causal self attention, predicting probabilities for the next token, selecting one token, appending it to the context, and repeating the process until a stopping condition is reached.
What is self-attention in an LLM?
Self attention is a mechanism that allows an LLM to compare each token with other relevant tokens in the same sequence and decide which ones should influence its representation. It is one of the core components of the Transformer architecture. Why is it called self attention? It is called self attention because the queries, keys, and values all come from the same sequence. For example, in: the representation of: can use information from earlier tokens such as: to build a more context aware representation. What happens inside self attention? Each token representation is transformed into three learned representations: Query Key Value The model compares a token's query with the keys of other allowed tokens. These comparisons produce attention scores that determine how strongly different token representations should contribute. Conceptually: Does every token receive equal attention? No. The attention weights depend on the current token and the surrounding sequence. Some tokens may receive high weight because they are especially relevant, while others receive much less. This allows the model to build different contextual representations for the same word in different sentences. Is self attention the same in every LLM? No. Different models can use different forms of attention, such as: Causal self attention Multi head attention Grouped query attention Multi query attention Sliding window attention Sparse attention These variants change which positions can interact or how attention is computed efficiently. What about causal LLMs? Decoder only LLMs usually use causal self attention . In that setup, a token can attend only to permitted earlier positions, not future tokens. This preserves next token prediction during autoregressive generation. Causal self attention is covered separately in “What is causal self attention?” Is self attention the whole Transformer? No. Self attention is one major component. Transformer layers also include feed forward networks, residual connections, normalization, and positional information. For the broader architecture, see “What architecture is commonly used in Large Language Models?” In short: self attention lets an LLM dynamically combine information from other relevant tokens in the same sequence, producing context aware token representations that the Transformer can use in later computations.
How does self-attention help an LLM understand context?
Self attention helps an LLM understand context by allowing each token to use information from other relevant tokens in the same sequence when building its representation. This means the meaning of a token is not determined only by the token itself—it is influenced by the surrounding text. Example Consider the sentence: The word: could mean a financial institution or the side of a river. Self attention allows the representation of bank to use nearby words such as: which makes the financial meaning more likely. How does self attention choose relevant context? Each token is transformed into: A query A key A value The token's query is compared with the keys of other allowed tokens. These comparisons produce attention scores. Higher scores mean that certain tokens should contribute more strongly to the current token's representation. Conceptually: How does this help with relationships across a sentence? Self attention can connect words that are not directly next to each other. For example: When processing: the model can give more attention to: than to unrelated nearby words. This helps the model capture relationships across longer distances. Does self attention use the same context for every token? No. Each token can assign different attention weights to the rest of the sequence. So the representation of one token may focus on grammar, while another may focus on a referenced object, instruction, name, or earlier fact. This makes token representations dynamic and context dependent. What about decoder only LLMs? Decoder only LLMs use causal self attention . A token can use earlier tokens in the sequence, but not future tokens that have not yet been generated. For example: the model can use all available earlier tokens when predicting the next token. Does attention mean the model understands like a human? No. Self attention is a mathematical mechanism for combining token information. It gives the model rich contextual representations, but it should not be interpreted as human like understanding or awareness. For the core mechanism, see “What is self attention in an LLM?” In short: self attention helps an LLM use surrounding and distant tokens to build context dependent representations, allowing words and phrases to be interpreted according to the information that is most relevant to them.
What is causal self-attention?
Causal self attention is a form of self attention that prevents each token from using information from future tokens in the sequence. It is commonly used in decoder only LLMs so the model learns to predict text from left to right. Why is it called causal? When predicting a token, the model should only use information that would already be available at that point. For example, in: when processing the token: the model can use: but it must not use: because those tokens come later. How is future information blocked? A causal attention mask is applied to the attention scores. Conceptually, the allowed attention pattern looks like: Each position can attend to itself and earlier positions, but not later ones. Why is this important during training? During training, the full sequence is already available. Without a causal mask, the model could look ahead at the token it is supposed to predict, which would make next token prediction meaningless. Causal self attention ensures that training follows the same information constraint used during generation: How does it work during generation? During inference, future tokens do not exist yet. The model receives the prompt and previously generated tokens, predicts one new token, appends it to the sequence, and repeats. Causal self attention ensures each prediction depends only on the information generated or supplied so far. Is causal self attention the same as ordinary self attention? It is a restricted form of self attention. Standard bidirectional self attention can allow a token to use information from both earlier and later positions. Causal self attention blocks later positions to preserve autoregressive generation. For the broader mechanism, see “What is self attention in an LLM?” In short: causal self attention lets each token attend only to itself and earlier tokens, preventing access to future information and making autoregressive next token prediction possible.