Large Language Models
Large language models are deep learning models, usually transformer-based, trained to predict tokens from large corpora. After pretraining, they are often instruction-tuned and aligned so they can follow prompts and assist users.
Why can the same word use multiple tokens?
The same word can use multiple tokens because an LLM tokenizer does not necessarily treat every complete word as one vocabulary entry. Instead, many tokenizers use subword pieces , so a word may be split into several known tokens. For example, a tokenizer might represent: as: or as several even smaller pieces. The exact split depends on the tokenizer vocabulary. Why does this happen? A tokenizer has a fixed vocabulary. If the complete word is not stored as one token, the tokenizer represents it using smaller pieces that are available. Common words may therefore use one token, while rare or complex words may require several. For example: The exact result varies between tokenizers. Can the same written word tokenize differently? Yes. Tokenization can also depend on how the word appears in the text. Factors such as these may affect the token sequence: Leading whitespace Capitalization Punctuation Unicode representation The tokenizer being used For example, some tokenizers may have different vocabulary entries for a word at the beginning of text and the same word preceded by a space. Why not make every word one token? Creating one token for every possible word would require an enormous vocabulary and would still struggle with: New words Names Technical terminology Misspellings Word variations Multiple languages Using reusable subword pieces keeps the vocabulary manageable while allowing almost any text to be represented. Does using more tokens change the meaning? Not by itself. The model processes the sequence of tokens together and builds contextual representations from them. A word represented by three tokens can still be understood as part of the surrounding text, although using more tokens consumes more of the model's context window and requires more computation. For the underlying mechanism, see “What is subword tokenization?” In short: the same word can require multiple tokens when it is not available as one vocabulary entry. The tokenizer then represents it using smaller known pieces, and the exact tokenization can also vary with formatting and tokenizer design.
Why do different LLMs tokenize the same text differently?
Different LLMs can tokenize the same text differently because they may use different tokenizer algorithms, vocabularies, training data, and text processing rules . A tokenizer is created separately from the LLM's main training process, so each model family can have its own way of splitting text into tokens. 1. Different tokenizers use different vocabularies One tokenizer may contain a complete word as a single token, while another may only contain smaller pieces. For example, the word: might be represented by one tokenizer as: and by another as: These are only illustrative splits—the actual result depends on the tokenizer. 2. Different algorithms learn different token boundaries LLMs may use tokenizer methods such as: Byte Pair Encoding (BPE) WordPiece Unigram tokenization Byte level tokenization Each method learns or applies token boundaries differently. 3. Tokenizer training data matters A tokenizer trained on mostly English text may build a vocabulary that represents English words efficiently. A tokenizer trained on multilingual text, code, or domain specific data may allocate more vocabulary entries to those patterns instead. This can change how many tokens the same sentence requires. 4. Text normalization and whitespace rules can differ Tokenizers may handle: Spaces Capitalization Unicode characters Punctuation Line breaks Special symbols in different ways. Some tokenizers even have vocabulary entries that include leading whitespace as part of the token. 5. Vocabulary size also affects tokenization A larger vocabulary can store more common text pieces directly, while a smaller vocabulary may need to split the same text into more tokens. However, vocabulary size alone does not determine tokenizer quality. Why does this matter? Different tokenization can affect: Token count Context window usage API cost when pricing is token based Processing efficiency How efficiently different languages or code are represented So the same sentence may consume different numbers of tokens in different LLMs even though the visible text is identical. For related concepts, see “What is a tokenizer vocabulary?” and “What is vocabulary size in an LLM?” In short: different LLMs tokenize the same text differently because their tokenizers may use different algorithms, vocabularies, training corpora, and preprocessing rules.
How does tokenization affect LLM performance?
Tokenization affects LLM performance because it determines how efficiently text is converted into the token sequences the model actually processes. A good tokenizer represents common text compactly while still handling rare words, multiple languages, code, symbols, and unusual text reliably. 1. It affects sequence length If a tokenizer represents text with fewer tokens, the model has fewer positions to process. For example, one tokenizer might encode a sentence in: while another might require: Longer token sequences generally require more computation and consume more of the available context window. 2. It affects context window efficiency An LLM's context window is measured in tokens, not words. If a tokenizer splits text inefficiently, the same document can consume more of the context window. More efficient tokenization can therefore allow the model to fit more useful text into the same token limit. 3. It affects different languages differently A tokenizer trained mostly on one language may represent that language efficiently but split text from another language into many more tokens. This can affect: Context usage Processing cost Training efficiency Inference efficiency Multilingual tokenizer design therefore matters when an LLM is expected to work across many languages. 4. It affects rare words, code, and technical text A tokenizer must also represent names, source code, numbers, technical terminology, and uncommon words. If these are broken into too many small pieces, sequences become longer. If useful recurring patterns are represented as compact tokens, the model can process them more efficiently. 5. It influences what the model learns during training The model learns relationships between tokens. Tokenization determines the basic units presented during training, so the chosen vocabulary and token boundaries influence how patterns are represented and learned. However, tokenizer quality is only one factor. Model architecture, training data, training objective, parameter count, and optimization also strongly affect overall LLM performance. Does fewer tokens always mean better performance? No. A tokenizer should not simply make tokens as large as possible. A very large vocabulary increases the size of embedding and output layers and may reduce the usefulness of reusable subword pieces. Tokenizer design is therefore a tradeoff between: For the basic process, see “What is tokenization in LLMs?” In short: tokenization affects LLM performance by changing sequence length, context window usage, computational efficiency, multilingual coverage, and the basic text units from which the model learns.
How does tokenization affect LLM cost?
Tokenization affects LLM cost because LLMs process tokens , not raw words or characters. In many hosted LLM services, usage is measured using the number of input and output tokens. Even when pricing is not directly token based, longer token sequences generally require more computation and memory. 1. More tokens can mean higher API cost Suppose two tokenizers encode the same paragraph differently: If an API charges for input tokens, the second representation can cost more even though the visible text is identical. The same idea applies to generated output: a response that requires more tokens can increase usage cost. 2. Tokenization affects context usage An LLM context window is measured in tokens. If text is tokenized inefficiently, it consumes more of the available context window. That can reduce how much useful information fits into a single request and may require: Shorter prompts More aggressive truncation Summarization Splitting documents into multiple requests Those extra requests can increase total cost. 3. Different languages can have different token costs The same amount of human readable text may require different numbers of tokens depending on the language and tokenizer. A tokenizer whose vocabulary represents one language efficiently may need many more tokens for another language. As a result, tokenization can affect the cost of multilingual applications. 4. Code and structured data can also tokenize differently Source code, JSON, numbers, URLs, and unusual symbols may be split into many tokens depending on the tokenizer. For applications that process large amounts of structured text, tokenizer efficiency can therefore have a noticeable effect on total usage. 5. More tokens also increase computational work Longer sequences require the model to process more token positions. This can increase: Inference time Memory usage Compute requirements Serving cost For self hosted models, these infrastructure costs matter even when there is no per token API bill. Can tokenization alone determine LLM cost? No. Overall cost also depends on factors such as: Model size Input length Output length Provider pricing Hardware Batching and caching Inference optimizations Tokenization mainly affects how much text becomes how many tokens , which then influences these costs. For the performance side of this topic, see “How does tokenization affect LLM performance?” In short: tokenization affects LLM cost because different tokenizers can represent the same text using different numbers of tokens. More tokens generally consume more context, computation, memory, and—when pricing is token based—money.
What are special tokens in an LLM?
Special tokens are reserved tokens that carry structural or control information rather than ordinary text content. They help an LLM or its tokenizer mark boundaries, distinguish roles, represent padding, indicate the end of generation, or encode other special instructions. Common types of special tokens Depending on the model, special tokens can include: Beginning of sequence token — marks the start of a sequence. End of sequence token — marks the end of a sequence and can tell generation to stop. Padding token — fills unused positions when sequences of different lengths are batched together. Unknown token — represents text that a tokenizer cannot otherwise encode in some tokenizer designs. Separator token — separates different text segments. Mask token — hides a token during masked language model training. Chat or role tokens — distinguish system, user, assistant, or other message roles in chat models. Not every LLM uses all of these. Are special tokens normal words? No. A special token may have a readable representation such as: but internally it is usually represented by a reserved token ID in the tokenizer vocabulary. For example: The exact token names and IDs differ between model families. Why are special tokens needed? Normal text alone does not always provide enough structure for model training and inference. Special tokens can tell the model things such as: Where a sequence begins Where it ends Which tokens are padding Where one segment ends and another begins Which speaker produced a message When generation should stop This makes them useful for both training and serving LLMs. How are they used in chat models? Chat models often use reserved tokens or equivalent formatting to represent conversation structure. Conceptually, a conversation may be encoded like: The actual format is model specific and may use hidden or documented control tokens rather than these exact strings. This structure helps the model distinguish different message roles. Can users type special token text directly? Typing text that looks like a special token does not always make it behave as one. Tokenizer implementations may reserve these tokens, escape them, reject them, or require special handling in the API. The model's official tokenizer and chat template determine how special tokens are inserted and interpreted. For individual token types, see the dedicated articles on beginning of sequence , end of sequence , and padding tokens. In short: special tokens are reserved tokens used to represent structure and control information—such as sequence boundaries, padding, masks, separators, or chat roles—so the model can interpret text in the correct format.
What is a beginning-of-sequence token?
A beginning of sequence token , often called a BOS token , is a special token used to mark the start of a token sequence. It gives the model an explicit signal that a new sequence is beginning. What does a BOS token look like? The exact representation depends on the tokenizer and model. Conceptually, it may look like: A tokenized sequence might therefore look like: Internally, the BOS token is represented by a reserved token ID rather than ordinary text. Why is a BOS token useful? A BOS token can help the model distinguish the start of a new sequence from ordinary text. During training, this gives the model a consistent starting point for predicting subsequent tokens. Conceptually: For an autoregressive language model, the BOS token can provide the initial context from which the first normal token is predicted. Is a BOS token always required? No. Not every LLM or tokenizer uses an explicit BOS token. Some models automatically add one, some expect the application to add it, and others do not use one at all. The correct behavior depends on the model's tokenizer configuration and prompt format. Is BOS visible to the user? Usually not. When you type a prompt into an LLM application, the tokenizer or serving system may insert special tokens automatically. The user normally interacts with ordinary text while the underlying model receives a formatted token sequence. BOS in chat models Chat models may use a more complex prompt format containing special tokens for: Message boundaries User and assistant roles System instructions Conversation starts In such systems, a traditional BOS token may be used alongside other control tokens, or the chat template may handle sequence initialization differently. BOS vs. EOS The two tokens serve opposite boundaries: BOS marks the beginning of a sequence. EOS marks the end of a sequence. The EOS token is covered separately in “What is an end of sequence token?” For the broader category, see “What are special tokens in an LLM?” In short: a beginning of sequence token is a reserved special token that marks the start of a token sequence and can provide the model with a consistent initial context before ordinary text begins.
What is an end-of-sequence token?
An end of sequence token , often called an EOS token , is a special token used to mark that a sequence has ended. For generative LLMs, producing an EOS token can also signal that the model should stop generating more text. What does an EOS token look like? The exact representation depends on the model and tokenizer. Conceptually, it may look like: A sequence might therefore be represented as: Internally, the EOS token is represented by a reserved token ID. How does an LLM use it during generation? During autoregressive generation, the model predicts one token at a time. At each step, the EOS token is one of the possible next tokens. Conceptually: If the EOS token is selected, the generation system can stop producing additional tokens. Is EOS the only way generation can stop? No. Generation can also stop because of: A maximum output token limit A configured stop sequence Application specific stopping logic Other model specific control tokens So EOS is an important stopping mechanism, but it is not the only one. Is an EOS token always visible? Usually not. Applications normally remove special control tokens before displaying the final response to the user. You typically see only the generated text, not the internal EOS marker. How is EOS used during training? Training examples may include an EOS token at the end of a sequence. This allows the model to learn that after a complete passage, conversation turn, or document, predicting the end of sequence token can be appropriate. EOS vs. BOS The two tokens represent opposite sequence boundaries: BOS marks the beginning of a sequence. EOS marks the end of a sequence. For the starting boundary, see “What is a beginning of sequence token?” In short: an end of sequence token is a reserved special token that marks the end of a sequence and can tell a generative LLM when it is appropriate to stop generating.
What is a padding token?
A padding token is a special token used to make sequences in the same batch have equal length. LLMs often process multiple sequences together for efficiency, but those sequences may contain different numbers of tokens. Padding fills the unused positions in shorter sequences. Why is padding needed? Suppose a batch contains two token sequences: To process them together as a rectangular tensor, the shorter sequence can be padded: Conceptually: Does the model treat padding as normal text? Usually no. An attention mask is commonly used to tell the model which positions contain real tokens and which positions are padding. Conceptually: This prevents padding positions from influencing the meaningful parts of the sequence. What does a padding token look like? Its representation depends on the tokenizer. It may appear conceptually as: Internally, it has a reserved token ID. Some model families have a dedicated padding token, while others may reuse another special token for padding depending on their tokenizer and serving setup. Left padding vs. right padding Padding can be added: After the real tokens — right padding Before the real tokens — left padding For example: Which approach is appropriate depends on the model architecture and inference implementation. Is padding always necessary? No. A single sequence does not usually need padding. Some modern serving systems also use techniques that reduce or avoid unnecessary padding when batching variable length requests. Padding is mainly useful when sequences must be arranged into equal sized batches. For the broader category, see “What are special tokens in an LLM?” In short: a padding token is a reserved token used to fill unused positions so sequences of different lengths can be processed together efficiently, while an attention mask usually prevents those padding positions from affecting the model's actual computation.
What is a token ID?
A token ID is the numeric identifier assigned to a token in an LLM tokenizer's vocabulary. LLMs do not process raw text directly. After tokenization, each token is converted into an integer ID that the model can use. Simple example Suppose a tokenizer vocabulary contains: The text: might therefore become a sequence such as: The actual IDs depend entirely on the tokenizer. Why are token IDs needed? Neural networks work with numerical data rather than text strings. Token IDs provide a compact way to identify vocabulary entries. Conceptually: The token ID itself is not the token's meaning. It is mainly an index used to look up the token's learned embedding. Are token IDs universal? No. The same token can have different IDs in different tokenizers. For example: might have one ID in one model's tokenizer and a completely different ID in another. This is because each tokenizer has its own vocabulary and ID mapping. Can different tokens have the same token ID? Within a single tokenizer vocabulary, each vocabulary entry normally has its own unique ID. However, the same numeric ID can refer to completely different tokens in different tokenizer vocabularies. Do special tokens also have token IDs? Yes. Special tokens such as beginning of sequence, end of sequence, or padding tokens are also usually assigned reserved IDs. For example: The exact values are model specific. Is a token ID the same as an embedding? No. A token ID is an integer that identifies a vocabulary entry. An embedding is a learned numerical vector associated with that token ID and used by the model for computation. For the next step in this process, see “How is text converted into token IDs?” In short: a token ID is the integer assigned to a token in a tokenizer's vocabulary. It lets the model convert tokenized text into numerical indices that can be mapped to learned embeddings.
How is text converted into token IDs?
Text is converted into token IDs by an LLM's tokenizer . The tokenizer first splits the input text into tokens and then looks up each token in its vocabulary to find the corresponding numeric ID. 1. Start with raw text Suppose the input is: The tokenizer receives this text before it reaches the language model. 2. Split the text into tokens Depending on the tokenizer, the text may be divided into words, subwords, characters, bytes, or a mixture of these. Conceptually, it might become: The exact split depends on the tokenizer vocabulary and tokenization algorithm. 3. Match each token to the vocabulary Every token in the tokenizer vocabulary has a numeric ID. For example: These numbers are only illustrative. 4. Produce the token ID sequence The original text is now represented as: This sequence of integers is what is passed to the model. 5. Convert token IDs into embeddings The model does not perform its main computations directly on the integer IDs. Instead, each token ID is used to look up a learned embedding vector. Conceptually: The embedding vectors provide the numerical representations that the model processes. What about special tokens? The tokenizer or application may also add special tokens such as: Beginning of sequence tokens End of sequence tokens Padding tokens Chat role or separator tokens These also have their own reserved token IDs. Can the same text produce different token IDs in different LLMs? Yes. Different LLMs may use different tokenizers and vocabularies, so the same text can be split differently and mapped to completely different token IDs. For the meaning of those numerical identifiers, see “What is a token ID?” In short: text is converted into token IDs by splitting it into tokenizer defined tokens and replacing each token with the integer assigned to it in the tokenizer vocabulary. Those IDs are then mapped to embeddings before the LLM processes them.