Large Language Models

Large language models are deep learning models, usually transformer-based, trained to predict tokens from large corpora. After pretraining, they are often instruction-tuned and aligned so they can follow prompts and assist users.

What is vocabulary size in an LLM?

Vocabulary size in an LLM is the total number of distinct tokens that the model's tokenizer can represent. Each token in the vocabulary has its own numeric token ID . For example, if a tokenizer has a vocabulary size of: then its token IDs may range roughly from: depending on how the tokenizer is designed. What is included in the vocabulary? A tokenizer vocabulary can contain: Whole words Subword pieces Characters or byte level pieces Punctuation Whitespace patterns Numbers or number fragments Special tokens For example, a tokenizer might contain separate entries for: Why does vocabulary size matter? Vocabulary size affects how text is represented. A larger vocabulary can represent more common words or text patterns as single tokens, which may shorten token sequences. A smaller vocabulary uses fewer token entries but may split text into more pieces, producing longer sequences. This creates a tradeoff: Does a larger vocabulary always make an LLM better? No. A larger vocabulary can improve tokenization efficiency for some text, but it also increases the number of parameters needed in parts of the model such as the embedding and output layers. Tokenizer quality, training data, language coverage, and tokenization strategy are more important than vocabulary size alone. Is vocabulary size the same as parameter count? No. Vocabulary size tells us how many different tokens the tokenizer knows. Parameter count tells us how many learned numerical values exist inside the model. They are related because vocabulary size influences the size of embedding and output matrices, but they measure different things. For the underlying concept, see “What is a tokenizer vocabulary?” In short: vocabulary size is the number of distinct tokens available to an LLM tokenizer. It influences how efficiently text is represented and how large some model components need to be.

What is subword tokenization?

Subword tokenization is a tokenization method that represents text using pieces that can be smaller than complete words but larger than individual characters. The goal is to keep common words or word pieces compact while still being able to represent rare or previously unseen words. For example, a tokenizer might split: into pieces such as: The exact split depends on the tokenizer and its vocabulary. Why use subwords? A word level tokenizer would need a very large vocabulary to represent every possible word. A character level tokenizer avoids that problem, but it produces much longer sequences. Subword tokenization provides a compromise: How does it handle uncommon words? Suppose a tokenizer does not contain the complete word: It may still be able to represent it using known pieces such as: This allows the model to process new names, technical terms, word variations, and other uncommon text without requiring every possible word to exist as a separate vocabulary entry. How is the subword vocabulary created? A tokenizer is usually built from a large text corpus. The tokenizer analyzes recurring text patterns and chooses useful pieces for its vocabulary. Different tokenization algorithms use different rules for doing this. Common approaches include: Byte Pair Encoding (BPE) WordPiece Unigram tokenization These methods are covered separately in their dedicated articles. Does every word get split? No. A frequent word may already exist as a single token, while a less common word may require several subword tokens. For example: The exact result varies between tokenizers. Why does this matter for LLMs? Subword tokenization helps keep the vocabulary manageable while supporting a large variety of text. It also affects: Token counts Context window usage Inference cost Multilingual efficiency How words and code fragments are represented For the broader process, see “What is tokenization in LLMs?” In short: subword tokenization breaks text into reusable pieces that are often smaller than words, allowing an LLM to represent both common and rare text without requiring an enormous word level vocabulary.

Why is subword tokenization used in LLMs?

Subword tokenization is used in LLMs because it provides a practical balance between word level and character level tokenization. Instead of requiring every possible word to exist in the vocabulary, the tokenizer can represent unfamiliar or rare words using smaller reusable pieces. For example: might be represented as: The exact split depends on the tokenizer. 1. It keeps the vocabulary manageable A word level tokenizer would need a huge vocabulary to cover: Common words Rare words Names Technical terms Spelling variations Newly created words Subword tokenization avoids this by reusing smaller pieces across many words. 2. It handles rare and unseen words If a complete word is not in the vocabulary, the tokenizer can often split it into known subwords. For example: This greatly reduces the problem of unknown words. 3. It avoids very long character sequences Character level tokenization can represent almost any text, but it often creates long sequences. For example: contains eight characters, while a subword tokenizer might represent it with one or only a few tokens. Shorter token sequences reduce the amount of sequence processing the model must perform. 4. It captures useful word structure Subword pieces can also capture recurring linguistic patterns such as: These reusable pieces can help the model learn relationships between related words. 5. It supports many kinds of text Subword tokenization works well for: Multiple languages Code Numbers Compound words Technical terminology Names and uncommon expressions The exact efficiency varies by tokenizer and language. Why not just use whole words? Whole word tokenization is simple, but it struggles with vocabulary growth and unknown words. Subword tokenization instead allows: For the basic concept, see “What is subword tokenization?” In short: LLMs use subword tokenization because it keeps vocabulary size practical, handles rare or unseen text, and avoids the very long sequences produced by purely character level tokenization.

What is Byte Pair Encoding (BPE)?

Byte Pair Encoding (BPE) is a tokenization method that builds a vocabulary by repeatedly merging frequently occurring adjacent text units into larger tokens. In LLM tokenizers, BPE helps create a vocabulary that contains common words or word pieces while still allowing uncommon text to be represented using smaller pieces. How does BPE build a vocabulary? A simplified BPE training process starts with small units, such as characters or bytes. Suppose the training text contains many examples of: The tokenizer counts adjacent pairs. If: appears very frequently, those two units may be merged into: If: is also common, another merge may create: This process continues until the tokenizer reaches its chosen vocabulary size or another stopping condition. What happens after the merges are learned? The learned merge rules become part of the tokenizer. When new text is tokenized, the tokenizer applies those learned rules to split the text into known vocabulary pieces. For example, a common word may become one token, while a rare word may be represented using several smaller tokens. Why is BPE useful for LLMs? BPE provides a useful balance between: Word level tokenization , which can require a huge vocabulary Character level tokenization , which can create very long token sequences It allows common patterns to become compact tokens while keeping smaller units available for unusual or unseen text. Is BPE always character based? Not necessarily. Some modern tokenizers use byte level BPE , where the initial units are bytes rather than ordinary characters. This helps ensure that almost any input text can be represented without requiring a traditional unknown word token. Byte level tokenization is covered separately in “What is byte level tokenization?” BPE vs. subword tokenization BPE is one specific method for creating a subword vocabulary. Other approaches include WordPiece and Unigram tokenization. For the broader concept, see “What is subword tokenization?” In short: BPE creates a tokenizer vocabulary by repeatedly merging frequent neighboring units into larger tokens, giving LLMs an efficient way to represent both common and uncommon text.

How does Byte Pair Encoding work?

Byte Pair Encoding (BPE) works by learning which adjacent text units frequently occur together and repeatedly merging those pairs into larger tokens. The result is a vocabulary that contains a mixture of small pieces and frequently occurring larger pieces. 1. Start with small units A simplified BPE process begins with small units such as characters. Suppose the training data contains: Initially, the words can be represented as: 2. Count adjacent pairs BPE counts how often neighboring units appear together. For example, pairs such as: may occur multiple times. 3. Merge the most frequent pair If: is the most frequent pair, BPE merges it into a new unit: The sequences now become something like: 4. Repeat the process BPE counts adjacent pairs again. If: is now frequent, it may merge them into: The process continues, gradually creating larger and more useful tokens. Conceptually: 5. Use the learned merges on new text After training, the tokenizer keeps the learned vocabulary and merge rules. When new text arrives, it applies those rules to break the text into known token pieces. Common patterns may become single tokens, while uncommon words may remain split into several smaller pieces. Why does this help? BPE allows an LLM tokenizer to: Represent common text compactly Break rare words into reusable pieces Keep vocabulary size manageable Avoid relying entirely on whole word tokens Reduce the unknown word problem Modern implementations may use variations such as byte level BPE, but the central idea remains the same: frequent neighboring units are progressively merged into larger tokens . For the basic definition, see “What is Byte Pair Encoding (BPE)?” In short: BPE learns token pieces by repeatedly finding frequent adjacent units, merging them, and saving those merges so future text can be segmented using the learned vocabulary.

What is WordPiece tokenization?

WordPiece tokenization is a subword tokenization method that represents text using a vocabulary of whole words and smaller word pieces. If a complete word is not available in the vocabulary, WordPiece breaks it into smaller known pieces. For example, a tokenizer might represent: as something like: The exact pieces depend on the tokenizer vocabulary. In some WordPiece implementations, a prefix such as indicates that a token continues a word rather than starting a new one. How is the vocabulary built? WordPiece starts from small units and gradually builds a vocabulary of useful subword pieces from training text. Instead of simply merging the most frequent adjacent pair, as standard BPE does, WordPiece chooses candidate merges using a scoring strategy that favors combinations considered useful relative to the frequencies of their individual parts. The goal is to create a vocabulary that can represent common text compactly while still handling rare words through smaller pieces. How is new text tokenized? When processing a word, WordPiece typically looks for the longest vocabulary token that matches the beginning of the remaining text. For example, if the vocabulary contains: then: can be represented as: The tokenizer continues until the entire word has been represented. Why is WordPiece useful? WordPiece helps: Keep vocabulary size manageable Represent rare words using known subwords Reuse common word pieces across many words Reduce dependence on complete word level tokens Keep sequences shorter than purely character level tokenization Where has WordPiece been used? WordPiece is well known for its use in models such as BERT and related Transformer models. WordPiece vs. BPE Both are subword tokenization methods. The main difference is how the vocabulary is learned: BPE repeatedly merges frequent adjacent units WordPiece uses a scoring criterion to choose useful merges Their exact implementations can vary. For the broader idea, see “What is subword tokenization?” In short: WordPiece is a subword tokenization method that builds a reusable vocabulary of word pieces and represents unfamiliar words by combining the longest matching pieces from that vocabulary.

What is SentencePiece tokenization?

SentencePiece is a tokenizer framework that learns a fixed vocabulary of text pieces directly from raw text. Unlike tokenizers that first split text on spaces or apply language specific preprocessing, SentencePiece can treat the input as a continuous sequence of Unicode characters and learn useful token boundaries from the data itself. How does SentencePiece represent text? SentencePiece can learn subword vocabularies using algorithms such as: Byte Pair Encoding (BPE) Unigram language modeling The tokenizer then breaks new text into pieces from that learned vocabulary. For example, a word such as: might be represented as several subword tokens, depending on the learned vocabulary. How are spaces handled? SentencePiece treats whitespace as part of the text rather than discarding it during preprocessing. It commonly represents a space using a visible marker such as: For example: might be tokenized into pieces that conceptually look like: This allows tokenization and detokenization to preserve word boundaries consistently. Why is SentencePiece useful? SentencePiece is useful because it: Does not require language specific word segmentation Works directly from raw text Supports subword vocabularies Can represent languages that do not use spaces between words Provides deterministic encoding and decoding Can be used with different vocabulary learning algorithms This makes it especially practical for multilingual language models. Is SentencePiece itself one tokenization algorithm? Not exactly. SentencePiece is a tokenization framework . It can train and use different subword algorithms, including BPE and Unigram. So: SentencePiece vs. WordPiece Both can produce subword tokens, but SentencePiece is designed to work directly on raw text and does not require whitespace based pre tokenization. For the broader idea, see “What is subword tokenization?” In short: SentencePiece is a language independent tokenization framework that learns subword vocabularies directly from raw text and can use algorithms such as BPE or Unigram to split text into tokens.

What is byte-level tokenization?

Byte level tokenization represents text using the bytes that make up its encoded form, rather than relying only on complete words or characters. Text is typically encoded as bytes, such as UTF 8. Because a byte can take only 256 possible values, a tokenizer can start from a small base set that is capable of representing essentially any text. How does it work? Suppose the input contains ordinary text, symbols, emojis, or characters from different languages. The tokenizer first considers the underlying byte sequence. Those bytes can then be used directly as tokens or combined into larger learned token pieces. Conceptually: Why is byte level tokenization useful? A major advantage is that it can represent almost any input without needing a traditional unknown word token. This includes: Rare words Multiple languages Emojis Unusual symbols Misspellings Source code Previously unseen text Even if the tokenizer has never seen a complete word before, its bytes can still be represented. Does one byte always equal one token? No. In some tokenizers, bytes are only the starting units. Methods such as byte level BPE can merge frequently occurring byte sequences into larger tokens. For example, a common word may eventually be represented by one token, while an unusual sequence may require several smaller byte based tokens. Byte level vs. character level tokenization A character and a byte are not always the same thing. In UTF 8: Many common ASCII characters use one byte Other Unicode characters may require multiple bytes So one visible character can sometimes correspond to several bytes. What is the tradeoff? Byte level tokenization provides excellent coverage, but uncommon characters or languages can sometimes require more tokens if their byte sequences are not represented efficiently by the learned vocabulary. Tokenizer design therefore affects how efficiently different kinds of text are encoded. For a related method, see “What is Byte Pair Encoding (BPE)?” In short: byte level tokenization starts from the bytes that encode text, giving the tokenizer a reliable way to represent virtually any input while still allowing frequently occurring byte sequences to be combined into larger tokens.

What is the difference between word-level and subword tokenization?

Word level tokenization splits text into complete words, while subword tokenization can split words into smaller reusable pieces. For example: The exact subword split depends on the tokenizer. Word level tokenization In word level tokenization, each complete word is treated as one token. This is easy to understand, but it creates problems because language contains an enormous number of possible words, including: Names Technical terms Misspellings Word variations Newly created words If a word is not in the vocabulary, the tokenizer may need an unknown word token. Subword tokenization Subword tokenization keeps common words or word pieces as tokens and breaks rare words into smaller known pieces. For example: This allows the tokenizer to represent unfamiliar words without needing every possible word in its vocabulary. Why are subwords usually preferred for LLMs? Subword tokenization provides a useful balance: It keeps the vocabulary much smaller than a pure word level vocabulary. It handles rare and unseen words more effectively. It can reuse meaningful pieces across related words. It usually produces shorter sequences than character level tokenization. Word level tokenization may create shorter sequences for known words, but its vocabulary can become extremely large and inflexible. Simple comparison Word level: one token per known word, but poor handling of unseen words. Subword: words may use one or several tokens, but rare words can still be represented from known pieces. Modern LLM tokenizers therefore commonly use subword based approaches such as BPE, WordPiece, or Unigram rather than relying only on complete words. For the broader concept, see “What is subword tokenization?” In short: word level tokenization treats complete words as tokens, while subword tokenization breaks uncommon words into reusable pieces, making it more flexible and practical for large language models.

How are unknown words handled by LLM tokenizers?

LLM tokenizers usually handle unknown words by breaking them into smaller pieces that already exist in the tokenizer vocabulary. This is one of the main reasons modern LLMs use subword or byte level tokenization instead of relying only on complete words. 1. Break the word into known subwords Suppose the tokenizer has never seen the complete word: It may split it into known pieces such as: or into even smaller pieces, depending on the vocabulary. This allows the model to process words that were not stored as single vocabulary entries. 2. Use smaller units when necessary If a word cannot be represented with larger subwords, the tokenizer may continue breaking it into smaller units. Depending on the tokenizer, these units can be: Smaller subwords Characters Bytes Byte level tokenizers are especially robust because any UTF 8 text can ultimately be represented using bytes. 3. Some tokenizers can use an unknown token Older or more restrictive tokenizers may use a special token such as: when they cannot represent a word using their vocabulary. However, many modern LLM tokenizers are designed to avoid this situation by using subword or byte level representations. Example Imagine the tokenizer has never seen: It might represent it as: or: The exact split depends on the tokenizer. Does the model understand the new word automatically? Not necessarily. Tokenization only ensures that the text can be represented. The model still has to infer the meaning of the unfamiliar word from: The individual token pieces The surrounding context Similar patterns learned during training For example: Even if the complete word is unfamiliar, the surrounding context can help the model infer how it is being used. For the underlying mechanism, see “What is subword tokenization?” and “What is byte level tokenization?” In short: unknown words are usually handled by splitting them into smaller known subwords, characters, or bytes. This lets modern LLM tokenizers represent unfamiliar text without requiring every possible word to exist in the vocabulary.