Why do LLMs use tokens instead of words?

LLMs use tokens instead of whole words because tokens provide a more flexible and efficient way to represent language.

If a model used only complete words, it would need an enormous vocabulary containing every possible word, spelling variation, number, name, technical term, and newly invented expression.

Tokens solve this by allowing text to be represented using reusable pieces.

1. Tokens can represent parts of words

A tokenizer can split an uncommon word into smaller pieces.

For example:

unhappiness

might be represented as pieces such as:

un + happi + ness

The exact split depends on the tokenizer.

This means the model does not need a separate vocabulary entry for every possible word.

2. Tokens can handle unfamiliar words

Suppose the model encounters a rare company name, scientific term, or newly created word.

Even if the entire word is not in the vocabulary, the tokenizer can often represent it using smaller known pieces.

That makes token-based vocabularies much more flexible than word-only vocabularies.

3. Tokens keep the vocabulary manageable

A pure word-level vocabulary could grow to millions of entries.

A tokenizer instead uses a fixed vocabulary containing common words, subwords, punctuation, characters, or byte-level pieces.

This keeps the model's embedding and output layers more manageable.

4. Tokens work better across different kinds of text

LLMs process much more than ordinary sentences.

They may need to represent:

• Different languages
• Numbers
• URLs
• Source code
• Emojis
• Punctuation
• Misspellings
• Technical terms

Tokenization provides a common mechanism for representing all of these.

Why not use individual characters?

Character-level tokenization would avoid unknown words, but sequences would become much longer.

For example:

language

contains eight characters but might require only one or a few subword tokens.

Modern tokenizers therefore usually aim for a compromise:

whole words when efficient
+
subword or smaller pieces when necessary

Does every LLM use the same tokens?

No.

Different models can use different tokenizer algorithms and vocabularies, so the same sentence may produce different tokens and token counts across models.

For the tokenization process itself, see “What is tokenization in LLMs?”

In short: LLMs use tokens instead of only whole words because tokens keep the vocabulary manageable while still allowing the model to represent rare words, multiple languages, code, numbers, and unfamiliar text efficiently.