What is a token in an LLM?

A token is a unit of text that an LLM processes.

Before an LLM can work with text, a tokenizer converts the text into a sequence of tokens. Depending on the tokenizer, a token may represent:

• A whole word
• Part of a word
• Punctuation
• A space or other character pattern
• A special control symbol

For example, a tokenizer might split:

unbelievable

into smaller pieces such as:

un + believable

The exact split depends on the tokenizer being used.

Why does an LLM use tokens?

Neural networks operate on numbers rather than raw text.

Each token is therefore mapped to a numeric token ID.

Conceptually:

Text
  ↓
Tokens
  ↓
Token IDs
  ↓
Embeddings
  ↓
LLM

The model then converts each token ID into an embedding vector and processes the sequence through its Transformer layers.

Is a token the same as a word?

No.

Some common words may correspond to one token, while uncommon or complex words may be split into multiple tokens.

For example:

playing

could be represented as one token in one tokenizer but multiple tokens in another.

This is why token count and word count are not the same thing.

Why do tokens matter?

Tokens affect several practical aspects of an LLM:

• How much text fits inside the context window
• How much input and output is processed
• The computational cost of inference
• API usage when pricing is based on token counts
• How efficiently different languages or text patterns are represented

What are special tokens?

Some tokenizers also include special tokens used for structural purposes, such as marking the beginning or end of text, separating messages, or representing other control information.

These tokens are part of the model's vocabulary even though they may not correspond to ordinary visible words.

For the process that converts text into tokens, see “What is tokenization in LLMs?”

In short: a token is the basic text unit an LLM processes. Text is split into tokens, each token is converted into a numeric ID, and those IDs are transformed into embeddings that the model can work with.