How does an LLM tokenizer work?

An LLM tokenizer works by converting raw text into a sequence of token IDs that the language model can process.

It does this using a fixed vocabulary and a set of tokenization rules learned or defined before the LLM is trained.

1. The tokenizer receives raw text

Suppose the input is:

Machine learning is powerful.

The tokenizer first examines the characters and text patterns in the input.

2. It splits the text into known token pieces

Instead of requiring every possible word to exist in the vocabulary, modern tokenizers usually work with reusable subword or byte-level pieces.

A tokenizer might represent:

unbelievable

as something like:

un + believable

while a common word might remain a single token.

The exact result depends on the tokenizer and its vocabulary.

3. Each token is mapped to an ID

Every token in the tokenizer vocabulary has a numeric identifier.

Conceptually:

"Machine"  →  4821
" learning" →  6975
" is"       →  374
" powerful" →  8162
"."         →  13

The LLM receives these token IDs rather than the original text.

4. Special tokens may be added

A tokenizer can also insert special tokens used by the model or application.

These can mark things such as:

• Beginning or end of a sequence
• Message boundaries
• System, user, or assistant roles
• Padding or other control information

Which special tokens exist depends on the model.

5. Token IDs are converted into embeddings

Inside the LLM, each token ID is used to look up a learned embedding vector.

The overall pipeline is:

Raw text
   ↓
Tokenizer
   ↓
Token pieces
   ↓
Token IDs
   ↓
Embeddings
   ↓
Transformer

How is the tokenizer vocabulary created?

Before model training, a tokenizer is typically trained or configured using a large text corpus.

Algorithms such as Byte Pair Encoding (BPE), WordPiece, or Unigram can identify useful recurring text pieces and build a vocabulary of manageable size.

Different models may use different tokenization methods.

How does text come back from tokens?

Tokenization also works in reverse.

After the model generates token IDs, the tokenizer's decoding process maps those IDs back into token pieces and combines them into readable text.

For the broader concept, see “What is tokenization in LLMs?”

In short: an LLM tokenizer breaks text into vocabulary pieces, maps those pieces to numeric token IDs, handles any required special tokens, and can later decode generated token IDs back into text.