How does Byte Pair Encoding work?

Byte Pair Encoding (BPE) works by learning which adjacent text units frequently occur together and repeatedly merging those pairs into larger tokens.

The result is a vocabulary that contains a mixture of small pieces and frequently occurring larger pieces.

1. Start with small units

A simplified BPE process begins with small units such as characters.

Suppose the training data contains:

low
lower
lowest

Initially, the words can be represented as:

l o w
l o w e r
l o w e s t

2. Count adjacent pairs

BPE counts how often neighboring units appear together.

For example, pairs such as:

l + o
o + w
w + e

may occur multiple times.

3. Merge the most frequent pair

If:

l + o

is the most frequent pair, BPE merges it into a new unit:

lo

The sequences now become something like:

lo w
lo w e r
lo w e s t

4. Repeat the process

BPE counts adjacent pairs again.

If:

lo + w

is now frequent, it may merge them into:

low

The process continues, gradually creating larger and more useful tokens.

Conceptually:

small units
   ↓
count adjacent pairs
   ↓
merge the most frequent pair
   ↓
repeat
   ↓
final vocabulary + merge rules

5. Use the learned merges on new text

After training, the tokenizer keeps the learned vocabulary and merge rules.

When new text arrives, it applies those rules to break the text into known token pieces.

Common patterns may become single tokens, while uncommon words may remain split into several smaller pieces.

Why does this help?

BPE allows an LLM tokenizer to:

• Represent common text compactly
• Break rare words into reusable pieces
• Keep vocabulary size manageable
• Avoid relying entirely on whole-word tokens
• Reduce the unknown-word problem

Modern implementations may use variations such as byte-level BPE, but the central idea remains the same: frequent neighboring units are progressively merged into larger tokens.

For the basic definition, see “What is Byte Pair Encoding (BPE)?”

In short: BPE learns token pieces by repeatedly finding frequent adjacent units, merging them, and saving those merges so future text can be segmented using the learned vocabulary.