How is perplexity calculated for an LLM?

Perplexity (PPL) for a causal LLM is calculated by measuring how much probability the model assigns to the actual next token at each scored position, averaging those negative log probabilities, and taking the exponential.

The formula is:

PPL = exp( - (1/N) × Σ log P(token_i | preceding tokens) )
    = exp(average next-token cross-entropy loss)

Here N is the number of valid predicted tokens, not the number of sentences or the vocabulary size. Logarithms are natural logs. For the intuition behind this metric, see “What is perplexity in language models?”

Step-by-step numerical example

Suppose an LLM predicts three observed tokens and assigns these probabilities to the tokens that actually occur:

• Token 1: 0.50
• Token 2: 0.25
• Token 3: 0.125

Step 1 — Take the negative natural log of each probability:

-ln(0.50)  = 0.6931
-ln(0.25)  = 1.3863
-ln(0.125) = 2.0794

Step 2 — Average the losses across the three scored tokens:

Average NLL = (0.6931 + 1.3863 + 2.0794) / 3
            ≈ 1.3863

Step 3 — Exponentiate the average:

PPL = exp(1.3863) ≈ 4

Result: Perplexity ≈ 4. Lower loss produces lower perplexity. Do not average the probabilities first and invert the result; that is generally a different calculation.

The example assumes preceding context exists for each token. In a real model, the first input token usually has no preceding token to predict it unless a beginning-of-sequence token is provided.

Where do the token probabilities come from?

1. Tokenize a held-out evaluation text with the model's tokenizer.
2. Run the model using the reference tokens as preceding context (often called *teacher forcing*).
3. For each scored position, obtain the model's next-token probability distribution.
4. Select the probability assigned to the reference next token, not necessarily the highest-probability token.
5. Sum its negative log probabilities, divide by the number of valid scored tokens, and exponentiate.

You do not need the model to freely generate an answer and then compare the generated paragraph to the reference.

Practical Python example with Hugging Face

This example calculates perplexity for one short text using a causal LM such as GPT-2:

PYTHON
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "gpt2"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
model.eval()

text = "The capital of France is Paris."
inputs = tokenizer(text, return_tensors="pt")

with torch.no_grad():
    outputs = model(
        input_ids=inputs["input_ids"],
        labels=inputs["input_ids"],
    )

ppl = torch.exp(outputs.loss).item()
print(f"Perplexity: {ppl:.2f}")

For GPT-2 and similar Hugging Face causal-LM classes, passing labels=input_ids invokes the next-token label shift inside the model. outputs.loss is the mean cross-entropy over the predicted positions, and torch.exp(outputs.loss) gives their perplexity. This is a demonstration for a short text, not a reliable benchmark by itself.

How do you calculate corpus-level perplexity correctly?

A serious evaluation uses many held-out tokens. Track:

total_NLL   = Σ (chunk_mean_NLL × valid_tokens_in_chunk)
total_tokens = Σ valid_tokens_in_chunk

Corpus_PPL = exp(total_NLL / total_tokens)

Do not simply average perplexity values from different chunks, particularly if they contain different numbers of valid tokens. First aggregate token-weighted losses, then exponentiate once.

Ignore padding or context-only positions in the loss. Hugging Face models commonly use label value `-100` to mask ignored positions. When a causal model internally shifts labels, count only the labels that actually contribute to its returned loss.

What if the text exceeds the context window?

For fixed-context Transformers, evaluating disjoint chunks independently loses preceding context at every new chunk boundary. A strided sliding window is usually a better approximation:

• Reuse overlapping context from the previous window.
• Calculate loss only for the new target tokens, not the overlapping context tokens.
• Sum those losses and divide by the true number of predicted tokens.

This may produce a different—often lower—perplexity from disjoint chunking because each target token gets more relevant context.

Common mistakes

• Incorrect token alignment: Predict token i+1 using logits from position i, not logits from the same target position.
• Wrong denominator: Divide by scored tokens, excluding padding and other masked positions.
• Comparing different tokenizers: Token-level PPL is sensitive to tokenization and test preprocessing.
• Mixing evaluation protocols: Keep the test corpus, context window/stride, model mode, and prompt formatting consistent.
• Interpreting PPL as accuracy: Low perplexity measures predictive fit, not guaranteed factual correctness or reasoning quality.

In short: Get the probabilities of the actual next tokens, calculate their mean negative log-likelihood, and apply `exp`. For valid model comparisons, also use consistent tokenization and context-window evaluation.

References: Hugging Face: Perplexity of fixed-length models and Hugging Face: Causal language modeling.