What is speculative decoding, and how does it accelerate LLM inference?

Speculative decoding is an LLM inference technique that lets a fast draft model propose several output tokens, then uses the larger target model to verify them together. When enough draft tokens are accepted, the target model needs fewer sequential decoding passes, reducing response latency.

The key idea is: make cheap predictions first; verify several of them with the expensive model in parallel.

How does it work?

1. Draft: A small, fast model proposes a short sequence of candidate tokens.
2. Verify: The target LLM evaluates those candidates in one forward pass. Because all proposed tokens are available as input, its causal attention can evaluate several positions together.
3. Accept or correct: The algorithm accepts a valid *consecutive prefix* of the draft. At the first rejected token, the target model supplies a corrected continuation using the appropriate verification rule.
4. Repeat: The process continues until generation ends.

Draft model:   "The answer is probably 42"
                ↓  proposes multiple tokens
Target model:  verifies proposed positions together
                ↓
Accepted prefix + any corrected token → final output

A practical example

Suppose a draft model predicts the next four tokens as:

["The", "capital", "is", "London"]

The target model may accept the first three but reject "London" and generate "Paris" instead. The final output follows the target model's verified decision, not the draft model's guess.

For greedy decoding, a candidate is accepted when it matches the target's required next token. For sampling, standard speculative decoding uses an acceptance/rejection-and-correction rule, rather than merely comparing the most likely tokens.

Why can this be faster?

Ordinary autoregressive decoding typically needs another large-model forward pass before each next token can be chosen. Speculative decoding can validate several proposed positions in one target-model pass.

That is useful because short-sequence decoding on modern accelerators is often limited by memory movement and sequential execution, while checking a batch of proposed positions can use the hardware more efficiently.

The speedup comes from fewer serial target-model calls, minus the extra cost of drafting and verification. It is not a guaranteed 2× improvement.

Does speculative decoding reduce accuracy?

Standard, correctly implemented speculative sampling can preserve the target model's exact output probability distribution (subject to normal numerical and implementation constraints). The draft is only a proposal mechanism; the target's distribution remains authoritative.

This differs from using a smaller model alone or changing the target with quantization. It does not mean independent random runs must produce the same exact words.

When does it work well—and when not?

It works best when:

• The draft model is much cheaper than the target.
• The target accepts many proposed tokens.
• The workload benefits from lower decode latency.

It can disappoint when the draft is slow or inaccurate, proposals are frequently rejected, verification has high overhead, or the serving system already uses highly efficient large-batch decoding. Some implementations also place constraints on batched inputs and tokenizer compatibility.

Related techniques

• KV caching saves previously computed attention keys and values; it tackles repeated work, not the same serial-token bottleneck.
• Self-speculative decoding can draft with a model's own early layers.
• Prompt-lookup decoding can propose tokens copied from matching prompt segments.
• Distillation may create a suitable small draft model, but speculative decoding itself does not require training the target.

In short: speculative decoding trades a little cheap drafting for fewer expensive, sequential target-model passes—potentially making LLM generation faster without changing the target model's sampling distribution.

References: Hugging Face: Assisted decoding, Fast Inference from Transformers via Speculative Decoding, and Accelerating LLM Decoding with Speculative Sampling.