What is classifier-free guidance in diffusion models?

Classifier-free guidance (CFG) improves the ability of a diffusion model to follow a text prompt or class label by combining two noise predictions: one with the requested condition and one without it. It does not require a separately trained image classifier. The technique was introduced by Ho and Salimans and is widely used in conditional generation.

Why is it called classifier-free?

In classifier guidance, a separate classifier trained to recognize conditions in noisy inputs supplies gradients that steer image generation. That additional network must work across diffusion timesteps.

In classifier-free guidance, the diffusion model itself estimates how conditioning changes its denoising prediction:

• Conditional branch: Predict noise from the noisy sample xₜ, timestep t, and requested condition c. Write this as ε_cond = εθ(xₜ,t,c).
• Unconditional branch: Predict noise from the same xₜ and t but a null or dropped condition ∅. Write this as ε_uncond = εθ(xₜ,t,∅).
• Guidance direction: Subtract the unconditional prediction from the conditional prediction.
• Guided prediction: Extrapolate in that direction to strengthen the contribution of the condition in the next reverse denoising step.

A practical implementation usually uses one shared denoiser with two forward evaluations, rather than two independently trained generators. The word "classifier-free" does not mean the model is trained without classes, captions, or other useful conditions.

How does the model learn both conditional and unconditional predictions?

A standard method is condition dropout during training:

1. Take a clean training image and its associated condition (a class label such as dog or a caption such as red bicycle).
2. Construct a noisy training sample at a randomly selected timestep, with Gaussian noise known to the training code.
3. Randomly keep the condition for most examples and replace it with a designated null condition in a minority of examples.
4. Predict the noise with the same denoiser. If the condition was retained, it learns conditional behavior; if dropped, it learns unconditional behavior.
5. Compare the prediction to the same known sampled noise using the ordinary noise-prediction loss, and backpropagate through the shared parameters.
6. Repeat across different data, timesteps, prompts and noise draws.

Numerical example of condition dropout: If the probability of dropping the condition is 0.10, then among 100 independently sampled examples, the expected number of unconditional examples is 100×0.10=10 and the expected conditional count is 90. A particular random batch might have more or fewer than 10 dropped conditions.

The dropout probability is a tuning choice, not a universal constant. With zero dropout, the model might never learn how to respond to its null condition. With 100% dropout, it cannot learn to use the actual prompt. Some architectures use a learned null embedding, others a consistently defined empty-conditioning representation.

Important: Condition dropout changes only the condition presented to the denoiser; it does not replace the training noise target or introduce a second independently optimized classifier.

The classifier-free guidance equation

Let the unconditional and conditional noise predictions be ε_uncond and ε_cond. Under the guidance convention used in this lesson, the combined noise prediction is:

ε_guided = ε_uncond + s × (ε_cond − ε_uncond)

Equivalently, ε_guided = (1−s) × ε_uncond + s × ε_cond.

The parameter s is the guidance scale:

• s=0: Exactly the unconditional prediction.
• s=1: Exactly the ordinary conditional prediction.
• s>1: Extrapolation past the conditional prediction in the direction away from the unconditional prediction.

Beware guidance conventions: Some literature or libraries instead write ε_guided = ε_cond + w × (ε_cond − ε_uncond). Both formulas describe the same guidance when s = 1+w. Thus ordinary conditional sampling corresponds to s=1 in the first version or w=0 in the second version. Check the implementation before choosing a scale.

Beginner-friendly two-element numerical example

Imagine the diffusion model is working with a small two-feature noisy sample at one fixed timestep, and the requested prompt is "a red bicycle." These are illustrative model outputs, not measured probabilities or actual pixel labels.

Step 1 — Run the unconditional branch.

Assume it predicts ε_uncond = [0.2, −0.4].

Step 2 — Run the conditional branch at the SAME state and timestep.

Assume it predicts ε_cond = [0.5, −0.1].

Step 3 — Find the difference.

ε_cond − ε_uncond = [0.5−0.2, −0.1−(−0.4)] = [0.3, 0.3].

Step 4 — Use guidance scale s=2.

ε_guided = [0.2,−0.4] + 2 × [0.3,0.3]

= [0.2+0.6, −0.4+0.6]

= [0.8, 0.2].

Step 5 — Compare three simple cases.

• At s=0, ε_guided = [0.2, −0.4] (unconditional).
• At s=1, ε_guided = [0.5, −0.1] (conditional without extrapolation).
• At s=2, ε_guided = [0.8, 0.2] (condition direction amplified).

The s=2 vector is not an image-quality score. It does not mean twice the prompt accuracy, twice the realism, or a twofold probability of success. It is a linear combination of two noise estimates.

Optional consequence for the clean-sample estimate

Suppose the noisy vector is xₜ=[1.2,−0.7] with cumulative alpha-bar ᾱₜ=0.64. For a noise estimate ε, a common predicted-clean-vector expression is:

x̂₀ = (xₜ − sqrt(1−ᾱₜ)×ε) / sqrt(ᾱₜ)

Because sqrt(1−0.64)=0.6 and sqrt(0.64)=0.8:

• Ordinary conditional, s=1: ([1.2,−0.7] − 0.6×[0.5,−0.1])/0.8 = [1.125, −0.800].
• Guided, s=2: ([1.2,−0.7] − 0.6×[0.8,0.2])/0.8 = [0.900, −1.025].

This shows how different guided noise predictions change a reverse-step input. The results are toy estimates, not observed samples or proof that the guided output is better.

Training versus inference

During training:

• Construct noisy examples from clean images, choosing timesteps and known Gaussian noise.
• Randomly retain or drop the conditioning information.
• Pass the resulting condition, noisy sample and timestep through the denoiser.
• Use the standard noise-prediction regression loss to learn shared weights.

During inference:

• Evaluate the denoiser with the requested prompt c and the null condition ∅ at identical xₜ and t.
• Compute ε_guided with the guidance equation.
• Supply this guided noise prediction to the diffusion sampler for its next reverse step.
• Repeat at subsequent timesteps until the sample is generated.

One simple pseudocode sequence:

1. pred_uncond = denoiser(x_t, t, null_condition)
2. pred_cond = denoiser(x_t, t, prompt)
3. pred_guided = pred_uncond + s × (pred_cond − pred_uncond)
4. next_x = sampler_step(x_t, t, pred_guided)

No retraining is required to change s when the model already supports both branches. However, applying this formula to a model that never learned an unconditional branch may give unreliable results.

Inference cost: Standard CFG generally makes two denoiser evaluations per sampling timestep. Batched evaluations may improve device throughput but still do more total work than a single conditional evaluation. Some accelerated/distilled systems reduce this overhead.

Practical applications

• Text-to-image generation: Strengthen generation of objects, attributes and scenes described by a caption.
• Class-conditional generation: Emphasize an object or category label without a separate noisy-image classifier.
• Editing or structured conditioning: Related methods may be used with masks, reference images, layouts or other conditions when supported by training.
• Controlled comparisons: A fixed initial-noise seed and a consistent sampler let researchers investigate how the condition affects outputs.

The main benefit is a simple inference-time steering mechanism using already learned conditional and unconditional behavior.

What are the trade-offs and limitations?

• Prompt adherence versus variety: Guidance may increase fidelity to the condition but reduce the diversity of valid outputs or the coverage of rare modes.
• Visual artifacts: Excessive extrapolation can produce oversaturated colors, unnatural edges, poor contrast or repeated compositions.
• More inference computation: Two model evaluations per step commonly increase latency and compute.
• Unreliable null branch: Insufficient condition dropout or mismatched null embeddings can make ε_uncond unsuitable for CFG.
• Prompt interpretation limitations: Rare objects, unsupported attributes, incorrect labels or ambiguous captions can yield images that do not match the requested meaning.
• Not an accuracy guarantee: A visually convincing prompt-aligned image can still contain incorrect counts, text, details or unsupported content.
• No universally best guidance strength: Optimal values depend on architecture, data, sampler, condition and evaluation criteria.

The detailed effect of sweeping guidance scale, including diversity and fidelity trade-offs, belongs to Question 291: How does guidance scale affect diffusion sampling? This lesson concentrates on the CFG mechanism and two-element arithmetic.

Common implementation mistakes

• Mismatching xₜ or t: Conditional and unconditional predictions must be computed at the same noisy state and timestep.
• Subtracting in the wrong order: With the stated convention, the difference is ε_cond−ε_uncond.
• Using an arbitrary null input: A blank string, zeros and a learned null token are not necessarily interchangeable. Use the null-conditioning representation seen during training.
• Confusing guidance conventions: s=1 here corresponds to ordinary conditional prediction, whereas the alternative offset parameter w=0 does.
• Missing condition dropout: Do not assume the model understands an absent condition if it was never trained on one.
• Mistaking noise values for probabilities: Components of ε_cond, ε_uncond and ε_guided are noise-prediction values, not confidence scores.
• Ignoring diversity: Compare multiple samples and conditions, not just one polished image.
• Ignoring compute and memory: Evaluate actual inference costs from evaluating both branches.

Evaluation: Check prompt/class adherence, sample fidelity, output diversity and coverage, stability across random seeds, and device latency. The size of the guidance vector is not a reliable standalone quality metric.

Curriculum boundary: Question 288 introduced noise-prediction training. Question 289 compared DDPM and DDIM sampling. This lesson covers only classifier-free conditioning and combining predictions. Question 291 will analyze guidance scale effects rather than repeating the core mechanism.

Key takeaway: Classifier-free guidance trains one denoising network with occasional condition dropout, then combines unconditional and conditional noise predictions to steer generation without an auxiliary classifier. It is useful but entails accuracy/diversity and compute trade-offs.

Research: Ho & Salimans, Classifier-Free Diffusion Guidance; Dhariwal & Nichol, Diffusion Models Beat GANs on Image Synthesis; Nichol et al., GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models.