What is the Evidence Lower Bound (ELBO) in a variational autoencoder?

The Evidence Lower Bound (ELBO) is the probability-based training objective of a Variational Autoencoder (VAE). It allows a generative model to learn from observed examples even when the exact probability of each example is too expensive to compute. ELBO rewards latent variables that explain the observed data while penalizing excessive mismatch between the encoder's latent distribution and a chosen prior.

What is the ELBO formula?

For one observed example x, and latent variable z, the standard VAE objective is:

ELBO(x) = E_{qφ(z|x)}[log pθ(x|z)] − KL(qφ(z|x) || p(z))

• `x` — observation: An image or another data example being modeled.
• `z` — latent variable: An unobserved numerical representation of the example.
• `qφ(z|x)` — approximate posterior: The encoder's learned distribution over possible latent values for this *specific* x; it approximates the true posterior pθ(z|x).
• `p(z)` — prior: A distribution chosen before observing the data, commonly N(0,1) for each latent dimension.
• `pθ(x|z)` — decoder likelihood: How probable the observed x is under the decoder for a sampled z; this depends on the decoder's chosen observation model.
• Expected reconstruction log-likelihood: E_q[log pθ(x|z)], which favors latent descriptions that explain the observed data. Larger is better.
• KL divergence: KL(qφ(z|x) || p(z)), a nonnegative difference between the encoder distribution and the prior, subtracted from the objective.

Using natural logarithms gives terms measured in nats. The likelihood is a probability model, not automatically a pixel-wise error.

Why is it a *lower bound*?

The model's exact probability for the observation is its marginal likelihood (also called evidence):

pθ(x) = ∫ pθ(x|z)p(z) dz

Computing this integral over all possible z is generally difficult for a deep decoder. Instead, a VAE trains an encoder to provide a tractable approximate posterior.

The mathematical identity behind ELBO is:

log pθ(x) = ELBO(x) + KL(qφ(z|x) || pθ(z|x))

• The KL here compares the approximate posterior to the true posterior, and is always nonnegative.
• Consequently, ELBO(x) ≤ log pθ(x).
• If the approximate and true posteriors match, the bound is tight.
• Do not confuse the two KL terms: The training ELBO compares qφ(z|x) to the prior `p(z)`, whereas the gap identity compares it to the true posterior `pθ(z|x)`.

Beginner-friendly numerical example

For understandable arithmetic, imagine a VAE modeling one observed binary pixel `x=1`. This could be one component of a binarized image. The decoder probabilities and latent samples below are hypothetical, not measurements from a trained model.

Step 1 — Specify prior and approximate posterior.

• Prior: p(z) = N(0,1).
• Encoder's approximate posterior for the observation: qφ(z|x=1) = N(0.5,1).
• Both have variance 1. The closed-form KL for these two Gaussians reduces to 0.5 × (0.5²) = 0.1250.

So the KL penalty = 0.1250 nats.

Step 2 — Estimate the reconstruction log-likelihood.

Suppose we draw two latent samples, z₁=0.3 and z₂=0.8, from the encoder distribution. The Bernoulli decoder assigns the actually observed pixel x=1 these probabilities:

• pθ(x=1|z₁) = 0.8, giving ln(0.8) ≈ −0.2231.
• pθ(x=1|z₂) = 0.6, giving ln(0.6) ≈ −0.5108.

Approximate the expected log-likelihood using the average of two log-probabilities:

Reconstruction term ≈ [ln(0.8) + ln(0.6)]/2 = -0.3670

This is a two-sample Monte Carlo estimate of the expected reconstruction term, not the exact expectation.

Step 3 — Calculate the estimated ELBO.

Estimated ELBO ≈ -0.3670 − 0.1250 = -0.4920

For these illustrative values, the reconstruction log-likelihood contributes -0.3670, and the KL divergence contributes 0.1250.

Step 4 — Translate maximization into a training loss.

The standard training objective is to maximize ELBO. Since many optimizers minimize losses, implementations instead minimize negative ELBO:

Negative ELBO loss ≈ −(-0.4920) = 0.4920

A higher ELBO and a lower negative ELBO indicate improvement in the same objective; they differ by a minus sign, not by the model being trained.

Important numerical caveat: The exact expectation-based ELBO is a mathematical lower bound on log pθ(x). A finite-sample estimate such as -0.4920 fluctuates with random latent samples and need not itself be below the true log evidence for every individual draw.

Why do VAEs need ELBO?

• Tractable learning: Replaces the generally intractable marginal-likelihood optimization with an objective that can be estimated and differentiated.
• An encoder and a decoder learn together: The encoder approximates inference in latent space; the decoder learns a probability distribution for reconstructing or generating observations from latent variables.
• Probabilistic latent space: The prior allows new latent samples to be passed through the trained decoder to generate new data.
• A principled objective: Reconstruction likelihood and KL penalty both arise from one lower-bound calculation rather than being unrelated heuristic scores.

Practical applications and limitations

• Applications: Generative image or signal modeling, representation learning, missing-data imputation and anomaly scoring when the observation model is suitable.
• Loose lower bound: A restrictive approximate-posterior family may fail to match the true posterior, leaving a significant gap between ELBO and exact log evidence.
• Likelihood assumptions: Bernoulli observations suit binary data; continuous-valued observations need appropriate likelihood models. Decoder assumptions change how reconstruction quality is measured.
• Sampling variability: Using few latent samples introduces noise into the numerical reconstruction estimate.
• Model quality: A higher training ELBO does not guarantee visually convincing samples or features useful for a particular downstream task. Evaluate held-out data and generated samples.

Curriculum boundary: The general VAE components were introduced in Question 97. Question 274 addressed a distinct *masked autoencoder* architecture. Question 276 examines the detailed reconstruction/KL trade-off, while Question 277 treats differentiable latent sampling. Those details are intentionally left for their own lessons.

Key takeaway: ELBO = expected decoder reconstruction log-likelihood − KL divergence of the approximate posterior from the prior. Maximizing it (or minimizing its negative) is the basic trainable objective of a VAE.

Research: Kingma & Welling, *Auto-Encoding Variational Bayes*; Rezende, Mohamed & Wierstra, *Stochastic Backpropagation and Approximate Inference in Deep Generative Models* (ICML 2014); Doersch, *Tutorial on Variational Autoencoders*.