Research

Auto-Encoding Variational Bayes: The Foundations of Variational Autoencoders

The paper introduces a practical way to train probabilistic latent-variable models with neural networks by optimizing a variational lower bound and using the reparameterization trick.

Diederik P. Kingma, Max WellingDec 20, 2013International Conference on Learning Representations (ICLR 2014)13 min readMachine Learning Group, Universiteit van Amsterdam
FOUNDATIONALGenerative AIIntermediate

Key takeaway

A VAE has a probabilistic encoder that maps an input to a distribution over latent variables and a decoder that reconstructs or generates data from a latent sample. The encoder predicts a mean and variance; the reparameterization trick writes a sample as z = μ + σ ⊙ ε, where ε is standard Gaussian noise. Training maximizes the ELBO: expected reconstruction log-likelihood minus the KL divergence between the approximate posterior and the prior. This makes sampling differentiable with respect to encoder parameters and allows both networks to be trained jointly with stochastic gradient methods.

Updated on Nov 17, 2026

01

Abstract

How can we perform efficient inference and learning in directed probabilistic models, in the presence of continuous latent variables with intractable posterior distributions, and large datasets? We introduce a stochastic variational inference and learning algorithm that scales to large datasets and, under some mild differentiability conditions, even works in the intractable case. Our contributions are two-fold. First, we show that a reparameterization of the variational lower bound yields a lower bound estimator that can be straightforwardly optimized using standard stochastic gradient methods. Second, we show that for i.i.d. datasets with continuous latent variables per datapoint, posterior inference can be made especially efficient by fitting an approximate inference model (also called a recognition model) to the intractable posterior using the proposed lower bound estimator. Theoretical advantages are reflected in experimental results.

02

Methodology

The Core Idea

A VAE is an autoencoder whose latent representation is a probability distribution rather than a single fixed vector.

The encoder receives x and predicts parameters of an approximate posterior qφ(z|x), commonly a Gaussian mean μ(x) and standard deviation σ(x). A latent vector is sampled from this distribution. The decoder receives z and models the likelihood pθ(x|z), describing how likely the original observation is under the generative model.

The encoder learns to produce useful latent distributions; the decoder learns to generate observations from them. A regularization term encourages the encoded distributions to remain close to a prior, commonly N(0,I), so that sampling from the prior can produce new examples.

Figure 1

Learned two-dimensional Frey Face manifold

Original Figure 4(a) from the paper: generated Frey Face examples arranged across a two-dimensional latent manifold. Moving across the grid changes the latent coordinates and produces smooth variations in the decoded face.

Kingma and Welling, Auto-Encoding Variational Bayes, Figure 4(a), arXiv:1312.6114.

03

Architecture

The VAE Architecture

The paper's example uses two neural networks and a simple Gaussian latent-variable model.

The probabilistic encoder qφ(z|x) is a recognition network. In the paper's example, it outputs the mean and diagonal covariance (or standard deviations) of a Gaussian approximation to the posterior. The prior is an isotropic standard Gaussian p(z)=N(0,I).

The probabilistic decoder pθ(x|z) maps a latent sample to a distribution over observations. The paper uses a Bernoulli likelihood for binary-valued data and a Gaussian likelihood for real-valued data. The encoder and decoder parameters φ and θ are optimized jointly.

04

Training

How to Implement a Basic VAE

A practical implementation can use a multilayer perceptron encoder and decoder for a first experiment.

PyTorch-style training step:

mu, logvar = encoder(x)\nstd = torch.exp(0.5 logvar)\neps = torch.randn_like(std)\nz = mu + std eps\nreconstruction = decoder(z)\nrecon_loss = binary_cross_entropy(reconstruction, x, reduction="sum")\nkl_loss = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())\nloss = recon_loss + kl_loss\nloss.backward()\noptimizer.step()

For binary inputs, a Bernoulli decoder with a numerically stable binary cross-entropy objective is a common choice. For continuous observations, choose a likelihood model that matches the data, such as a Gaussian decoder. The exact reduction and scaling of reconstruction and KL terms must be kept consistent across the minibatch.

After training, generate samples by drawing z ~ N(0,I) and passing it through the decoder. For reconstruction, encode an input and decode a sampled or mean latent representation.

05

Results

Experiments and Findings

The authors evaluate generative models on MNIST and the Frey Face dataset, comparing AEVB with the wake-sleep algorithm and, for selected marginal-likelihood experiments, Monte Carlo EM.

The paper reports that AEVB converged considerably faster and reached a better variational lower bound than wake-sleep across the tested latent dimensionalities. It also demonstrates that a two-dimensional latent representation can be used to visualize a learned data manifold. The authors note that adding latent dimensions did not produce the expected overfitting in their experiments, attributing this behavior to the regularizing effect of the variational bound.

These are results for the datasets, architectures and optimization setup studied in the paper; they should not be read as a universal guarantee that VAEs always outperform other generative models.

Result board

Datasets

MNIST and Frey Face

Main comparison

AEVB versus wake-sleep

Result

AEVB achieved faster convergence and a better lower bound in the reported experiments

Result

Low-dimensional latent spaces enabled visualization of learned data manifolds

06

Limitations

Limitations and Practical Challenges

The method in this paper is developed for continuous latent variables and relies on a suitable differentiable reparameterization. Discrete latent variables do not admit the same simple Gaussian pathwise estimator and generally require different estimators or relaxations.

The encoder uses an approximate posterior family. If that family is too restrictive, it may not represent the true posterior well. The standard ELBO also creates a trade-off: excessive pressure toward the prior can reduce the information carried by the latent code, while weak regularization can make prior samples poorly aligned with encoded examples.

Finally, good reconstruction does not guarantee perceptually sharp samples. Later VAE research explores richer posterior families, alternative priors, hierarchical latent variables and modified objectives to address these limitations.

07

Conclusion

Key Takeaways

A VAE is not merely an autoencoder with random noise added. It is a probabilistic generative model trained by maximizing a principled variational lower bound.

The encoder predicts a distribution, the reparameterization trick makes sampling differentiable, and the KL term shapes the latent space relative to a prior. These three ideas explain the standard VAE training loop and why it can both reconstruct observations and generate new samples.

Continue reading

Related research

Browse all research