Auto-Encoding Variational Bayes: The Foundations of Variational Autoencoders
The paper introduces a practical way to train probabilistic latent-variable models with neural networks by optimizing a variational lower bound and using the reparameterization trick.
Key takeaway
A VAE has a probabilistic encoder that maps an input to a distribution over latent variables and a decoder that reconstructs or generates data from a latent sample. The encoder predicts a mean and variance; the reparameterization trick writes a sample as z = μ + σ ⊙ ε, where ε is standard Gaussian noise. Training maximizes the ELBO: expected reconstruction log-likelihood minus the KL divergence between the approximate posterior and the prior. This makes sampling differentiable with respect to encoder parameters and allows both networks to be trained jointly with stochastic gradient methods.
Updated on Nov 17, 2026
01
Abstract
How can we perform efficient inference and learning in directed probabilistic models, in the presence of continuous latent variables with intractable posterior distributions, and large datasets? We introduce a stochastic variational inference and learning algorithm that scales to large datasets and, under some mild differentiability conditions, even works in the intractable case. Our contributions are two-fold. First, we show that a reparameterization of the variational lower bound yields a lower bound estimator that can be straightforwardly optimized using standard stochastic gradient methods. Second, we show that for i.i.d. datasets with continuous latent variables per datapoint, posterior inference can be made especially efficient by fitting an approximate inference model (also called a recognition model) to the intractable posterior using the proposed lower bound estimator. Theoretical advantages are reflected in experimental results.
02
Methodology
The Core Idea
A VAE is an autoencoder whose latent representation is a probability distribution rather than a single fixed vector.
The encoder receives x and predicts parameters of an approximate posterior qφ(z|x), commonly a Gaussian mean μ(x) and standard deviation σ(x). A latent vector is sampled from this distribution. The decoder receives z and models the likelihood pθ(x|z), describing how likely the original observation is under the generative model.
The encoder learns to produce useful latent distributions; the decoder learns to generate observations from them. A regularization term encourages the encoded distributions to remain close to a prior, commonly N(0,I), so that sampling from the prior can produce new examples.
Learned two-dimensional Frey Face manifold
Original Figure 4(a) from the paper: generated Frey Face examples arranged across a two-dimensional latent manifold. Moving across the grid changes the latent coordinates and produces smooth variations in the decoded face.
Kingma and Welling, Auto-Encoding Variational Bayes, Figure 4(a), arXiv:1312.6114.
03
Architecture
The VAE Architecture
The paper's example uses two neural networks and a simple Gaussian latent-variable model.
The probabilistic encoder qφ(z|x) is a recognition network. In the paper's example, it outputs the mean and diagonal covariance (or standard deviations) of a Gaussian approximation to the posterior. The prior is an isotropic standard Gaussian p(z)=N(0,I).
The probabilistic decoder pθ(x|z) maps a latent sample to a distribution over observations. The paper uses a Bernoulli likelihood for binary-valued data and a Gaussian likelihood for real-valued data. The encoder and decoder parameters φ and θ are optimized jointly.
04
Training
How to Implement a Basic VAE
A practical implementation can use a multilayer perceptron encoder and decoder for a first experiment.
PyTorch-style training step:
mu, logvar = encoder(x)\nstd = torch.exp(0.5 logvar)\neps = torch.randn_like(std)\nz = mu + std eps\nreconstruction = decoder(z)\nrecon_loss = binary_cross_entropy(reconstruction, x, reduction="sum")\nkl_loss = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())\nloss = recon_loss + kl_loss\nloss.backward()\noptimizer.step()
For binary inputs, a Bernoulli decoder with a numerically stable binary cross-entropy objective is a common choice. For continuous observations, choose a likelihood model that matches the data, such as a Gaussian decoder. The exact reduction and scaling of reconstruction and KL terms must be kept consistent across the minibatch.
After training, generate samples by drawing z ~ N(0,I) and passing it through the decoder. For reconstruction, encode an input and decode a sampled or mean latent representation.
05
Results
Experiments and Findings
The authors evaluate generative models on MNIST and the Frey Face dataset, comparing AEVB with the wake-sleep algorithm and, for selected marginal-likelihood experiments, Monte Carlo EM.
The paper reports that AEVB converged considerably faster and reached a better variational lower bound than wake-sleep across the tested latent dimensionalities. It also demonstrates that a two-dimensional latent representation can be used to visualize a learned data manifold. The authors note that adding latent dimensions did not produce the expected overfitting in their experiments, attributing this behavior to the regularizing effect of the variational bound.
These are results for the datasets, architectures and optimization setup studied in the paper; they should not be read as a universal guarantee that VAEs always outperform other generative models.
Result board
Datasets
MNIST and Frey Face
Main comparison
AEVB versus wake-sleep
Result
AEVB achieved faster convergence and a better lower bound in the reported experiments
Result
Low-dimensional latent spaces enabled visualization of learned data manifolds
06
Limitations
Limitations and Practical Challenges
The method in this paper is developed for continuous latent variables and relies on a suitable differentiable reparameterization. Discrete latent variables do not admit the same simple Gaussian pathwise estimator and generally require different estimators or relaxations.
The encoder uses an approximate posterior family. If that family is too restrictive, it may not represent the true posterior well. The standard ELBO also creates a trade-off: excessive pressure toward the prior can reduce the information carried by the latent code, while weak regularization can make prior samples poorly aligned with encoded examples.
Finally, good reconstruction does not guarantee perceptually sharp samples. Later VAE research explores richer posterior families, alternative priors, hierarchical latent variables and modified objectives to address these limitations.
07
Conclusion
Key Takeaways
A VAE is not merely an autoencoder with random noise added. It is a probabilistic generative model trained by maximizing a principled variational lower bound.
The encoder predicts a distribution, the reparameterization trick makes sampling differentiable, and the KL term shapes the latent space relative to a prior. These three ideas explain the standard VAE training loop and why it can both reconstruct observations and generate new samples.
08
References
Source trail
Primary papers, implementation pages, and external source records used to ground this explainer.
The Wake-Sleep Algorithm for Unsupervised Neural Networks
Hinton et al., Science, 1995.
Earlier recognition-model learning for unsupervised neural networks.
Stochastic Variational Inference
Hoffman et al., JMLR, 2013.
Scalable variational inference using stochastic optimization and minibatches.
Stochastic Backpropagation and Approximate Inference in Deep Generative Models
Rezende, Mohamed and Wierstra, 2014.
Related pathwise-gradient work for deep latent Gaussian models.
β-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework
Higgins et al., ICLR 2017.
A later VAE variant that changes the KL weighting to encourage disentangled representations.
01Original arXiv paper02Full paper HTML with figures03arXiv DOIContinue reading