  Variational autoencoders employ an amortized inference model to approximate
the posterior of latent variables. However, such amortized variational
inference faces two challenges: (1) the limited posterior expressiveness of
fully-factorized Gaussian assumption and (2) the amortization error of the
inference model. We present a novel approach that addresses both challenges.
First, we focus on ReLU networks with Gaussian output and illustrate their
connection to probabilistic PCA. Building on this observation, we derive an
iterative algorithm that finds the mode of the posterior and apply
full-covariance Gaussian posterior approximation centered on the mode.
Subsequently, we present a general framework named Variational Laplace
Autoencoders (VLAEs) for training deep generative models. Based on the Laplace
approximation of the latent variable posterior, VLAEs enhance the
expressiveness of the posterior while reducing the amortization error.
Empirical results on MNIST, Omniglot, Fashion-MNIST, SVHN and CIFAR10 show that
the proposed approach significantly outperforms other recent amortized or
iterative methods on the ReLU networks.

Machine Learning