Ladder Variational Autoencoders
Variational autoencoders are powerful models for unsupervised learning. However deep models with several layers of dependent stochastic variables are difficult to train which limits the improvements obtained using these highly expressive models. We propose a new inference model, the Ladder Variational Autoencoder, that recursively corrects the generative distribution by a data dependent approximate likelihood in a process resembling the recently proposed Ladder Network. We show that this model provides state of the art predictive log-likelihood and tighter log-likelihood lower bound compared to the purely bottom-up inference in layered Variational Autoencoders and other generative models. We provide a detailed analysis of the learned hierarchical latent representation and show that our new inference model is qualitatively different and utilizes a deeper more distributed hierarchy of latent variables. Finally, we observe that batch-normalization and deterministic warm-up (gradually turning on the KL-term) are crucial for training variational models with many stochastic layers.
The authors propose a modification of the variational auto-encoder that consists in correcting the model's generative distribution with a data dependent likelihood factors, in a process that resembles the one used by the ladder network: a non-probabilistic auto-encoder that tries to match the values of the hidden layers in the encoder and decoder networks.  The proposed modification is based on using a deterministic inference network in the variational auto-encoder that samples the top latent variables from a Gaussian distribution, as in the variational auto-encoder, after that the latent variables at intermediate layers are sampled from a product of Gaussian factors, where one of the factors come from the original generative model, and the others come from the inference network.  The experiments show that the proposed modification results in better estimates of the test-log-likelihood. The proposed model learns richer latent distributions. The experiments performed also show that using batch normalization and a deterministic warm-up approach during training are important for obtaining improved results.  
Quality: the paper seems to be technically sound. However, I do not like that the authors report results on table 1 that are not directly comparable because of differences in the training procedure.  Clarity: the paper is clearly written. However, the authors could have done a much better job at explaining how the inference network generates values for the latent variables. I took me too long to fully understand the whole process. It is not clear how the authors apply the batch normalization procedure during training. They should describe this in the text. I assume they only use batch-normalization in the inference network since using it in the generative network would change the generative model. What are the discontinuous lines in Figure 2? They should be described in the caption of the figure.  Originality: the contribution is original. The authors proposed the first probabilistic equivalent of the ladder network, a model that has been very successful. They also provide significant insights on how to improve the VAE by using sensible modifications for the training process by using warm-ups and batch normalization.  Significance: the experimental results clearly show the advantages of the proposed approach. The modifications are therefore significant. However, it still not clear why this approach should be preferred with respect to other possible alternatives.