An Architecture for Deep, Hierarchical Generative Models
We present an architecture which lets us train deep, directed generative models with many layers of latent variables. We include deterministic paths between all latent variables and the generated output, and provide a richer set of connections between computations for inference and generation, which enables more effective communication of information throughout the model during training. To improve performance on natural images, we incorporate a lightweight autoregressive model in the reconstruction distribution. These techniques permit end-to-end training of models with 10+ layers of latent variables. Experiments show that our approach achieves state-of-the-art performance on standard image modelling benchmarks, can expose latent class structure in the absence of label information, and can provide convincing imputations of occluded regions in natural images.
Arguing that the success of DRAW-like models is due their incremental approach to generating data combined with deterministic connections present between each latent variable and the observation, the authors apply the same design principles to hierarchical latent variable models. The resulting models, called Matryoshka Networks (MatNets), consist of a bottom-up (BU) network, top-down network (TD), and merge modules that directs information flow between the two. The MatNet architecture is reminiscent of Probabilistic Ladder Networks (PLNs) [28], with a deterministic pathway from the observation to a latent layers in the inference model and a top-down factorization of the variational posterior. The primary difference are deterministic connections between the latents and the observation in the Matryoshka Network, which are not present in PLNs. MatNets are evaluated at density modeling on MNIST, Omniglot, and CIFAR achieving either highly competitive or state-of-the-art results.
This is an interesting and fairly well executed paper.  The main contribution of the paper is a hierarchical VAE architecture with deterministic connections in the inference and generative models, implementing incremental generation of observations. A similar approach has been pioneered by DRAW, but in DRAW-like models the layers of latent variables don't form a hierarchy (and are all at essentially the same level). Prob Ladder Nets have all the proposed features except for the deterministic connections in the generative model. Thus the main novel contribution here is the introduction of deterministic connections in a generative hierarchical model. Surprisingly, the experimental section has no experiments demonstrating the importance of deterministic connections in the inference model and/or the generative model. Instead, the experiments show that the proposed architecture performs very well on several dataset, which though interesting but does not provide direct evidence for the importance of the paper's central claim.  The paper is fairly well written, but the presentation of the proposed architecture is not clear enough to be easily accessible. It would be much easier to read if the high-level (probabilistic) description of the model came before the specification of the exact computations involved (i.e. Procedural Description in Sec. 2.1).  In a few places the lack of citations can make some statements appear to be claims of novelty. For example, the discussion of using convolutional GRUs modules fails to mention the prior use of these in generative models in [24]. Given the many similarities with DRAW and PLNs, it is surprising that the related work section mentions neither. While the relation of PLNs to  DRAW is well explained in the introduction, PLNs are not mentioned in the paper at all, which is something the authors should correct.  Was the inference regularization technique from Section 2.3 used for any of the experiments?  Typos: Equations 11 and 12 both should have log in front for p(x|z).