Flexible and accurate inference and learning for deep generative models
We introduce a new approach to learning in hierarchical latent-variable generative models called the "distributed distributional code Helmholtz machine", which emphasises flexibility and accuracy in the inferential process. Like the original Helmholtz machine and later variational autoencoder algorithms (but unlike adversarial methods) our approach learns an explicit inference or "recognition" model to approximate the posterior distribution over the latent variables. Unlike these earlier methods, it employs a posterior representation that is not limited to a narrow tractable parametrised form (nor is it represented by samples). To train the generative and recognition models we develop an extended wake-sleep algorithm inspired by the original Helmholtz machine. This makes it possible to learn hierarchical latent models with both discrete and continuous variables, where an accurate posterior representation is essential. We demonstrate that the new algorithm outperforms current state-of-the-art methods on synthetic, natural image patch and the MNIST data sets.
The paper describes a novel approach to perform posterior inference and learning in deep generative models. This work is mainly motivated by the fact that recent VAE approaches still have difficulties in learning models with multiple layers, and that they have difficulties with adequately capturing structured posteriors. The key contributions in this paper are  * a novel approach to capture the posterior indirectly, using an inference network predicting expectations of random statistics w.r.t. the true posterior, and * demonstrating how to exploit this indirect representation for learning the generative model. To that end, the authors propose the Distributed Distributional Code (DDC) Helmholtz Machine, an extension to the classical Helmholtz Machine, including an extended wake-sleep algorithm.  Strengths: + a quite refreshing approach to approximate inference and learning. The proposed approach combines several existing techniques in an intriguing way. In particular, I think that the approach to represent the posterior via estimated expectations of random statistics is a very powerful idea, with a potentially large impact on approximate inference/learning. Also the approach to circumvent an explicit representation of the approximate posterior density, but rather to derive required expectations (gradients) in a direct way, is a promising direction. + Experimental evaluations demonstrate -- to a certain extent -- the efficacy of the approach, with partially dramatic improvements over variational approaches.  Weaknesses: - the approach inherits the disadvantages of wake-sleep. Thus, there is no clear objective optimized here and the method might even diverge. Furthermore, there is an additional approximation step which was not present in the original wake-sleep algorithm, namely the prediction of the required expectations/gradients. While the authors mention that the original wake-sleep algorithm has these drawbacks, they don't discuss them at all in their approach. - the motivation to fix VAEs' problem to learn deep models is addressed rather briefly in the experiments, by learning a model over binarized MNIST and 3 hidden sigmoid layers. This is not too convincing.  Quality: The paper is technically sound, in that way that the choices made are clear and very convincing. However, as the approach is based in wake-sleep, it is clear that the approach is heuristic, in particular as it includes a further approximation (expectations/gradients used for learning). The consequences of this fact and weaknesses of the model should be discussed in a more open way and ideally be addressed in the experimental evaluation (e.g. showing fail cases, or discussing why there are none).   Clarity: The paper is very well written and easy to follow.  Originality: The approach presented is, to the best of my knowledge, novel, natural and elegant.   Significance: Although the proposed is heuristic, it has a high potential to trigger further research in this direction.  Summary: while the paper has its weaknesses (heuristic, which should be better discussed), its originality and potential inspiration for further work in this direction are the main reasons why I recommend an accept.   *** EDIT *** I read the authors' reply and I'm rather happy with it.  While their point is convincing that the approximation in DDC-HM can be made arbitrarily close, I'm still not sure if we can guarantee stable training. The authors claim that it can be shown that the algorithm approaches the vicinity of a stationary point of the likelihood, if the gradient approximation error is small. I think this might be a crucial point, but I don't fully understand if this implies convergences (rather than oscillatory behavior) and how small the error needs to get. In any case, the authors should include this result in their paper.  And again, nice and refreshing work, I stick with my original rating.