There have been numerous recent advancements on learning deep generative models with latent variables thanks to the reparameterization trick that allows to train deep directed models effectively. However, since reparameterization trick only works on continuous variables, deep generative models with discrete latent variables still remain hard to train and perform considerably worse than their continuous counterparts. In this paper, we attempt to shrink this gap by introducing a new architecture and its learning procedure. We develop a hybrid generative model with binary latent variables that consists of an undirected graphical model and a deep neural network. We propose an efficient two-stage pretraining and training procedure that is crucial for learning these models. Experiments on binarized digits and images of natural scenes demonstrate that our model achieves close to the state-of-the-art performance in terms of density estimation and is capable of generating coherent images of natural scenes.
Summary of the paper:
The paper proposes to augment a variational auto encoder (VAE) with an binary restricted Boltzmann machine (RBM) in the role of the prior of the generative model. To yield a good initialisation of the parameters of the RBM and the inference network a special pertaining procedure is introduced. The model produces competitive Likelihood results on MNIST and was further tested on CIFAR 10. 
Clarity and quality: 
1. From the description of the pertaining procedure and the appendix B I got the impression that the inference network maps into [0,1] and not into {0,1}.  Does it mean, you are not really considering binary latent variables (making the RBM model the values in [0,1] by its probability p(z|h))? 
2. on page 2:
RWS...."derive a tighter lower bound": Where does the "tighter" refer to? 
3. "multivariate Bernoulli modeled by an RBM": Note,  while in a multivariate Bernoulli the binary variables would be independent from each others, this is usually not the case for the visible variables of RBMs (only in the conditional distribution given the state of the hidden variables).
4. The notation could be improved, e.g.:
-x_data and x_sample are not explained
- M is not defined in equation 5. 
5. "this training method has been previously used to produce the best results on MNIST" Note, that parallel tempering often leads to better results when training RBMs (see http://proceedings.mlr.press/v9/desjardins10a/desjardins10a.pdf) . Furthermore, centred RBMs are also get better results than vanilla RBMs (see: http://jmlr.org/papers/v17/14-237.html).
Originality and significance:
As already mentioned in a comment on open-review the current version of the paper misses to mention one very related work: "discrete variational auto encoders". Also "bidirectional Helmholtz machines" could be mentioned as generative model with discrete latent variables.  The results for both should also be reported in Table 1 (discrete VAEs: 81,01, BiHMs: 84,3). 
From the motivation the advantages of the model did not become very clear to me. Main advantage seems to be the good likelihood result on MNIST (but likelihood does not improve compared to IWAE on CIFAR 10 for example). However, using an RBM as prior has the disadvantage that sampling from the generative model requires running a Markov chain now while having a solely directed generative model allows for fast sampling. 
Experiments show good likelihood results on MNIST. Best results are obtained when using a ResNet decoder. I wondered how much a standard VAE is improved by using such a powerful decoder. Reporting this, would allow to understand, how much is gained from using a RBM for learning the prior. 
Minor comments:
page 1:
"debut of variational auto encoder (VAE) and reparametrization trick" -> debut of variational auto encoders (VAE) and the reparametrization trick",
page 2:
"with respect to the parameter of  p(x,z)" -> "with respect to the parameters of  p(x,z)"
"parameters in p" -> "parameters of p" 
"is multivariate Bernoulli" ->  "is a multivariate Bernoulli"
"we compute them" -> "we compute it" 
page 3:
"help find a good" ->  "help to find a good"
page 7:
"possible apply" -> "possible to apply"