Recently proposed normalizing flow models such as Glow (Kingma & Dhariwal, 2018) have been shown to be able to generate high quality, high dimensional images with relatively fast sampling speed. Due to their inherently restrictive architecture, however, it is necessary that they are excessively deep in order to train effectively. In this paper we propose to combine Glow model with an underlying variational autoencoder in order to counteract this issue. We demonstrate that such our proposed model is competitive with Glow in terms of image quality while requiring far less time for training. Additionally, our model achieves state-of-the-art FID score on CIFAR-10 for a likelihood-based model.
The paper proposes a combination of a conditioned flow-based model with a VAE. The main contribution of the paper is a two-phase training that allows to train the model. Unfortunately, a joint training of the model failed. In general, combining VAEs with flow-based models is an important research direction. Unfortunately, the paper is not clearly written. A lot of details are missing that makes the paper impossible to reproduce and fully understand. Moreover, the most interesting part about training procedure, is discussed superficially. Finally, I find the comparison to SOTA methods disappointing. The authors skipped many recent papers. I do not expect to see SOTA results among all models, but at least comparable results within a group of approaches. However, the authors provide only a small subset of models that makes me wonder whether they are aware of other papers.
Remarks:
- The following statement is not fully correct:
"In our implementation we use a normalizing flow similar in structure to Real NVP Dinh et al. (2016) (which is a special case of autoregressive normalizing flows Papamakarios et al. (2017)), as it allows both efficient training and sampling"
Real NVP is a bipartite flow, while Masked Autoregressive Flow is an autoregressive flow. In special cases, these two flows coincide, but their original implementations are different.
- The paper misses a lot of important details. For instance, a reader needs to figure out what is the objective function. Further, the authors do not mention how they deal with images represented by integers. Do they use dequantization as in other papers (e.g., Theis, L., van den Oord, A., and Bethge, M.  A note on the evaluation of generative models. In Workshop Track,ICLR, 2016)?  These are very important details to fully understand the proposed approach. Providing a very general diagram (Figure 1) and generic equations (2-4) are not sufficient. Currently, there are different implementations of flow components, and describing them would be definitely beneficial. Moreover, it is important to show how conditioning is used in the flow model.
- Section 3.4 is incredibly interesting part of the paper and it should be further analyzed. The proposed two-phase is reasonable, but it leaves an open question why a joint training fails. I agree with the authors that a possible explanation is training instability. Nevertheless, I would be more than interested in seeing a more thorough analysis of this phenomenon.
- I do not understand why the authors skipped reporting bpd for CelebA. 
- In general, the comparison in Table 2 is far from being complete. For instance, please see the following paper:
Ho, J., Chen, X., Srinivas, A., Duan, Y., & Abbeel, P. (2019). Flow++: Improving flow-based generative models with variational dequantization and architecture design. arXiv preprint arXiv:1902.00275.
On Cifar10, current non-autoregressive models are able to achieve 3.08 bpd (Flow++)  and 3.11 bpd (IAF-VAE). This is much better than the reported 3.17 bpd.
- In Section 4.2.2, first line, a number to a figure is missing.
======== AFTER REBUTTAL ========
I would like to thank the authors for their rebuttal. However, I am not fully satisfied with some answers. First of all, the paper should be clearly written so that a reader could be able to find all necessary details and seamlessly implement the presented idea. If some details do not fit the main text, then they should be included in an appendix. Second, the objective function is an important component of any ML problem and, therefore, it should be included in the paper. Sometimes a verbal description is not sufficient. Third, by conditioning a flow I meant whether the based distribution was conditional or/and a single flow layer was conditioned on z. Currently, there are multiple ways to condition and I was curious which of them was used by the authors. Fourth, I do not believe that achieving nicely looking images is the ultimate task for generative models. However, this is an open discussion. Fifth, I agree, the discussion on why the joint training failed is an incredibly interesting question.
Again, I really appreciate all the answers, and I believe that the authors did their best to improve the paper. However, as a reviewer I must look for novelty and evaluate how the paper is readable for others. In my opinion, combining VAE with GLOW is not sufficiently novel idea. If the paper was very clearly written, I would think about rising the paper. However, right now, I decide to keep my original score.