In this work we propose Generative Latent Flow (GLF), an algorithm for generative modeling of data distributions. GLF uses an auto-encoder to learn latent representations of the data, and a normalizing flow to map the distribution of the latent variables to that of a standard Gaussian. GLF can be seen as a variational auto-encoder, with normalizing flow prior, and a vanishing limit of the pixel-wise variance of the data. We carefully study this relationship and the pros and cons of using an auto-encoder vs. a variational auto-encoder. In contrast to a number of other auto-encoder based generative models, which use various regularizers to encourage the encoded latent distribution to match the prior distribution, our model explicitly constructs a mapping between these two distributions, leading to better density matching while avoiding over regularizing the latent variables. We compare our model with several related techniques, and show that under standard quantitative evaluations, our method achieves state-of-the-art sample quality and diversity among AE based models on commonly used datasets and is competitive with GANs' benchmarks.
The paper proposes a deterministic Auto-Encoder with a trained marginal distribution over latent variables, p(z), to be able to sample from the model. For this purpose, the authors propose to use a flow-based model for p(z), and regularize the AE objective (i.e., MSE) with a cross-entropy between q(z) = 1/N \sum_n E(x_n) and p(z). In general, I find the idea quite interesting. The construction of the objective and motivation behind is rather clear. The experiments are rather convincing (however, the FID metric is subjective, but it's impossible to calculate the log-likelihood scores). The main disadvantage of the paper is its language. There are many typos and difficult to follow sentences. But besides that, the main ideas are well explained. Please find more specific remarks below. 
Remarks
- The language in the paper is a bit off. There are sentences that sound very peculiar, e.g., "Deep generative models can be roughly classified into explicit and implicit models. The former class assumes explicit parametric specification of the distribution, whereas the latter does not." 
The introduction should be re-written. Similarly, Section 2 is hard to follow.
There are places where a word or a punctuation mark is missing, for instance:
* The first line misses a word: "(...) on deep learning."
* First paragraph, Section 2: "This distribution is unknown and possibly The model also has a predefined prior distribution p(z) on Z."
* Section 3.2, below Eq. 2: "(...) assumptio (...)".
* Section 3.4, last paragraph: "Our work is differs in two ways (...)".
- The authors should refer to the following very relevant paper:
* Xu, H., Chen, W., Lai, J., Li, Z., Zhao, Y., & Pei, D. (2019). On the Necessity and Effectiveness of Learning the Prior of Variational Auto-Encoder. arXiv preprint arXiv:1905.13452.
- It is unclear how the authors dequantize image data that are typically represent as integers from 0 to 255 (or 0 and 1 in the binary case). The authors mention in Section 3.2 that they use the MSE loss for \mathcal{L}_{recon}. This corresponds to taking a Gaussian decoder in the VAE framework. However, the manner how the images are dequantized is extremely important to properly evaluate all results. For instance, if the authors add a uniform noise, then it highly influences the final quality of a model.
- I do not fully follow why the authors call the objective part for learning the marginal distribution over latents "the NLL loss". It is rather the cross entropy between q(z) = 1/N \sum_n E(x_n) and p(z). This follows from the notation and the description that we stop the gradient for E(.). I find it confusing.
- Figure 4 is extremely important for understanding why we should stop gradient. However, jumping between Section 4.2 and E is extremely annoying. The authors can use up to 10 pages, so adding a half of a page would not make a difference, but it would help a reader a lot.
- The main difference between the objective in Eq. 2 and the ELBO lies in considering a deterministic encoder and entails skipping the entropy term for the variational posterior. Could the authors comment why this is so important to choose a deterministic encoder? I can easily imagine taking q(z|x) = Normal(z | mu(z), a1), i.e., fixing the variance to some value a (e.g., a=1), that would result in Entropy[q] = const. Hence, we can skip entropy from the objective, but still we use a stochastic encoder. 
===== AFTER REBUTTAL =====
I would like to thank the authors for their response and new version of the paper. I must admit that I had a very hard nut to crack. From one side, I really appreciate all the effort the authors put to improve the paper. However, on the other hand, I tend to agree with the reviewer #3 that the paper is interesting from the engineering perspective and it lacks novelty. Eventually, I decided to sustain my score. There are two reasons for that. First, more analysis of the model would be helpful. For instance, analyzing the latent space would be interesting (a suggestion from the reviewer #3). Second, considering a pure AE with a trainable flow-based prior is interesting, but it is also very limiting in the sense of obtaining nicely looking pictures instead of being able to provide (approximate) probability of an image. In my opinion, from the decision making perspective, having a probability of an object is much more important  than being able to generate a crisp image.