Generative models that can model and predict sequences of future events can, in principle, learn to capture complex real-world phenomena, such as physical interactions. However, a central challenge in video prediction is that the future is highly uncertain: a sequence of past observations of events can imply many possible futures. Although a number of recent works have studied probabilistic models that can represent uncertain futures, such models are either extremely expensive computationally as in the case of pixel-level autoregressive models, or do not directly optimize the likelihood of the data. To our knowledge, our work is the first to propose multi-frame video prediction with normalizing flows, which allows for direct optimization of the data likelihood, and produces high-quality stochastic predictions. We describe an approach for modeling the latent space dynamics, and demonstrate that flow-based generative models offer a viable and competitive approach to generative modeling of video.
This paper presents a stochastic model based on Glow for conditional video generation. The major novelty of this work is to introduce the flow-based models to video modeling and learn the video dynamics via the dependencies of the latent variables. The general idea is reasonable and the proposed model is technically correct, but I have the following concerns mainly about the originality and the experiments.  
**Above all, most of the text in Section 3 is very similar (or exactly the same) to the text of the Glow paper (the background section).**
— Significance and originality —
a.1) In Section 1, the authors discussed some possible application scenarios of video prediction models, e.g. learning from unlabeled data and being used for downstream tasks. However, all models mentioned here, including [Mathieu et al. 2016] and [Finn et al. 2016], are deterministic models. Thus, in what way can the stochastic model proposed in this paper be used in real applications?
a.2) VideoFlow can be viewed as an extension of the Glow model. There are two problems. First, the originality is limited. I don’t think modeling the temporal dependencies of the latent variables with a convolutional network is a significant contribution to the conditional flow-based methods. Second, this paper is not self-contained. After reading Section 4.2, I have to check the previous literature to find the objective function, the network details, or the training procedure.
— Experiments —
b.1) Throughout the experiments, the VideoFlow model is mainly compared with two stochastic video prediction models that were probably proposed by the same research group. If it is possible, the authors might include other stochastic models such as the SVG-LP [Denton & Fergus 2018], and at least one deterministic model such as the E3D-LSTM [Wang et al. 2019] as well.
[Wang et al. 2019] E3D-LSTM: A Model for Video Prediction and Beyond.
b.2) The evaluation metric bits-per-pixel was not directly optimized by the previous video generation/prediction models. Thus, the comparisons in Table 2 might be unfair.
b.3) Since training the Glow model requires a huge computational cost, how is the training efficiency of the VideoFlow model compared with other stochastic video generation models?
— Other —
c.1) In Section 4, it is not clear what the temporal border effect means?
AFTER REBUTTAL:
Though the overall novelty is still not fully convincing, this paper may shed some insights into video generation by introducing flow-based models to this topic. I have increased my score from 3 to 6.