Being able to predict what may happen in the future requires an in-depth understanding of the physical and causal rules that govern the world. A model that is able to do so has a number of appealing applications, from robotic planning to representation learning. However, learning to predict raw future observations, such as frames in a video, is exceedingly challenging-the ambiguous nature of the problem can cause a naively designed model to average together possible futures into a single, blurry prediction. Recently, this has been addressed by two distinct approaches: (a) latent variational variable models that explicitly model underlying stochasticity and (b) adversarially-trained models that aim to produce naturalistic images. However, a standard latent variable model can struggle to produce realistic results, and a standard adversarially-trained model underutilizes latent variables and fails to produce diverse predictions. We show that these distinct methods are in fact complementary. Combining the two produces predictions that look more realistic to human raters and better cover the range of possible futures. Our method outperforms prior works in these aspects.
The paper introduces a generative model for video prediction. The originality stems from a new training criterion which combines a VAE and a GAN criteria. At training time, the GAN and the VAE are trained simultaneously with a shared generator; at test time, prediction conditioned on initial frames is performed by sampling from a latent distribution and generating the next frames via an enhanced conv LST . Evaluations are performed on two movement video datasets classically used for benchmarking  this task - several quantitative evaluation criteria are considered.
The paper clearly states the objective and provides a nice general description of the method.  The proposed model extends previous work by adding an adversarial loss to a VAE video prediction model.  The evaluation compares different variants of this model to two recent VAE baselines. A special emphasis is put on the quantitative evaluation: several criteria are introduced for characterizing different properties of the models with a focus on diversity. w.r.t. the baselines, the model behaves well for the “realistic” and “diversity” measures. The results are more mitigated for measures of accuracy. As for the qualitative evaluation, the model corrects the blurring effect of the reference SV2P baseline, and produces quite realistic predictions on these datasets. The difference with the other reference model (SVG) is less clear.
While the general description of the model is clear, details are lacking. It would probably help to position the VAE component more precisely w.r.t. one of the two baselines, by indicating the differences. This would also help to explain the difference of performance/ behavior  w.r.t. these models (Fig. 5).
It seems that the discriminator takes a whole sequence as input, but some precision on how this done could be added.  Similarly, you did not indicate what the deterministic version of your model is.
The generator model with its warping component makes a strong hypothesis on the nature of the videos: it seems especially well suited for translations or for other simple geometric transformations characteristics of the benchmarking videos .  Could you comment on the importance of this component? Did you test the model on other types of videos where this hypothesis is less relevant? It seems that the baseline SVG makes use of simpler ConLSTM for example.
The description of the generator in the appendix is difficult to follow. I missed the point in the following sentence: “For each one-step prediction, the network has the freedom to choose to copy pixels from the previous frame, used transformed versions of the previous frame, or to synthesize pixels from scratch” .
Also, it is not clear from the discussion on z, whether sampling is performed once for each video of for each frame.
Overall, the paper proposes an extension of VAE based video prediction models and produces an extensive evaluation. While the model seems to perform well, the originality and the improvement w.r.t. baselines are somewhat limited.