STOCHASTIC VARIATIONAL VIDEO PREDICTION
Predicting the future in real-world settings, particularly from raw sensory observations such as images, is exceptionally challenging. Real-world events can be stochastic and unpredictable, and the high dimensionality and complexity of natural images require the predictive model to build an intricate understanding of the natural world. Many existing methods tackle this problem by making simplifying assumptions about the environment. One common assumption is that the outcome is deterministic and there is only one plausible future. This can lead to low-quality predictions in real-world settings with stochastic dynamics. In this paper, we develop a stochastic variational video prediction (SV2P) method that predicts a different possible future for each sample of its latent variables. To the best of our knowledge, our model is the first to provide effective stochastic multi-frame prediction for real-world videos. We demonstrate the capability of the proposed method in predicting detailed future frames of videos on multiple real-world datasets, both action-free and action-conditioned. We find that our proposed method produces substantially improved video predictions when compared to the same model without stochasticity, and to other stochastic video prediction methods. Our SV2P implementation will be open sourced upon publication.
1) Summary
This paper proposed a new method for predicting multiple future frames in videos. A new formulation is proposed where the frames’ inherent noise is modeled separate from the uncertainty of the future. This separation allows for directly modeling the stochasticity in the sequence through a random variable z ~ p(z)  where the posterior  q(z | past and future frames) is approximated by a neural network, and as a result, sampling of a random future is possible through sampling from the prior p(z) during testing. The random variable z can be modeled in a time-variant and time-invariant way. Additionally, this paper proposes a training procedure to prevent their method from ignoring the stochastic phenomena modeled by z. In the experimental section, the authors highlight the advantages of their method in 1) a synthetic dataset of shapes meant to clearly show the stochasticity in the prediction, 2) two robotic arm datasets for video prediction given and not given actions, and 3) A challenging human action dataset in which they perform future prediction only given previous frames.
2) Pros:
+ Novel/Sound future frame prediction formulation and training for modeling the stochasticity of future prediction.
+ Experiments on the synthetic shapes and robotic arm datasets highlight the proposed method’s power of multiple future frame prediction possible.
+ Good analysis on the number of samples improving the chance of outputting the correct future, the modeling power of the posterior for reconstructing the future, and a wide variety of qualitative examples.
+ Work is significant for the problem of modeling the stochastic nature of future frame prediction in videos.

3) Cons:
Approximate posterior in non-synthetic datasets:
The variable z seems to not be modeling the future very well. In the robot arm qualitative experiments, the robot motion is well modeled, however, the background is not. Given that for the approximate posterior computation the entire sequence is given (e.g. reconstruction is performed), I would expect the background motion to also be modeled well. This issue is more evident in the Human 3.6M experiments, as it seems to output blurriness regardless of the true future being observed. This problem may mean the method is failing to model a large variety of objects and clearly works for the robotic arm because a very similar large shape (e.g. robot arm) is seen in the training data. Do you have any comments on this?
Finn et al 2016 PNSR performance on Human 3.6M:
Is the same exact data, pre-processing, training, and architecture being utilized? In her paper, the PSNR for the first timestep on Human 3.6M is about 41 (maybe 42?)  while in this paper it is 38.
Additional evaluation on Human 3.6M:
PSNR is not a good evaluation metric for frame prediction as it is biased towards blurriness, and also SSIM does not give us an objective evaluation in the sense of semantic quality of predicted frames. It would be good if the authors present additional quantitative evaluation to show that the predicted frames contain useful semantic information [1, 2, 3, 4]. For example, evaluating the predicted frames for the Human 3.6M dataset to see if the human is still detectable in the image or if the expected action is being predicted could be useful to verify that the predicted frames contain the expected meaningful information compared to the baselines.
Additional comments:
Are all 15 actions being used for the Human 3.6M experiments? If so, the fact of the time-invariant model performs better than the time-variant one may not be the consistent action being performed (last sentence of 5.2). The motion performed by the actors in each action highly overlaps (talking on the phone action may go from sitting to walking a little to sitting again, and so on). Unless actions such as walking and discussion were only used, it is unlikely the time-invariant z is performing better because of consistent action. Do you have any comments on this?
4) Conclusion
This paper proposes an interesting novel approach for predicting multiple futures in videos, however, the results are not fully convincing in all datasets. If the authors can provide additional quantitative evaluation besides PSNR and SSIM (e.g. evaluation on semantic quality), and also address the comments above, the current score will improve.
References:
[1] Emily Denton and Vighnesh Birodkar. Unsupervised Learning of Disentangled Representations from Video. In NIPS, 2017.
[2] Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryull Sohn, Xunyu Lin, and Honglak Lee. Learning to generate long-term future via hierarchical prediction. In ICML, 2017.
[3] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive Growing of GANs for Improved Quality, Stability, and Variation. arXiv preprint arXiv:1710.10196, 2017.
[4] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved Techniques for Training GANs. In NIPS, 2017.
Revised review:
Given the authors' thorough answers to my concerns, I have decided to change my score. I would like to thank the authors for a very nice paper that will definitely help the community towards developing better video prediction algorithms that can now predict multiple futures.