Temporal Coherency based Criteria for Predicting Video Frames using Deep Multi-stage Generative Adversarial Networks
Predicting the future from a sequence of video frames has been recently a sought after yet challenging task in the field of computer vision and machine learning. Although there have been efforts for tracking using motion trajectories and flow features, the complex problem of generating unseen frames has not been studied extensively. In this paper, we deal with this problem using convolutional models within a multi-stage Generative Adversarial Networks (GAN) framework. The proposed method uses two stages of GANs to generate crisp and clear set of future frames. Although GANs have been used in the past for predicting the future, none of the works consider the relation between subsequent frames in the temporal dimension. Our main contribution lies in formulating two objective functions based on the Normalized Cross Correlation (NCC) and the Pairwise Contrastive Divergence (PCD) for solving this problem. This method, coupled with the traditional L1 loss, has been experimented with three real-world video datasets viz. Sports-1M, UCF-101 and the KITTI. Performance analysis reveals superior results over the recent state-of-the-art methods.
This method provides 2 contributions for next frame prediction from video sequences. 
The first is the introduction of a normalized cross correlation loss, which provide a better similarity score to judge if the predicted frame is close to the true future. The second is the pairwise contrastive divergence loss, based on the idea of similarity of the image features. Results are presented on the UCF101 and Kitti datasets, and a numerical comparison using image similarity metrics (PSNR, SSIM) with Mathieu et al ICLR16 is performed. 
Comments:
The newly proposed losses are interesting, but I suspect a problem in the evaluation.
I fear that the evaluation protocol comparing the present approach to previous work [13] is different to [13]: In the table, the authors don't mention evaluating their method on 10% of the test set and only in the moving area as Mathieu et al. did.
Did the authors performed the sanity check of comparing the obtained PSNR/SSIM score of frame copy to see if they match the ones reported in [13]?
Please be more specific about the details of the evaluation (full test set? Full images? -> in this case, what are the scores for a frame copy baseline?)
The authors answered my concerns in the rebuttal.
- Providing generated videos for such paper submission would be appreciated.
In the displayed examples, including the ones from the supplementary material, the movements are barely visible. 
- Predicting several frames at a time versus one?
In Mathieu et al. work, results showed that predicting only one frame at the time worked better. Have you tried such experiment?
l.138: I don't understand the h+4?
Minor
The first sentence, "Video frame prediction has always been one of the fundamental problems in computer vision as it caters to a wide range of applications [...]" is somehow controversial in my opinion: The task was, in the best of my knowledge, introduced only in 2014. As for the applications, even if i believe they will exist soon, I am not aware of systems using next frames predictions yet.
l. 61 uses -> use
l. 65 statc -> statistical ?
l 85. [5]is -> [5] is
Eq 4: use subscripts instead of exponents for indexes
l 119 (b)P -> (b) P
l 123-125 weird sentence
l. 142 ,u -> , u
l. 153 tries to -> is trained to
l. 163 that, -> that
l. 164 where, -> where   
[13] ICLR 16
[17] ICLR 16
please check other references