Generative models of natural images have progressed towards high fidelity samples by the strong leveraging of scale. We attempt to carry this success to the field of video modeling by showing that large Generative Adversarial Networks trained on the complex Kinetics-600 dataset are able to produce video samples of substantially higher complexity and fidelity than previous work. Our proposed model, Dual Video Discriminator GAN (DVD-GAN), scales to longer and higher resolution videos by leveraging a computationally efficient decomposition of its discriminator. We evaluate on the related tasks of video synthesis and video prediction, and achieve new state-of-the-art Fréchet Inception Distance for prediction for Kinetics-600, as well as state-of-the-art Inception Score for synthesis on the UCF-101 dataset, alongside establishing a strong baseline for synthesis on Kinetics-600.
Summary:
This paper presents a method for training a generative adversarial network on high resolution videos and complex datasets. They propose decomposing the discriminator in Adversarial Networks into a spatial and temporal discriminator similar to previous works, however, the temporal discriminator downsamples the input using average pooling before forwarding it through the network. In experiments, the presented method outperforms previous state-of-the-art methods in the used metrics. In addition, the videos generated from the Kinetic-600 dataset in a non-conditional setting are the most realistic looking up to date. 
Pros:
+ Best generated video quality out there (In the Kinetics-600 dataset)
+ Numerically outperforms baselines in all datasets
Weaknesses / comments:
* What is the message of the paper?
- It is not clear to me whether this paper is about a new method or if it’s about an empirical study of the network size for video prediction algorithms. If the paper is about a new method, then the novelty is lacking. Decomposing the discriminator into space and time discriminators was originally proposed by MoCoGAN. Therefore, the only difference I can see is that the input to the time discriminator passes through a downsampling function (average pooling). On the other hand, if the paper is about showing that a bigger video model generates better video, the paper is lacking a more in depth evaluation. It seems that the size (frame resolution and number of frames) is only done on the Kinetics-600 dataset (Table 1). However, it is not clear to me whether there is a conclusion to be made from these results about the model size. The FID seems to favor smaller models and IS favors bigger models? But the last (biggest) model does not increase the IS? I am not sure what to make of these results. In addition, there is no such results on the other datasets (UCF101 and BAIR) or on the frame conditioned experiments for Kinetics-600. Do the authors have any comments or clarifications on this?
* Lacking proof that the network is not just overfitting to each class.
- It seems that the authors train separate models for each video class. The generated videos are the best I have seen in such complicated dataset. The authors claim that the fact that the Kinetics-600 dataset is large automatically removes the concern of overfitting. However, I would like to see  evidence that this is true. Can the authors do something like sample a video and find its nearest neighbor in the training data and compare them to see if the model is simply memorizing the training set? Something like this could serve as evidence that the network is not simply memorizing the training data.
* FVD Evaluation on Robot Push (marginal improvement based on FVD paper)
- In Table 3, the authors present quantitative comparisons for frame-conditioned video prediction with state-of-the-art methods. The authors of the FVD paper state the following: 
“In Figure 5 it can be seen that when the difference in FVD is smaller than 50, the agreement with human raters is close to random (but never worse).”
The difference between DVD-GAN and SAVP / Video Transformer is below 50. Does this mean that this result may not significant? It seems that basically the top 3 methods are performing similarly. Can the authors comment on this?
* Comparing 11 frame conditional generation with 16 frame unconditional generation (DVD-GAN)?
- In Table 4, the authors are comparing DVD-GAN-FP, Video Transformer and DVD-GAN. DVD-GAN-FP clearly outperforms Video Transformer. However, the authors make a comment about the DVD-GAN result being much better than the other two and that “the synthesis model’s improved performance on this task seems to indicate that the advantage of being able to select videos to generate is greater than the advantage of having a ground truth distribution of starting frames.”. However, the DVD-GAN model generates 16 frames, and not 11 like the other 2. Is this comparison correct? Also the other 2 methods (DVD-GAN-FP and Video Transformer) are constrained by what they must predict given input frames, does this fact affect the comparison / conclusion by the authors?
* Claim of diversity but no evidence in the video prediction setting (frame conditional generation).
- The authors claim that their method generates diverse videos, however, there is no palpable evidence that this is true. Something like showing examples of different videos generated given the same condition should be good or per-frame evaluations like PSNR and SSIM with error bars of the N number of samples generated given the same condition could also serve as proof that the generation is diverse.
* Video results only provided for Kinetics-600
- It looks like the authors only provide actual video files for the un-conditional generative model for Kinetics-600. Could the authors provide videos of the frame conditioned experiments as well (UCF101, BAIR and Kinetics-600)?
* Argument in paper: “The synthesis model’s improved performance on this task seems to indicate that the advantage of being able to select videos to generate is greater than the advantage of having a ground truth distribution of starting frames”
- I do not fully agree with this argument. Optimally, a frame-conditional model should identify all objects/background observed in the input frame and object/background dynamics if multiple frames are given. Given the optimally identified factors, video generation/prediction should be much better or the same as a purely generative model. The fact that the experiments in this paper show that the conditional model is not as good as the non-conditional model is most likely due to the model and/or objective function and not the generation scenario. Do the authors have any comments on this?
Conclusion:
This paper does indeed present the best video generation up-to-date given the dataset difficulty. However, the paper itself has multiple issues as stated above. Please try to address these in the next revision.