We focus on temporal self-supervision for GAN-based video generation tasks. While adversarial training successfully yields generative models for a variety of areas, temporal relationship in the generated data is much less explored. This is crucial for sequential generation tasks , e.g. video super-resolution and unpaired video translation. For the former, state-of-the-art methods often favor simpler norm losses such as L 2 over adversarial training. However, their averaging nature easily leads to temporally smooth results with an undesirable lack of spatial detail. For unpaired video translation, existing approaches modify the generator networks to form spatio-temporal cycle consistencies. In contrast, we focus on improving the learning objectives, and propose a temporally self-supervised algorithm. For both tasks, we show that temporal adversarial learning is key to achieving temporally coherent solutions without sacrificing spatial detail. We also propose a novel Ping-Pong loss to improve the long-term temporal consistency. It effectively prevents recurrent networks from accumulating artifacts temporally without depressing detailed features. We also propose a first set of metrics to quantitatively evaluate the accuracy as well as the perceptual quality of the temporal evolution. A series of user studies confirms the rankings computed with these metrics.
Summary:
This paper proposes a training objective for higher quality video generation for the tasks of Video Super Resolution (VSR) and Unpaired Video Translation (UVT) and also two evaluation metrics tOF and tLP. They provide a comprehensive ablative study of the proposed method and also show comparisons against baselines for the tasks of VSR and UVT.
Pros:
+ Novel video generation method for VSR and UVT.
+ Novel metrics
+ Well written paper
Weaknesses / comments:
- Confused about inputs to discriminator (I_{w,g} and I_{w,b})
I am not sure I understand the inputs I_{w,g} and I_{w,b} to the discriminator network. Based on the description, I_{w,g} = {W(g_{t-1}, v_t), g_t, W(g_{t+1}, v_t^’)} = {g_t, g_t, g_t}, assuming v_t^” means reverse flow. A similar process seems to be happening with I_{w,b}. If this is the case, will the discriminator optimally get the same frame concatenated together (i.e. {g_t, g_t, g_t})? Now, if that’s the case, how is the discriminator modeling temporal consistency other than making the generator learn to put pixels forward and back in place since the “original triplets” (real data) are {g_{t-1}, g_t, g_{t+1}} and {b_{t-1}, g_t, g_{t+1}}, respectively. This is very confusing to me. It would be good if the authors can clarify this in the rebuttal.
- How do you prevent the zero output in PP loss?
The PP-loss compares forward and backward prediction by || g_t - g_t^’ ||. If not careful, the neural network can just learn to output zeros or the same frame for the full video. Did the authors see any behavior like this? Or did the proposed formulations prevent this? Or was there a very small weight applied to this loss?
- TecoGAN vs baselines generator parameters (rather than TecoGAN^{-}).
Based on the experimental section, it looks like TecoGAN is the main network being compared to the baseline methods. TecoGAN has more parameters in the generator compared to TecoGAN^{-}. Did the authors make sure that the generator had the same number of parameters as the other methods? The authors mention a performance difference between TecoGAN and DUF due to DUF having more parameters, but what about the other baselines?
- Evaluation metric contributions in the supplementary material?
The description of the two proposed metrics tOF and tLP have been placed in Appendix B. These are mentioned as contributions of the paper so their description should be in the main text. Please make sure that Appendix B is in the main text in future versions of this paper.
- UVT task evaluation in the supplementary material?
The UVT task is mentioned in the abstract and as a target task in this work, however, the evaluations for this are in the Appendix. Please move them to the main text in future versions of this paper. Secondary experiments should be in the Appendix but I feel this is primary.
- UVT task only evaluated with the proposed evaluations?
The UVT evaluation in the Appendix only has the proposed metrics as evaluation. I understand that, since it’s an unpaired video translation task, there is no ground truth to compare against. However, a human based study could be done where human raters would judge for the more realistic of N videos.
- The same qualitative comparisons in the paper are not in the provided website.
The provided website does not have the same qualitative evaluations as the paper does. For example: There are 7 methods being compared for the same video in Figure 8 in the supplementary material, but I don’t see anything like that in the website.
- For the UVT task, we omit the DsDtPP model because …….
In the paper, the authors mention that they don’t provide the DsDtPP baselines because it requires a lot of computation. However, I feel these missing baselines make the experiments incomplete since it’s a different task.
- Figure 5 has no point of reference with which to compare the frames.
Figure 5 points out that the video b) is better than video a). While it is true that one seems more noisy than the other, there should be a point of reference for readers to make sure of it.
Conclusion:
In conclusion, the paper seems to present a novel method and evaluation metrics but has many issues as stated above. It would make the submission better if the authors can address them in the rebuttal.