Video-to-Video Synthesis
We study the problem of video-to-video synthesis, whose goal is to learn a mapping function from an input source video (e.g., a sequence of semantic segmentation masks) to an output photorealistic video that precisely depicts the content of the source video. While its image counterpart, the image-to-image translation problem, is a popular topic, the video-to-video synthesis problem is less explored in the literature. Without modeling temporal dynamics, directly applying existing image synthesis approaches to an input video often results in temporally incoherent videos of low visual quality. In this paper, we propose a video-to-video synthesis approach under the generative adversarial learning framework. Through carefully-designed generators and discriminators, coupled with a spatio-temporal adversarial objective, we achieve high-resolution, photorealistic, temporally coherent video results on a diverse set of input formats including segmentation masks, sketches, and poses. Experiments on multiple benchmarks show the advantage of our method compared to strong baselines. In particular, our model is capable of synthesizing 2K resolution videos of street scenes up to 30 seconds long, which significantly advances the state-of-the-art of video synthesis. Finally, we apply our method to future video prediction, outperforming several competing systems. Code, models, and more results are available at our website.
This paper focuses on video-2-video synthesis, i.e. given a real video the goal is to learn a model that outputs a new photorealistic and temporally consistent video with (ideally) the same data distribution, preserving the content and style of the source video. Existing image-2-image methods produce photorealistic images, but they do not account for the temporal dimension, resulting in high-frequency artifacts across time. This work builds on existing image-2-image works and mainly extends them into the temporal dimension to ensure temporal coherence. By employing conditional GANs the method provides high-level control over the output, e.g. certain parts of the dynamic scene can be introduced or replaced though segmentation masks. Although the theoretical background and components are employed from past work, there is significant amount of effort in putting them together and adding the temporal extension. Although this work does not solve the problem per se, it seems like an important step for which the community would be interested in, while the authors also explicitly promise to release their implementation. The results look reasonable, though I would like to see a more involved discussion about current disadvantages and future directions. Furthermore, some clarifications are needed and suggestions for further experiments are given. Although I am not an expert in this narrow field, I am leaning towards supporting this work, on the condition that feedback will be incorporated. More detailed feedback follows below.   - L10, L38 - I am not very convinced about the novelty of the 'new generator and discriminator architectures', in the technical section they are discussed as straightforward extensions.  - L167-168 - My understanding is that this is not what you do. So I am not sure about the use of this comment in the manuscript. Can you please comment further? What is used for the provided results in the manuscript and supplementary material? The suggested speedup trick would force hallucination for the foreground areas, which is in contrast with the goal of this work as explicitly mentioned in L3 (and elsewhere), temporal consistency and content preservation would be harder for the foreground. If, in contrast to my understanding, this is what you do, can you please be more specific about the trade-offs?  - L119 - The phrase 'between 0 and 1' sounds a bit ambiguous, the values are binary.  - Eq4 - Maybe you can help the reader with more intuition, e.g. the first part corresponds to non-occluded regions (synthesising by warping) while the second corresponds to occluded regions (synthesising by hallucinating).  - Eq7 - In the first term, shouldn't the superscript for w be i-2 ? - Eq8, L153 - It is counter-intuitive to see the two lambdas to have the same value. Maybe you can combine them in one lambda and move it from Eq8 to Eq5 ? - The lambda(s) in Eq8(Eq5) is an important steering weight for the training objective, how do you set this? I guess it is too expensive to provide evaluation, so some intuition would be helpful. - L159 - Please compare with L160, it seems there is a \widetilde{x} (etc) missing in H_F. - L192, L193 - I would love to hear some more specific details in the supplementary material, for better self-consistency. Why is this mentioned here and not after Eq5? - Nice idea to include videos in the pdf. There is a clear improvement in the qualitative results, however I would like to hear some discussion about current disadvantages and pointers for future directions. In the long videos of the supplementary material replacing objects through the segmentation masks is interesting but has obvious artifacts (buildings, trees), any pointers for improvements?  - L108 - Can you please discuss how you set L=2? It would be great if you could provide at least qualitative comparison for a short video that gives intuition to the reader. Furthermore, could you please add a comment on what is the price to pay for a bigger L in terms of modelling/training difficulty? - The method employs optical flow estimation and occlusion mask estimation (L123), and both are sources of noise. It would be great if you could replace estimation of them with ground truth on some synthetic dataset, this could give pointers and prioritisation on what would improve your results (e.g. parts of lanes disappear/blur after some frames).   - minor typos:    - L120 - 'of the image'    - L274 - 'allows the user'    - L275 - 'synthesise a new' - Thank you for explicitly promising to release code/models, reproducibility could be hard without this.   ==================================    Updates after authors' response:  - I welcome the authors' responce and clarifications. My new comments are again towards constructive refining of the manuscript. Some additions are asked below, in order to incorporate the responce in the manuscript in case of acceptance. I understand that there are space limitations, so the authors should do their best to communicate the points in the main paper even more briefly than in the rebuttal, but worst case they should be in the supplementary material and video.  - "Using ground truth flows" - Thank you for performing the requested experiment. Please incorporate in the paper and in the supplementary video. - "Limitations and future work" - Thank you for the discussion. I often feel that in certain communities this is underplayed, while it points the community to interesting steps. Again, please incorporate in the paper and in the supplementary video. For point 3 I am not fully convinced about the coarser semantic labels (training less affected by the label shapes), but maybe I am missing something. - Thank you for offering to downplay the "new" term and novelty of architecture. I do agree that the extension is not trivial. I am not sure about the comment on combining different resolution, if I remember correctly this is not tackled.  - "Synthesis in the foreground area" - Please add in the paper the comment about flickering and "visual quality". I am not sure what you mean with the term "visual quality", so please reword to communicate it better. The flag in the code is a great idea. Please add a qualitative comparison in the video, it would be very informative wrt the trade-off between image quality and decision on FG synthesis.  - "Setting parameters and losses" - Please add very brief comment on similarity for 1..10.  - L=2 is chosen but L=4 (more temporal info/frames) gives worse performance. This shows that the temporal extension is not trivial, and points that more future work is needed to cleverly use temporal information. This should be part of "future work" and must be mentioned. This is not to downplay this work, just to underline that the problem is non-trivial. Please mind that the resolution due to time constraints was decreased for this experiment.