Unsupervised image-to-image translation is a recently proposed task of translating an image to a different style or domain given only unpaired image examples at training time. In this paper, we formulate a new task of unsupervised video-tovideo translation, which poses its own unique challenges. Translating video implies learning not only the appearance of objects and scenes but also realistic motion and transitions between consecutive frames. We investigate the performance of per-frame video-to-video translation using existing image-to-image translation networks, and propose a spatio-temporal 3D translator as an alternative solution to this problem. We evaluate our 3D method on multiple synthetic datasets, such as moving colorized digits, as well as the realistic segmentation-to-video GTA dataset and a new CT-to-MRI volumetric images translation dataset. Our results show that frame-wise translation produces realistic results on a single frame level but underperforms significantly on the scale of the whole video compared to our three-dimensional translation approach, which is better able to learn the complex structure of video and motion and continuity of object appearance.
1) Summary
This paper proposes a 3D convolutional neural network based architecture for video-to-video translation. The method mitigates the inconsistency problem present when image-to-image translation methods are used in the video domain. Additionally, they present a study of ways to better setting up batched for the learning steps during networks optimization for videos, and also, they propose a new MRI-to-CT dataset for medical volumetric image translation. The proposed method outperforms the image-to-image translation methods in most measures.
2) Pros:
+ Proposed network architecture mitigates the pixel color discontinuity issues present in image-to-image translation methods.
+ Proposed a new MRI-to-CT dataset that could be useful for the community to have a benchmark on medical related research papers.
3) Cons:
Limited network architecture:
- The proposed neural network architecture is limited to only generate the number of frames it was trained to generate. Usually, in video generation / translation / prediction we want to be able to produce any length of video. I acknowledge that the network can be re-used to continue generating number of frames that are multiples of what the network was trained to generate, but the authors have not shown this in the provided videos. I would be good if they can provide evidence that this can be done with the proposed network.
Short videos:
- Another limitation that is related to the previously mentioned issue is that the videos are short, which in video-to-video translation, it should not be difficult to generate longer videos. It is hard to conclude that the proposed method will work for large videos from the provided evidence.
Lack of baselines:
- A paper from NVIDIA Research on video-to-video synthesis [1] (including the code)  came out about a month before the ICLR deadline. It would be good if the authors can include comparison with this method in the paper revision. Other papers such as [2, 3] on image-to-image translation are available for comparison. The authors simply say such methods do not work, but show no evidence in the experimental section. I peeked at some of the results in the papers corresponding websites, and the videos look consistent through time. Can the authors comment on this if I am missing something?
Additional comments:
The authors mention in the conclusion that this paper proposes “a new computer vision task or video-to-video translation, as well as, datasets, metrics and multiple baselines”. I am not sure that video-to-video translation is new, as it has been done by the papers I mention above. Maybe I am misunderstanding the statement? If so, please clarify. Additionally, I am not sure how the metrics are new. Human evaluation has been done before, the video colorization evaluation may be somewhat new, but I do not think it will generalize to tasks other than colorization. Again, If I am misunderstanding the statement, please let me know in the rebuttal.
4) Conclusion
The problem tackled is a difficult one, but other papers that are not included in experiments have been tested on this task. The proposed dataset can be of great value to the community, and is a clearly important piece of this paper. I am willing to change the current score if they authors are able to address the issues mentioned above.
References:
[1] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. "Video-to-Video Synthesis". In NIPS, 2018.
[2] Xun Huang, Ming-Yu Liu, Serge Belongie, Jan Kautz. Multimodal Unsupervised Image-to-Image Translation. In ECCV, 2018.
[3] Ming-Yu Liu, Thomas Breuel, Jan Kautz. Unsupervised Image-to-Image Translation Networks. In NIPS, 2017.