Our work offers a new method for domain translation from semantic label maps and Computer Graphic (CG) simulation edge map images to photo-realistic images. We train a Generative Adversarial Network (GAN) in a conditional way to generate a photo-realistic version of a given CG scene. Existing architectures of GANs still lack the photo-realism capabilities needed to train DNNs for computer vision tasks, we address this issue by embedding edge maps, and training it in an adversarial mode 1. We also offer an extension to our model that uses our GAN architecture to create visually appealing and temporally coherent videos.
SUMMARY: Use HED to convert semantic maps to edge maps, then use GAN to generate realistic images, by adding several losses together. Also add a flow loss while generating videos.
Paper could be written much better, it is slightly confusing to follow.
The effort behind the work is commendable, they used good baselines to compare. However, the idea does not seem different from already existing ones. Although the authors mention this as a CG2real problem, as far as I understand this is a semantic maps to image problem. Maybe I'm missing something.
The only innovation is to have a middle step of converting these semantic maps to edge maps, and then from edge maps to realistic images. This is not altogether a big change, I was hoping to see it goes in a good direction. As far as I understand, the DNED model the authors use is a slight change to the HED model the authors cite, in that they sample different weights to combine to form different outputs.
The results look good from the examples shown. However, there is a lot more work to be done. Since architecturally there is not much difference in the already existing models, maybe focus on more use cases of edge maps. What is the advantage of using edge maps as an intermediate step, as opposed to some other representation? How do we exploit this fact better? These are good questions to explore.
The video results look good, and this is one example of exploiting the fact that edge maps are an intermediate step. However, more work needs to be done to support claims of diversity.
Overall, it seems intuitive to throw in a lot of losses and let the network optimize over all of them in an adversarial way, but a lot more effort needs to be put into thinking about any innovation there.