Unsupervised Image-to-Image Translation Networks
Unsupervised image-to-image translation aims at learning a joint distribution of images in different domains by using images from the marginal distributions in individual domains. Since there exists an infinite set of joint distributions that can arrive the given marginal distributions, one could infer nothing about the joint distribution from the marginal distributions without additional assumptions. To address the problem, we make a shared-latent space assumption and propose an unsupervised image-to-image translation framework based on Coupled GANs. We compare the proposed framework with competing approaches and present high quality image translation results on various challenging unsupervised image translation tasks, including street scene image translation, animal image translation, and face image translation. We also apply the proposed framework to domain adaptation and achieve state-of-the-art performance on benchmark datasets. Code and additional results are available in https://github.com/mingyuliutw/unit.
This paper investigated the problem of unsupervised image-to-image translation (UNIT) using deep generative models. In contrast to existing supervised methods, the proposed approach is able to learn domain-level translation without paired images from two domains. Instead, the paper proposed to learn a joint distribution (two domains) from a single shared latent variable using generative adversarial networks. Additional encoder networks are used to infer the latent code from each domain. The paper proposed to jointly train the encoder, decoder, and discriminator in an end-to-end style without paired images as supervision. Qualitative results on several image datasets demonstrate good performance of the proposed UNIT pipeline.
== Qualitative Assessment ==
The unsupervised domain transfer is a challenging task in deep representation learning and the proposed UNIT network demonstrated a promising direction based on generative models. In terms of novelty, the UNIT network can be treated as an extension of coGAN paper (reference 17 in the paper) with joint training objectives using additional encoder networks.
Overall, the proposed method is sound with sufficient details including both qualitative and quantitative analysis. I like Table 1 that interpolates and compares the existing work as subnetworks in UNIT. 
Considering the fact that UNIT extends coGAN paper, I would like to see more discussions and explicitly comparisons between coGAN and UNIT in the paper. Performance-wise, there seems to be slight improvement compared to coGAN (Table 3 in the paper). However, the evaluation is done on MNIST dataset which may not be conclusive at all. Also, I wonder whether images in Figure 8 are cherry-picked? Compared to coGAN figures, the examples in Figure 8 have less visual artifacts. I encourage authors to elaborate this a little bit.
* What’s the benefit of having an encoder network for image-to-image translation?
Image-to-image translation can also be applied to coGAN paper: (1) analysis-by-synthesis optimization [1] & [2]; or (2) train a separate inference network [3] & [4]. Can you possibly comment on the visual quality by applying optimization-based method to a pre-trained coGAN? I encourage authors to elaborate this a bit and provide side-by-side comparisons in the final version of the paper.
* What’s the benefit of joint training?
As being said, one can train a separate encoder/inference network. It is not crystal clear to me why VAE type of training is beneficial. Maybe the reconstruction loss can help to stabilize the training a little bit but I don’t see much help when you add KL penalty to the loss function. Please elaborate this a little bit.
Additional comments:
* Consider style-transfer as alternative method in unsupervised image-to-image translation?
Given the proposed weight sharing architecture, I would assume the translated image at least preserves the structure of input image (they are aligned to some extent). If so, style-transfer methods (either optimization based or feed-forward approximation) can serve as baseline in most experiments.
* Just for curiosity, other than visual examination, what is the metric used when selecting models for presentation?
Though the hyperparameters have little impact as described in ablation study, I wonder how did you find the good architecture for presentation? At least quantitative evaluations can be used as metric In MNIST experiment for model selection. Since you obtained good results in most experiments, I wonder if this can be elaborated a bit.
[1] Generative Visual Manipulation on the Natural Image Manifold, Zhu et al. In ECCV 2016;
[2] Reference 29 in the paper;
[3] Picture: A Probabilistic Programming Language for Scene Perception, Kulkarni et al. In CVPR 2015;
[4] Generative Adversarial Text-to-Image Synthesis, Reed et al. In ICML 2016;