Unsupervised Image-to-Image Translation Using Domain-Specific Variational Information Bound
Unsupervised image-to-image translation is a class of computer vision problems which aims at modeling conditional distribution of images in the target domain, given a set of unpaired images in the source and target domains. An image in the source domain might have multiple representations in the target domain. Therefore, ambiguity in modeling of the conditional distribution arises, specially when the images in the source and target domains come from different modalities. Current approaches mostly rely on simplifying assumptions to map both domains into a shared-latent space. Consequently, they are only able to model the domain-invariant information between the two modalities. These approaches usually fail to model domain-specific information which has no representation in the target domain. In this work, we propose an unsupervised image-to-image translation framework which maximizes a domain-specific variational information bound and learns the target domain-invariant representation of the two domain. The proposed framework makes it possible to map a single source image into multiple images in the target domain, utilizing several target domain-specific codes sampled randomly from the prior distribution, or extracted from reference images.
Summary:  The main contribution of this paper is a new framework for unsupervised image-to-image translation. Specifically, instead of learning only the domain-invariant information between two domains, the proposed approach extends CycleGAN to learn a domain-specific code for each modality and a domain-invariant code shared by the two modalities. The domain-specific code is learned via a domain-specific variational information bound. The trained model can be used for unsupervised one-to-many image-to-image translation.  Strengths:  (1) Significance: Though unsupervised image-to-image translation has been studied before, this paper pointed out one common drawback of such methods, i.e., no domain-specific code is learned for each individual modality. I think this is a good point, and the proposed method nicely solves this problem, and thus has some significance. The topic is also suitable for general NIPS readers.   (2) Originality: The proposed model is interesting and seems novel. I like the idea of decomposing the latent code into the domain-invariant and domain-specific parts. The idea of using a domain-specific variational information bound to enforce the model disentangle the latent code also seems interesting.   Weaknesses:  (1) Clarity: I like the first three sections, which introduces the problem and proposed approaches clearly. However, Section 4 is generally not well written, and not easy to follow. Please see the Questions section for more details.   For presentation, I think the introduction part talks about a little bit too many details. For example, it is not clear to me why putting Figure 1 & 2 in Section 1, and it seems to me moving these two figures to the experimental section and discussing the figures there might be better.  Also, in Table 1, the authors showed the human evaluation results. However, how this human evaluation is performed is not discussed. I would recommend describing this in the main text.   (2) Quality: I think the idea is novel, but the authors did not introduce the proposed variational information bound in Section 4 clearly, which seems to be a key contribution of this paper. Also, the authors did not provide enough experimental details for other people to reproduce the experimental results. Specifically, I have the following questions.      Questions:  (1) It is not clear to me why changing from Eqn. (2) to Eqn. (7), i.e., replacing KL with MMD will maximize the mutual information between x and v_{x1}. Please explain it clearly. I do not get the argument stated in Lines 166 to 169.  (2) I understand that the bound in Eqn. (8) is to make the mutual information trackable. However, the formulation of the variational approximation r(\hat{v}_{y1}) is never introduced. Please clarify this.    (3) There are just too many hyper-parameters in the final proposed method. For example, \lambda_{1,2,3}, \beta, \alpha_{1,2,3,4}^{1,2}, the bandwidth value \sigma in the Gaussian kernel of MMD. I'd like to know how these hyperparameters are selected, how sensitive these numbers are, and what are the numbers being used in the experiments. For the current draft, I think it will be difficult to reproduce the experiments.   Overall, I think this paper is marginally below the acceptance threshold. Though the idea is novel, the presentation is not satisfying. If Section 4 is described more clearly, it has a chance to be a good paper.   Minor issues:  (1) Line 13: "domain" => "domains", Line 77: "Works" => "Work"  (2) Equation (2) & (7): the second expectation term: "v_{x1} \sim p(v_{x1}|x)" => "v_{x1} \sim q_x(v_{x1}|x)"   (3) It seems to me Figure 5 & 6 demonstrates the same thing. I am not sure why plotting these two figures separately here. I suggest combining these two figures, and getting rid of one of Figure 5(b) and Figure 6(c). These two subfigures basically are the same.   -- After Rebuttal --  I have read the rebuttal and other reviewers' comments. Generally, I think the authors partially solved my concerns. For example, they discussed how they select hyper-parameters, and more technical details of the method are provided. As to quantitative analysis, it seems using LPIPS and fooling rate aligns well with the previous work. I feel this paper has some novelty inside, and in the camera-ready version, as the authors claim, I hope the authors will take the comments into consideration to make the presentation more clear.  I slightly prefer to voting acceptance, and therefore moving the score to a 6.