We consider the problem of translating, in an unsupervised manner, between two domains where one contains some additional information compared to the other. The proposed method disentangles the common and separate parts of these domains and, through the generation of a mask, focuses the attention of the underlying network to the desired augmentation alone, without wastefully reconstructing the entire target. This enables state-of-the-art quality and variety of content translation, as demonstrated through extensive quantitative and qualitative evaluation. Our method is also capable of adding the separate content of different guide images and domains as well as remove existing separate content. Furthermore, our method enables weakly-supervised semantic segmentation of the separate part of each domain, where only class labels are provided. Our code is available anonymously at http://bit.ly/2mXTizX.
This work proposed a mask based approach for instance-level unsupervised content transfer, which is an extension of the disentanglement work in (Press et al., 2019) and the attention guided translation (Chen et al., 2018, Mejjati et al., 2018). Unlike the disentanglement work, the introduced mask allows the adaptation to focus on the relevant content which substantially reduce the complexity of the generation. On the other hand, the proposed method extends the attention guided translation from the domain level to the instance level which allows more specific and diverse translations. Experiments on benchmark data shows both improved qualitative and quantitative results comparing to existing methods. It is really nice that the authors also considered the situation of generalization to out of domain images.
However, I would encourage the authors to spend more discussion on the "Method" and "Ablation Analysis" sections to give a better illustration. First is the choice of the L2 norm in all the reconstruction losses, which is different from L1 norm used in both (Press et al., 2019) and (Mejjati et al., 2018). What is the advantage of using L2 instead of L1 norm here? Does it work better with the mask generation? Second, the domain confusion loss. The presence of both equation (3) and (4) are quite confusing and the domain confusion loss (3) seems different from traditional ones. In Table 7, it shows that the learned mask is empty without any of the losses (3), (5), (7). But only loss (7) is directly related to the mask generation. How does the loss (3) or (5) impact the mask learning? It is also unclear why the losses introduced in (8) would encourage the mask to be minimal despite the quantitative results shown in Table 7. Actually, I am very curious about the performance of the loss introduced in (Press et al., 2019) on top of the network introduced in Figure 2.
Other comments:
- It would be nice to see the out of domain transfer in the "attribute" domain. Ideally, the network should be able to detect "difference" in the image from domain B and apply it to the image from domain A. For example, the model is trained on faces without and with glasses, but applied to faces without and with facial hair. Indeed, the introduction of mask alleviates the decoder to learn the attribute itself, and provides the ability to locate the place of difference.
- Please unify the citation style: there are both Press et al. (2019) and (Press et al., 2019) used.
- In Section 2 under "Mask Based Approaches", the authors argued that the existing attention guided translation "does not allow for the adaptation of the image information in the masked area". I do not think this is the case. The existing work also introduced adaptation of the image information in the masked area. For example, the equation (1) in (Mejjati et al., 2018).
- How is the binarized mask generated in inference? Specifically, how to determine the threshold?
- In Section 4.1, the authors argued that "without L_{Cycle} the masks produced include larger portions of the face". But this actually produces the second smallest mask in Table 7.
- What is the "L2 reg" in Table 7?
- It would be good to show the sensitivity of the lambdas in the overall loss.