Recently, image-to-image translation has seen a significant success. Among many approaches, image translation based on an exemplar image, which contains the target style information, has been popular, owing to its capability to handle multimodality as well as its suitability for practical use. However, most of the existing methods extract the style information from an entire exemplar and apply it to the entire input image, which introduces excessive image translation in irrelevant image regions. In response, this paper proposes a novel approach that jointly extracts out the local masks of the input image and the exemplar as targeted regions to be involved for image translation. In particular, the main novelty of our model lies in (1) co-segmentation networks for local mask generation and (2) the local mask-based highway adaptive instance normalization technique. We demonstrate the quantitative and the qualitative evaluation results to show the advantages of our proposed approach. Finally, the code is available at https://github.com/AnonymousIclrAuthor/ Highway-Adaptive-Instance-Normalization.
Summary--
The paper tries to address an issue existing in current image-to-image translation at the point that different regions of the image should be treated differently. In other word, background should not be transferred while only foreground of interest should be transferred. The paper propose to use co-segmentation to find the common areas to for image translation. It reports the proposed method works through experiments.
There are several major concerns to be addressed before considering to publish.
1) The paper says that "For example, in a person’s facial image translation, if the exemplar image has two attributes, (1) a smiling expression and (2) a blonde hair, then both attributes have to be transferred with no other options", but the model in the paper seems still incapable of transferring only one attribute. Perhaps an interactive transfer make more sense, while co-segmentation does not distinguish the part of interest to the user. Or training a semantic segmentation make more sense as the semantic segment can specify which region to transfer.
2) As co-segmentation is proposed to "capture the regions of a common object existing in multiple input images", why does the co-segmentation network only capture the eye and mouth part in Figure 2 and 3, why does it capture the mouth of different shape and style in the third macro column in Figure 4 instead of eyes? How to train the co-segmentation module, what is the objective function? Why not using a semantic segmentation model?
3) The "domain-invariant content code" and the "style code" seem rather subjective. Are there any principles to design content and style codes? In the experiments, it seems the paper considers five styles to transfer as shown in Table 1. Is the model easy to extend to novel styles for image translation?
4) What does the pink color mean in the very bottom-left or top-right heatmap images in Figure 2? There is no pink color reference in the colorbar.
5) Figure 5: Why there is similariy dark patterns on the mouth? Is it some manual manipulation for interactive transfer?
6) Though it is always good to see the authors are willing to release code and models, it appears uncomfortable that github page noted in the abstract reveals the author information. Moreover, in the github page,
even though it says "an example is example.ipynb", the only ipynb file contains nothing informative and this makes reviewers feel cheated.
Minor--
There are several typos, e.g., lightinig.