We propose a novel method for unsupervised image-to-image translation, which incorporates a new attention module and a new learnable normalization function in an end-to-end manner. The attention module guides our model to focus on more important regions distinguishing between source and target domains based on the attention map obtained by the auxiliary classifier. Unlike previous attention-based method which cannot handle the geometric changes between domains, our model can translate both images requiring holistic changes and images requiring large shape changes. Moreover, our new AdaLIN (Adaptive Layer-Instance Normalization) function helps our attention-guided model to flexibly control the amount of change in shape and texture by learned parameters depending on datasets. Experimental results show the superiority of the proposed method compared to the existing state-of-the-art models with a fixed network architecture and hyperparameters.
I have read the authors' rebuttal and satisfied with their response. Novelty is a little on the lower side, but thorough writing, results, and insightful comparisons make up for this in my opinion. I have updated my score to 8: Accept.
=====
This paper proposes an approach to perform image translation called U-GAT-IT. In image translation, the goal is to learn a mapping from images in a source domain to corresponding images in a target domain. Contemporary image translation approaches are able to transfer local texture but struggle to handle shape transfer. To address this concern, the authors introduce an attention mechanism based on CAM [1] and an adaptive normalization layer into a GAN-based image translation framework. Results indicate favorable quantitative and qualitative performance relative to a number of baselines.
Specific contributions include:
* Introduction of a normalization layer called AdaLIN that can interpolate between instance normalization and layer normalization based on the input.
* Introduction of an attention mechanism based on CAM [1] that allows the model to focus on specific parts of the image when either generating or discriminating.
* Collection and release of a selfie-to-anime dataset.
* Release of U-GAT-IT code.
  
In my opinion this paper is borderline, leaning towards weak accept. The experiments are thorough and the paper is well-written. I have concerns about the novelty and significance of the work, but overall the paper feels very close to being a finished piece of work in spite of its (relatively minor) flaws.
  
Strong points of this work include the writing and experiments. The paper is clearly organized and feels polished. It cites many relevant works, giving the reader a sense of the contemporary approaches for image translation. There is a thorough description of model architecture, dataset and tuning parameters in the appendix. In addition, code and the selfie-to-anime dataset have been released by the authors. In terms of experiments, the authors provide many qualitative visualizations comparing the proposed model to baselines on various datasets. Quantitative evaluation includes KID and a perceptual evaluation on human subjects.
Weak points include novelty and significance. The proposed approach combines two ideas already applied to image translation (adaptive normalization [3] and attention [4]).  It therefore synthesizes these ideas into an effective algorithm rather than directly adding something new. It is unclear to me how others can build on top of this work to further advance state-of-the-art in image translation. Are more sophisticated normalization and attention mechanisms truly the key to improving image translation in the future?
Specific comments:
* The formulation of AdaLIN in Equation (1) is vague. The text states "parameters are dynamically computed by a fully connected layer from the attention map", but it's not clear what those parameters are in the equation. Explicitly writing \gamma and \beta as functions of the fully-connected layer and \mu_I, \sigma_I, \mu_L, \sigma_L as the corresponding mean and standard deviation expressions would make things more clear.
* The motivation for using layer normalization was discussed in 2.1.1 but I still do not understand why it is beneficial.
* The term "importance weights" has a specific meaning in the context of Monte Carlo methods. I would suggest choosing a different term here.
Questions for the authors:
* How does U-GAT-IT compare to TransGaGa [2]? One of the stated goals of U-GAT-IT is to better handle shape when performing image translation. TransGaGa has a similar motivation and so I would have liked to see an experimental comparison or at the very least a description of how U-GAT-IT differs. What sorts of shape transfer could U-GAT-IT handle that TransGaGa couldn't and vice versa?
* What are the shortcomings of the model and how could they possibly be addressed? 
  
[1] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A. and Torralba, A., 2016. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2921-2929).
[2] Wu, W., Cao, K., Li, C., Qian, C. and Loy, C.C., 2019. Transgaga: Geometry-aware unsupervised image-to-image translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 8012-8021).
[3] Huang, X. and Belongie, S., 2017. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1501-1510).
[4] Mejjati, Y.A., Richardt, C., Tompkin, J., Cosker, D. and Kim, K.I., 2018. Unsupervised attention-guided image-to-image translation. In Advances in Neural Information Processing Systems (pp. 3693-3703).