Recently, Generative Adversarial Network (GAN) and numbers of its variants have been widely used to solve the image-to-image translation problem and achieved extraordinary results in both a supervised and unsupervised manner. However, most GAN-based methods suffer from the imbalance problem between the generator and discriminator in practice. Namely, the relative model capacities of the generator and discriminator do not match, leading to mode collapse and/or diminished gradients. To tackle this problem, we propose a GuideGAN based on attention mechanism. More specifically, we arm the discriminator with an attention mechanism so not only it estimates the probability that its input is real, but also does it create an attention map that highlights the critical features for such prediction. This attention map then assists the generator to produce more plausible and realistic images. We extensively evaluate the proposed GuideGAN framework on a number of image transfer tasks. Both qualitative results and quantitative comparison demonstrate the superiority of our proposed approach.
This paper proposes an extension of the conditional GAN objective, where the generator conditions on an attention map produced by the discriminator in addition to the input image. The motivation is that the discriminator is usually too powerful, and so the gradient the generator receives is often too small in magnitude. By conditioning on the attention map, the generator could leverage information about the regions in the image that the discriminator attends to and use it to generate a new image that better fools the discriminator. 
My main concern is about whether the proposed extension achieves the desired goal. The intuitive motivation provided in the paper aims to add a cooperative component to the two-player game, but the min-max objective corresponds to a zero-sum adversarial game. As a result, when training the discriminator, the discriminator is encouraged to reveal as little information as possible via the attention map, so that the loss maximized. This appears to be the opposite of the desired behavior, so the objective needs to be reformulated. 
Also, it is unclear how inference is performed: at test time, the attention map is unknown and so some placeholder must be used in its place. The paper should clarify what is done at test time, and clearly state the shortcomings as a result of this, i.e. different procedures are used for training and testing, which is not principled. I imagine the generator could rely too much on the attention map as a result - how this is alleviated/prevented should be explained. 
Figure 2: Only the qualitative results for unsupervised image-to-image translation are available; qualitative results for supervised image-to-image translation should also be provided. 
While the quantitative improvement over existing methods is somewhat insignificant, I appreciate the authors discussing their hypotheses why this might be the case. It would be more useful to empirically validate these hypotheses as well. For example, for the claim that "maybe the attention map only focuses on a few domain specific classes so the generator works too hard on those classes and ignores others", it might be good to compute the average per-class attention map intensity to show that some classes appear rarely in the attention map. 
The evaluation protocol should be explained in greater detail (perhaps in the appendix); the segmentation model (which I assume is FCN) should be described and each of the evaluation metrics (per-pixel acc., per-class acc. and IoU) should be described for the benefit of researchers outside the area. 
pg. 4: "in their implementation contains several Resblock (He et al., 2016), which makes it infeasible in our framework". Why is it infeasible?
pg. 6: What are the architectures used by the baselines? Are they comparable to the architecture the proposed method used?
Minor Issues:
pg. 3: "differences between P_x and G_Y \cdot P_y, P_y and G_X \cdot P_x are minimized" - confusing; should rephrase as "the difference between P_x and G_Y \cdot P_y and the difference between P_y and G_X \cdot P_x are minimized". Also should replace \cdot with \circ. 
pg. 4: "like random noisy" -> "like random noise"
pg. 4, last paragraph: "Our trainable attention module follows the same structure of the attention block in RAM (Wang et al., 2017). They built a very deep network with several such blocks, each containing two branches: mask branch and trunk branch. Mask branch cascades the input features through a bottom-up top- down architecture that mimics human attention. Trunk branch is applied as feature processing." - this is very confusing; it would be easier to refer readers to the appendix. 
pg. 5 - "Attention mask can potentially break good property of the raw input." - what does this mean?
pg. 6 - "as showed in Table 4.1" -> "as shown in Table 4.1"