HARMONIC UNPAIRED IMAGE-TO-IMAGE TRANSLA- TION
The recent direction of unpaired image-to-image translation is on one hand very exciting as it alleviates the big burden in obtaining label-intensive pixel-to-pixel supervision, but it is on the other hand not fully satisfactory due to the presence of artifacts and degenerated transformations. In this paper, we take a manifold view of the problem by introducing a smoothness term over the sample graph to attain harmonic functions to enforce consistent mappings during the translation. We develop HarmonicGAN to learn bi-directional translations between the source and the target domains. With the help of similarity-consistency, the inherent selfconsistency property of samples can be maintained. Distance metrics defined on two types of features including histogram and CNN are exploited. Under an identical problem setting as CycleGAN, without additional manual inputs and only at a small training-time cost, HarmonicGAN demonstrates a significant qualitative and quantitative improvement over the state of the art, as well as improved interpretability. We show experimental results in a number of applications including medical imaging, object transfiguration, and semantic labeling. We outperform the competing methods in all tasks, and for a medical imaging task in particular our method turns CycleGAN from a failure to a success, halving the mean-squared error, and generating images that radiologists prefer over competing methods in 95% of cases.
This paper adds a spatial regularization loss to the well-known CycleGAN loss for unpaired image-to-image translation (Zhu et al., ICCV17).  Essentially, the regularization loss (Eq. 6) is similar to imposing a CRF (Conditional Random Field) term on the network outputs, encouraging spatial consistency between patches within each generated image.
The paper is clear and well written.
Unpaired Image-to-Image translation is an important problem. 
The way the smoothness loss (Eq. 6) is presented gives readers the impression that spatial pairwise regularization is new, ignoring its long history (e.g., CRFs) in computer vision (not a single classical paper on CRFs is cited). Putting aside classical spatial regularization works, imposing pairwise regularization on the outputs of modern deep networks has been investigated in a very large number of works recently, particularly in the context of weakly-supervised semantic CNN segmentation, e.g.,  [Tang et al., On Regularized Losses for Weakly-supervised CNN Segmentation, ECCV 18 ], [Lin et al. : Scribblesup: Scribble-supervised convolutional networks for semantic segmentation, CVPR 2016], among  many other works. Very similar in spirit to this ICLR submission, these works impose within-image pairwise regularization (e.g., CRF) on the latent outputs of deep networks, with the main difference that these works use CNN semantic segmentation classifiers whereas here we have a CycleGAN for image generation.
Also, in the context of supervised CNN segmentation, CRFs have made a significant impact when used as post-processing step, e.g., very well known works such as [DeepLab by Chen et al. ICLR15] and [CRFs as recurrent Neural Networks by Zheng et al., ICCV 2015]. 
It might be a valid contribution to evaluate spatial regularization (e.g., CRFs losses) on image generation tasks (such as CycleGAN), but the paper really needs to acknowledge very related prior works on regularization (at least in the context of deep networks).
There are also related pioneering semi-supervised deep learning works based on graph Laplacian regularization, e.g., [Westen et al., Deep Learning via Semi-supervised embedding, ICML 2008], which the paper does not acknowledge/discuss. 
The manifold regularization terminology is misleading. The regularization is not over the feature space of image samples. It is within the spatial domain of each generated image (patch or pixel level); so, in my opinion, CRF (or spatial) regularization (instead of manifold regularization) is a much more appropriate terminology. 
Also, I would not call this approach HarmonicGan. I would call it CRF-GAN or Spatially-Regularized GAN. The computation of harmonic functions is just one way, among many other (potentially better) ways to optimize pairwise smoothness terms (including the case of the used smoothness loss). And, by the way, I did not get how the loss in (9) gives a harmonic function. Could you please clarify and give more details? In my understanding, the harmonic solution in [ Zhu and Ghahramani, ICML 2013] comes directly as a solution of the graph Laplacian (and it assumes some labeled points, i.e., a semi-supervised setting). Even, if the solution is correct (which I do not see how), I do not think it is an efficient way to handle pairwise-regularization problems in image processing, particularly when matrix  W = [w_{ij}] is dense (which might be the case here, unless you are truncating the Gaussian kernel with some heuristics). In this case, back-propagating the proposed loss would be of quadratic complexity w.r.t the number of image patches. Again, there is a long tradition in optimizing efficiently pairwise regularizers in vision/learning (even in the case of dense affinity matrices), and one very well-known work, which is currently being used a lot in the context imposing CRF structure on the outputs of deep networks, is  [Krahenbuhl and Koltun, Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials], NIPS 2011. This highly related and widely used inference work for dense pairwise regulation is not cited/discussed neither. The Gaussian filtering ideas of the work of Krahenbuhl and Koltun, which ease optimizing dense pairwise terms (from quadratic to linear) are applicable here (as a Gaussian kernel is used), and are widely used in computer vision, including closely related works imposing spatial regularization losses on the outputs of deep networks, e.g., [Tang et al., On Regularized Losses for Weakly-supervised CNN Segmentation, ECCV 18], among many others.  
  
When using feature from pre-training (VGG) in the CRF loss, the comparison with unsupervised CycleGAN is not fair. In Table 2 (Label translation on Cityscapes), CycleGAN outperforms the proposed method in all metrics when only unsupervised histogram features are used, which makes me doubt about the practical value of the proposed regularization in the context of image-translation tasks. Having said that, the histogram-based regularization is helping in the medical-imaging application (Table 1). By the way, the use of histograms (of patches or super-pixels) as unsupervised features in pairwise regularization is not new neither; see for instance [Lin et al.: Scribblesup: Scribble-supervised convolutional networks for semantic segmentation, CVPR 2016]. Also, it might be better to use super-pixels instead of patches. 
So, in summary, the technical contribution is minor, in my opinion (imposing pairwise regularization on the outputs of deep networks has been done in many works, but not for CycleGAN); optimization of the proposed loss as a harmonic function is not clear to me; using VGG in the comparisons with CycleGAN is not fair; and the long history of closely-related spatial regularization terms (e.g., CRFs) in computer vision is completely ignored.
Minor: please use ‘term’ instead of ‘constraint’. These are unconstrained optimization problems and there are no equality or inequality constraints here.