Toward Multimodal Image-to-Image Translation
Many image-to-image translation problems are ambiguous, as a single input image may correspond to multiple possible outputs. In this work, we aim to model a distribution of possible outputs in a conditional generative modeling setting. The ambiguity of the mapping is distilled in a low-dimensional latent vector, which can be randomly sampled at test time. A generator learns to map the given input, combined with this latent code, to the output. We explicitly encourage the connection between output and the latent code to be invertible. This helps prevent a many-to-one mapping from the latent code to the output during training, also known as the problem of mode collapse, and produces more diverse results. We explore several variants of this approach by employing different training objectives, network architectures, and methods of injecting the latent code. Our proposed method encourages bijective consistency between the latent encoding and output modes. We present a systematic comparison of our method and other variants on both perceptual realism and diversity.
Summary:
The paper enforces a cycle consistency between the latent code and the image output of a generative model that is applied to image-image translation problems. The cycle consistency encourages invertibility of a generated output to the latent code, which addresses the popular issue of mode collapse in generative adversarial networks. The paper extends the pix2pix [1] framework of image-image translation by conditioning the generation of an image not only on an input image but also on a randomly sampled latent code. The low dimensional latent code captures the ambiguous aspects of the output image such as (depending on application) color, texture, etc.  While Isola et al. [1] observe that randomly sampling a latent code alone doesn’t help with mode collapse, the proposed approach applies a cycle consistency loss on both the input-output image pairs and the latent code - output pairs via alternating joint optimization. 
Strengths:
* Mode collapse is a commonly occurring issue in generative adversarial networks, and a method to produce more diverse results could be potentially useful to a large community.   
* The approach is a natural extension of the cycle consistency idea of [2] to address the issue of mode collapse.  
* From the examples in the supplementary material it is clear that the proposed approach generates samples that are both realistic and diverse in the edges -> shoes task, when compared with baseline approaches such as pix2pix, pix2pix+noise, and an ablation of the overall approach, latent cycle GAN (described in 3.3). 
* The analyses in Section 4.2 are interesting. The analysis of the latent space reinforces the earlier claim regarding the poor distribution of latent codes when the model is trained using the Image Cycle.  
* The paper is well written and easy to understand.  
Clarification questions: 
* Are there any specific reasons or intuitions behind the decision to use the least squares loss in the experiment 3.1 (Equation (1)) instead of the discriminator loss used in [1]? 
* It is unclear whether the table in Figure 3., reports Google maps → Satellite images or the opposite transformation.  
* Did the authors ensure that the performance of different methods in the table in Figure 3 are comparable with pix2pix? Specifically, are the ground truth images processed identically as [1]? Please note that [2] states that there are a few differences in preprocessing and hence the performance of methods in [2] are not comparable with [1].  
* Have the authors performed human-perception experiments on other datasets such as shoes? As a consequence of their semantic content, these datasets appear to contain more diverse images in terms of colors, textures etc., than aerial photos.  
* It is unclear how the diversity score in the Table in Figure 3 is computed. Is it the cosine distance between vectors of neuron activations in a layer computed for pairs of images randomly sampled from the set B? 
Final recommendation:
The paper proposes an approach to overcome the popular issue of mode collapse in generative adversarial networks which could be useful to further research into generating diverse samples in conditional generative models. 
References:
[1] Isola et al.:	Image-to-image translation with conditional adversarial networks. CVPR, 2017. 
[2] Zhu et al.: Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv preprint 2017.