ROBUST CONDITIONAL GENERATIVE ADVERSARIAL NETWORKS
Conditional generative adversarial networks (cGAN) have led to large improvements in the task of conditional image generation, which lies at the heart of computer vision. The major focus so far has been on performance improvement, while there has been little effort in making cGAN more robust to noise. The regression (of the generator) might lead to arbitrarily large errors in the output, which makes cGAN unreliable for real-world applications. In this work, we introduce a novel conditional GAN model, called RoCGAN, which leverages structure in the target space of the model to address the issue. Our model augments the generator with an unsupervised pathway, which promotes the outputs of the generator to span the target manifold even in the presence of intense noise. We prove that RoCGAN share similar theoretical properties as GAN and experimentally verify that our model outperforms existing state-of-the-art cGAN architectures by a large margin in a variety of domains including images from natural scenes and faces.
General:
In general, this is a well-written paper. This work focuses on the robustness of conditional GAN(RoC-GAN) when facing the noise. The authors claim the generator of RoC-Gan will span the target manifold, even in the presence of large amounts of noise. The main contribution of the paper is to introduce a two-pathway model, where one of them is used to perform regression as ordinary GAN while the other one helps the whole model span the target domain.
Strength:
1. The idea is simple and straightforward. The authors provide necessary theoretical analysis and empirical validation for their model. 
2. The proposed method seems technically correct to me. i.e. Although I am not very sure how well it works in practice, the idea is fine.
Possible Improvements:
1. I agree adding another auto-encoder as a helper may give better generation results by spanning the whole target space, but I don't think this constraint is strong enough in practice. 
2. In section 3.3, the time complexity of computing 'L_deconv' seems extremely large. From the perspective of numerical optimization, optimizing such a matrix will cause trouble if the dimension of weight matrices are large. i.e. optimizing the high-dimensional covariance matrix seems a problem to me.
3. The experiments looks good. The experiments could be more convincing if using more complex data sets(e.g. CIFAR10, ImageNet) besides CelebA. My concern for using such data sets(the resolution of images is low and the distribution is simple)  is that: although the noise seems to corrupt most of the image, the distribution of the image is not complex, so the generative model can recover it easily. Since this is a more empirical paper, the experiments should be more convincing.
Conclusion:
The author(s) are thoughtful and they put lots of work on this paper. The proposed method is simple. For novelty and significance, I think the idea is not very fancy to me. I am not very convinced by the method proposed in the paper. Although the paper demonstrates the robustness of their model with different experiments, most of them were not performed on deep neural networks and complicated data sets. As a conclusion, I vote for weak rejection.
Minor suggestion:
Increase the resolution of the figures.
------------------------------- After Rebuttal ---------------------------------
I am very satisfied with the authors' response, so I will change my vote from rejection to acceptance.