In this paper we present, to the best of our knowledge, the first method to learn a generative model of 3D shapes from natural images in a fully unsupervised way. For example, we do not use any ground truth 3D or 2D annotations, stereo video, and ego-motion during the training. Our approach follows the general strategy of Generative Adversarial Networks, where an image generator network learns to create image samples that are realistic enough to fool a discriminator network into believing that they are natural images. In contrast, in our approach the image generation is split into 2 stages. In the first stage a generator network outputs 3D objects. In the second, a differentiable renderer produces an image of the 3D objects from random viewpoints. The key observation is that a realistic 3D object should yield a realistic rendering from any plausible viewpoint. Thus, by randomizing the choice of the viewpoint our proposed training forces the generator network to learn an interpretable 3D representation disentangled from the viewpoint. In this work, a 3D representation consists of a triangle mesh and a texture map that is used to color the triangle surface by using the UV-mapping technique. We provide analysis of our learning approach, expose its ambiguities and show how to overcome them. Experimentally, we demonstrate that our method can learn realistic 3D shapes of faces by using only the natural images of the FFHQ dataset.
I thank the authors for the rebuttal and the additional experiments. The additions do partially address my concerns, although not entirely. For instance, the experiments on non-face classes are very preliminary and it is unclear if they work at all (no other views shown). I hope the authors are right that the method will work on other classes after some tuning, but this is not demonstrated in the paper. Overall, I am quite in a borderline mode. I think the paper looks promising and after further improving the experimental evaluation it can become a great publication. But for now the experiments, especially the new ones, look somewhat incomplete and rushed, more suitable for a workshop paper. Therefore, I still lean towards rejection.
---
The paper proposes an approach to learning the 3D structure of images without explicit supervision. The proposed model is a Generative Adversarial Network (GAN) with an appropriate task-specific structure: instead of generating an image directly with a deep network, three intermediate outputs are generated first and then processed by a differentiable renderer. The three outputs are the 3D geometry of the object (represented by a mesh in this work), the texture of the object, and the background image. The final output of the model is produced by rendering the geometry with the texture and overlaying on top of the background. The whole system can be trained end-to-end with a standard GAN objective. The method is applied to the FFHQ dataset of face images, where it produces qualitatively reasonable results.
I am in the borderline mode about this paper. On one hand, I believe the task of unsupervised learning 3D from 2D is interesting and important, and the paper makes an interesting contribution in this direction. On the other hand, the experimental evaluation is quite limited: the results are purely qualitative, on a single dataset, and do not contain much analysis of the method. It would be great if the authors could add more experiments to the paper during the discussion phase.
More detailed comments:
Pros:
1) The paper is presented well, is easy to read. I like the detailed table with comparison to related works, and a good discussion of the limitations of the method and the tricks involved in making it work. I also like section 4 clearly discussing the assumptions of the work, although I think it could be shortened quite a bit.
2) The proposed method is reasonable and seems to work in practice, judging from the qualitative results.
cons:
1) The experiments are limited. 
1a) There are no quantitative results. I understand it is non-trivial to evaluate the method on 3D reconstruction, although one could either train a network inverting the generator, or, perhaps simpler, apply a pre-trained image-to-3D network to the generated images. But at least some image quality measures (FID, IS) could be reported. 
1b) The method is only trained on one dataset of faces. It would be great to apply the method to several other datasets as well, for instance, cars, bedrooms, animal faces, ShapeNet objects. This would showcase the generality of the approach. Otherwise, I am worried the method is fragile and only applies to very clean and simple data. Also, if the method is only applied to faces, it makes sense to mention faces in the title. 
1c) It would be very helpful to have more analysis of the different variants of the method, ideally with quantitative results (again, at least some image quality results). Figure 3 goes in this direction, but it is very small and does not give a clear understanding of the relative performance of diferent variants.
2) A missing very relevant citation of HoloGAN by Nguyen-Phuoc et al.  [1]. It is not yet published, but has been on arXiv for some time. I am a bit unsure about the ICLR policy in this case (this page https://iclr.cc/Conferences/2019/Reviewer_Guidelines suggests that arXiv paper may be formally considered prior work, in which case it should be discussed in full detail), but at least a brief mention would definitely be good.
[1] HoloGAN: Unsupervised learning of 3D representations from natural images. Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, Yong-Liang Yang. arXiv 2019.