Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
We study the problem of 3D object generation. We propose a novel framework, namely 3D Generative Adversarial Network (3D-GAN), which generates 3D objects from a probabilistic space by leveraging recent advances in volumetric convolutional networks and generative adversarial nets. The benefits of our model are three-fold: first, the use of an adversarial criterion, instead of traditional heuristic criteria, enables the generator to capture object structure implicitly and to synthesize high-quality 3D objects; second, the generator establishes a mapping from a low-dimensional probabilistic space to the space of 3D objects, so that we can sample objects without a reference image or CAD models, and explore the 3D object manifold; third, the adversarial discriminator provides a powerful 3D shape descriptor which, learned without supervision, has wide applications in 3D object recognition. Experiments demonstrate that our method generates high-quality 3D objects, and our unsupervisedly learned features achieve impressive performance on 3D object recognition, comparable with those of supervised learning methods.
This paper proposes to combine Generative Adversarial Network with Volumetric Convolutional Network to perform 3D shape generation. The combined network, called VAN,  learns a latent representation for the objects, which is also used to generate shape. In the experiment section, the shape generation pipeline performs reasonably well and produces very convincing results. The experiment also shows that the latent representation z is semantically meaningful.  
I think this paper presents an straightforward way to generate 3D objects. The idea of training a Generative Adversarial Network combined with a Volumetric Network to generate 3D shape is definitely novel. Furthermore, incriminating an image encoder to VAN in order to output the latent representation vector z from an image is interesting. The experiments are extensive and well-designed, and prove the effectiveness of the method.    However, it gives me the impression that the method proposed is a  combination of existing methods but applied to a novel problem setup, so the novelty is limited.    Here are my concerns:   1. The paper argues that the shape generated by VAN is similar but not identical to the nearest neighbor retrieved from the training set. It’s very hard to judge the similarity by just looking at Figure 2., and thus not very convincing. They looks kind of the same to me. Can you provide any quantitative evaluation?  2.It’s not clear what exactly you did with the output feature from the second to last layer in Section 4.3.Do you use these features to retrain a classifier to perform classification?   3.What is the intuition behind the loss function of VAE-VAN in Equation (2) ? How do you train the VAE-VAN? Is the training procedure the same as training the VAN?  4.In section 4.3, the paper shows an experiment on reconstructing 3D from single images using VAE-VAN. Figure 4 shows a large variety of examples of furnitures taken at widely different angles. Does that mean the method is invariant to viewing angle? Does the images of the same object taken at different angles produce the same latent representation z, as well as the final shape?   5.The paper performs classification using the output of the second to last layer of the discriminator network as feature, and the results looks very promising. But given that the latent representation z is so meaningful as shown in section 5, have you tried to use z as a feature to do classification? I wonder what the result would be.  6.The output mesh grid looks very coarse, and I guess that’s because the low resolution you are using. In that sense, the network is actually learning some general mid-level structures of the shapes, like armpits or legs of a chair, just as it is demonstrated in your experiment. Have you tried a higher resolution? Does this method work on higher resolution, where there are more details? Can it still learn as well?    Overall, I think this is an interesting paper. Please address my concerns.