Unsupervised Learning of 3D Structure from Images
A key goal of computer vision is to recover the underlying 3D structure that gives rise to 2D observations of the world. If endowed with 3D understanding, agents can abstract away from the complexity of the rendering process to form stable, disentangled representations of scene elements. In this paper we learn strong deep generative models of 3D structures, and recover these structures from 2D images via probabilistic inference. We demonstrate high-quality samples and report log-likelihoods on several datasets, including ShapeNet [2], and establish the first benchmarks in the literature. We also show how these models and their inference networks can be trained jointly, end-to-end, and directly from 2D images without any use of ground-truth 3D labels. This demonstrates for the first time the feasibility of learning to infer 3D representations of the world in a purely unsupervised manner.
This paper provides a generative model of 3D volumes and 2D images using an internal 3D latent representation. The model can be conditioned on a context, which in turn can be either a category or multiple views of the 3D volume. In a very similar way to DRAW, the model transforms a set of latent Gaussian variables into an internal 3D representation, which in turn is converted to a 2D image via a convolutional network or OpenGL renderer. Inference in this generative model is performed using variational inference with a recognition network. Five different tasks are considered.
The paper is well written, clear and contains appropriate references. Also the number of figures help understand and illustrate the text appropriately.  The technical content of the paper seems sound, but its ideas are not particularly novel. The proposed model is an extension of DRAW, which now includes a context (which is observed and used as an additional input to the recognition and generation networks) and generates 3D representations instead of 2D representations. This requires the addition of a projection step to 2D at the end of the generation process when the observations are 2D. Well-known variational inference is applied to the resulting model.  Some of the claims in the introduction are misleading. "We show how the aforementioned models and  inference networks can be trained end-to-end directly from 2D images without any use of ground-truth 3D labels.". This make it seem as if 3D latent structure was discovered exclusively from 2D information. This is not the case, and prior knowledge about 3D structure and projection from 3D to 2D is included in every version of the model. Only the last experiment seems not to use 3D input data directly (the multi-view experiment uses depth in the provided context). But in that experiment, an OpenGL renderer is introduced in the loop, providing, if we wish, an infinite amount of 3D ground truth by sampling from its input-output pairs. Additionally, not every 3D shape can be accommodated by the model of the last experiment due to its particular parameterization, which is not pointed out in the paper.  I would like more clarity regarding training and test splits. For instance, the objects in Figures 9 and 10 were observed at all during training or not? I would like to distinguish generalization to new perspectives from generalization to entirely new shapes.  One thing I found concerning about this paper is the lack of comparison with benchmarks. The experiments comprise 5 different tasks. I would like to see the performance of a reasonable benchmark for each of these tasks, so that it is possible to judge whether the results produced by this model are particularly impressive or not.  - For instance, Section 3.5 discusses multiview contexts. The text is not clear about the nature of the views, but the corresponding figure and legend clarify that the input views are 3D. It seems like it should be relatively easy to infer a 3D representation from multiple views with depth information. The obtained results should be compared with those of a different technique for the same task to ascertain whether the model is working particularly well or not.  - Similarly Section 3.6 uses a very specific parameterization of the input space that restricts the applicability of the model. My understanding is that the same experiment could not be performed with the objects from ShapeNet, for instance. This should be pointed out. Are the objects we see in Figure 10 used to train the model? If they are as I suspect, couldn't the model be basically memorizing 3D structures for each object and then inferring pose for each test image? (as opposed to being able to get 3D understanding of a new object). Then the problem could be posed as, given a set of training images, identify same-object clusters and find its 3D representation. A baseline for this task should be included so that the quality of the proposed model could be judged.  - Section 3.3 and Figure 7: Again the generalization capabilities are unclear. Is the model generating _new_ shapes within the same category or multiple rotations of already-observed shapes? The nearest training neighbor to each generation, computed using a meaningful metric (that matches shape regardless of rotation, translation, and scaling) could be useful here.  Minor:    Section A.3: "of the three kernels": there are only 2 kernels in the 2D ST. Section 2.1:  "fwrite(st,h_t-1;theta_w) = VST(g1(st),g2(st))" This equation doesn't seem correct, isn't h_t-1 used during writing?