Workshop track -ICLR 2018 LEARNING REPRESENTATIONS AND GENERATIVE MODELS FOR 3D POINT CLOUDS
Three-dimensional geometric data offer an excellent domain for studying representation learning and generative modeling. In this paper, we look at geometric data represented as point clouds. We introduce a deep autoencoder (AE) network with state-of-the-art reconstruction quality and generalization ability. The learned representations outperform existing methods on 3D recognition tasks and enable basic shape editing via simple algebraic manipulations, such as semantic part editing, shape analogies and shape interpolation. We perform a thorough study of different generative models including: GANs operating on the raw point clouds, significantly improved GANs trained in the fixed latent space of our AEs and, Gaussian mixture models (GMM). For our quantitative evaluation we propose measures of sample fidelity and diversity based on matchings between sets of point clouds. Interestingly, our careful evaluation of generalization, fidelity and diversity reveals that GMMs trained in the latent space of our AEs produce the best results.
Summary:
This paper proposes generative models for point clouds. First, they train an auto-encoder for 3D point clouds,  somewhat similar to PointNet (by Qi et al.). Then, they train generative models over the auto-encoder's latent space, both using a "latent-space GAN" (l-GAN) that outputs latent codes, and a Gaussian Mixture Model. To generate point clouds, they sample a latent code and pass it to the decoder. They also introduce a "raw point cloud GAN" (r-GAN) that, instead of generating a latent code, directly produces a point cloud.
They evaluate the methods on several metrics. First, they show that the autoencoder's latent space is a good representation for classification problems, using the ModelNet dataset. Second, they evaluate the generative model on several metrics (such as Jensen-Shannon Divergence) and study the benefits and drawbacks of these metrics, and suggest that one-to-one mapping metrics such as earth mover's distance are desirable over Chamfer distance. Methods such as the r-GAN score well on the latter by over-representing parts of an object that are likely to be filled.
Pros:
- It is interesting that the latent space models are most successful, including the relatively simple GMM-based model. Is there a reason that these models have not been as successful in other domains?
- The comparison of the evaluation metrics could be useful for future work on evaluating point cloud GANs. Due to the simplicity of the method, this paper could be a useful baseline for future work.
- The part-editing and shape analogies results are interesting, and it would be nice to see these expanded in the main paper.
Cons:
- How does a model that simply memorizes (and randomly samples) the training set compare to the auto-encoder-based models on the proposed metrics? How does the diversity of these two models differ?
- The paper simultaneously proposes methods for generating point clouds, and for evaluating them. The paper could therefore be improved by expanding the section comparing to prior, voxel-based 3D methods, particularly in terms of the diversity of the outputs. Although the performance on automated metrics is encouraging, it is hard to conclude much about under what circumstances one representation or model is better than another.
- The technical approach is not particularly novel. The auto-encoder performs fairly well, but it is just a series of MLP layers that output a Nx3 matrix representing the point cloud, trained to optimize EMD or Chamfer distance. The most successful generative models are based on sampling values in the auto-encoder's latent space using simple models (a two-layer MLP or a GMM).
- While it is interesting that the latent space models seem to outperform the r-GAN, this may be due to the relatively poor performance of r-GAN than to good performance of the latent space models, and directly training a GAN on point clouds remains an important problem.
- The paper could possibly be clearer by integrating more of the "background" section into later sections. Some of the GAN figures could also benefit from having captions.
Overall, I think that this paper could serve as a useful baseline for generating point clouds, but I am not sure that the contribution is significant enough for acceptance.