Point clouds are a flexible and ubiquitous way to represent 3D objects with arbitrary resolution and precision. Previous work has shown that adapting encoder networks to match the semantics of their input point clouds can significantly improve their effectiveness over naive feedforward alternatives. However, the vast majority of work on point-cloud decoders are still based on fully-connected networks that map shape representations to a fixed number of output points. In this work, we investigate decoder architectures that more closely match the semantics of variable sized point clouds. Specifically, we study sample-based point-cloud decoders that map a shape representation to a point feature distribution, allowing an arbitrary number of sampled features to be transformed into individual output points. We develop three sample-based decoder architectures and compare their performance to each other and show their improved effectiveness over feedforward architectures. In addition, we investigate the learned distributions to gain insight into the output transformation. Our work is available as an extensible software platform to reproduce these results and serve as a baseline for future work.
Summary
This paper studies the problem of 3D point cloud autoencoding in a deep learning setup, and in particular, the choice of the architecture of a 3D point cloud decoder. The main claim of the paper is the adaptation of the PointNet architecture (a set of fully-connected layers applied per point) from encoding to decoding purposes. To achieve that, the authors propose three similar mechanisms to inject noise in the bottleneck shape representation, which are used to sample per point feature vectors suitable for a PointNet-like decoder. These include appending a low dimensional gaussian noise sample to the bottleneck shape representation, adding individual gaussian noise sample to each bottleneck shape representation dimension or using trainable variances per bottleneck shape representation dimension. Resulting networks are optimized by minimizing Chamfer distance between the ground truth and output point clouds. The main motivation of their choice of these architectures is their ability to output an arbitrary size point clouds, compared to naive multilayered fully-connected network applied to a bottleneck shape representation and outputting a "flattened" fixed-size point cloud.
To verify their claims, the authors considered autoencoding task for ModelNet40 dataset. They compared a naive multilayered fully-connected decoder to PointNet-like decoders with all three proposed noise-injection mechanisms. This comparison was also performed for 5 variations of the proposed models differing is network size. To compare quantitatively, the authors present all results graphically, showing the dependency of resulting average Chamfer distance from the network size. This graph shows that using trainable per dimension variance for noise injection works consistently better, than other options; also, it shows inconsistent results for the baseline multilayered fully-connected decoder. In qualitative comparisons, the authors compare output point clouds visually and study the influence of sampling from various noise dimensions on the reconstructed points.
Review
Although the authors of this paper touch a very interesting topic in generative modeling of point clouds, I do not believe this paper should be published in its current state.
Cons:
1. The main issue in this paper is the absence of comparisons to any external approaches. Even though comparison to their variation of the baseline model is provided, it is impossible to verify where their results stand compared to the related work. There is a whole stream of works in the same task using ShapeNet dataset, which the authors completely omit. Even if the authors do not want to perform experiments on ShapeNet, they, at least, could have provided comparison to AAE and PC-GAN results from PC-GAN paper [1] for generative modeling metrics on ModelNet40. As the authors stated, there are at least two works, PC-GAN and PointFlow, which allow sampling of the arbitrary size point clouds and comparison to these approaches is crucial (especially since the proposed approach is technically significantly simpler compared to hierarchical GANs from PC-GAN and continuous normalizing flows combined with VAEs from PointFlow papers).
2. The paper misses numerous citations to related work in generative 3D shape modeling:
- voxel grid based, (since the authors mention them): [2, 3, 4, 5, 6, 7]
- point cloud based: [8, 9, 10]
- graph based: [11, 12]
All these approaches propose various types of decoders for 3D shapes represented accordingly and some of them perform evaluations for pure generative, autoencoding task, which, again, can be used for comparison.
3. I believe, there is a factual mistake and arguable diminishing of the current state of the art in the related work section:
The authors state, that both PC-GAN and PointFlow models "inhabit GAN settings". However, PointFlow is, in fact, an approximate likelihood model and is considered as an instance of Variational Autoencoder with a shape prior and a point cloud decoder implemented as continuous normalizing flows (a special case of invertible neural networks). The authors also state, that it is "unclear... if they are capable of outperforming the baseline MLP approach" when they talk about these papers, while, in fact, both PC-GAN and PointFlow explicitly show that by comparing to [9], which use the multilayered fully-connected baseline decoder. It is also not unclear if these approaches need specific loss functions, contrary to what authors say. They were designed so they need their particular loss functions, which does not disqualify them from the comparison. These papers are the main competitors of this work and should be addressed properly instead of being diminished.
Pros:
1. Besides the cons, the paper is written clearly and is easy to follow.
2. Rigorousness with respect to comparison of the proposed options (Table 1 and the fact that every model was retrained 15 times to obtain average result).
3. Qualitative experiments with control over noise dimensions are novel and interesting.
Overall, this could have been a paper about a nice and simple alternative to PC-GAN and PointFlow for arbitrary size point cloud modeling, but, unfortunately, it missed the opportunity to properly address the current state-of-the-art in its current version.
Small remarks and questions for authors:
1. Citation of Valsesia at al. in the related work should be reorganized to be consistent with other citations. Omitting full names there and in case of PC-GAN will also be more consistent.
2. All figures in the paper should be updated for better visibility of both graphics and text. Figures 1-4, 7 - text is barely visible, the scheme blocks and lines are too thin and overall lack readability. Figures 5-6, 8 - point clouds are almost invisible even with zooming, remove coordinate grid in the background, zoom in on the objects, so they take as much space in the images as possible, use bigger point sizes for better visibility.
3. Maybe I do not follow something, why have you ignored applications of your point cloud decoders, and particularly generative modeling task and its evaluation protocols, presented for example in [9]?
[1] C.-L. Li, M. Zaheer, Y. Zhang, B. Póczos and R. Salakhutdinov. Point Cloud GAN. In ICLR Workshop on Deep Generative Models for Highly Structured Data, 2019
[2] C. Choy, D. Xu, J.-Y. Gwak, K. Chen, and S. Savarese. 3D-R2N2: A unified approach for single and multi-view 3d object reconstruction. In ECCV, 2016.
[3] R. Girdhar, D. Fouhey, M. Rodriguez, and A. Gupta. Learning a predictable and gen- erative vector representation for objects. In ECCV, 2016.
[4] J. Wu, C. Zhang, T. Xue, W. Freeman, and J. Tenenbaum. Learning a probabilistic latent space of object shapes via 3D generative-adversarial modeling. In NeurIPS, 2016.
[5] M. Tatarchenko, A. Dosovitskiy, and T. Brox. Octree generating networks: Efficient
convolutional architectures for high-resolution 3D outputs. In ICCV, 2017.
[6] S. R. Richter and S. Roth. Matryoshka Networks: Predicting 3d geometry via nested shape layers. In CVPR, 2018.
[7] R. Klokov, J. Verbeek, and E. Boyer. Probabilistic Reconstruction Networks for 3D Shape Inference from a Single Image. In BMVC, 2019
[8] T. Groueix, M. Fisher, V. Kim, B. Russell, and M. Aubry. AtlasNet: A papier-mâché approach to learning 3D surface generation. In CVPR, 2018.
[9] P. Achlioptas, O. Diamanti, I. Mitliagkas, and L. Guibas. Learning Representations and Generative Models for 3D Point Clouds. In ICML, 2018.
[10] P. Mandikal, K. Navaneet, M. Agarwal, and R. Babu. 3D-LMNet: Latent embedding matching for accurate and diverse 3D point cloud reconstruction from a single image. In BMVC, 2018.
[11] N. Wang, Y. Zhang, Z. Li, Y. Fu, W. Liu, and Y. Jiang. Pixel2Mesh: Generating 3D
mesh models from single RGB images. In ECCV, 2018.
[12] C. Wen, Y. Zhang, Z. Li, and Y. Fu. Pixel2Mesh++: Multi-View 3D Mesh Generation via Deformation. In ICCV, 2019.