summaries: This manuscript focuses on reconstructing 3D shapes from point clouds, with applications for instance to 3D scanners. The contribution builds on an adversarial formulation of the reconstruction, as in GANs. The method uses an encoder to map the observed noisy set of points into a lower-dimensional latent space and a decoder for the inverse mapping. The training loss is based on an Earth Mover's Distance between points. The training is done with an adversarial (min-max) strategy, that seeks to align the behavior of the encoder / decoder across a clean complete dataset and a partially-observed noisy one, with a Hausdorff distance loss, to cater for partial matching. The method is benchmarked on simulated and real data. It outperforms the state of the art for unsupervised settings (no known reconstructions) but for supervised settings it is slightly below the PCN approach. The paper addresses the task of point cloud completion within an unpaired setting, where explicit correspondences between the partial and the complete shapes is not given. The setting represents significant interest in practice, e.g. in autonomous driving applications, where the precise completions of scanned objects, e.g. surrounding cars, are not necessary.  This paper proposes a new method for making 3D point clouds by automatically completing 3D scans. It does not require paired data samples for training which makes it possible to train it on real data instead of synthetic data. The authors use a generative adversarial network (GAN) to “generate” complete point clouds from noisy or partial point clouds obtained by 3D scanning. The generator learns to perform mapping from point set manifold of scanned noisy and partial input X_r to manifold of clean shapes X_c. The discriminator tries to tell between encoded clean shapes (synthetic data point clouds) and mappings of noisy input (point clouds from real-life data 3D scans). 
abstractText: As 3D scanning solutions become increasingly popular, several deep learning setups have been developed for the task of scan completion, i.e., plausibly filling in regions that were missed in the raw scans. These methods, however, largely rely on supervision in the form of paired training data, i.e., partial scans with corresponding desired completed scans. While these methods have been successfully demonstrated on synthetic data, the approaches cannot be directly used on real scans in the absence of suitable paired training data. We develop a first approach that works directly on input point clouds, does not require paired training data, and hence can directly be applied to real scans for scan completion. We evaluate the approach qualitatively on several real-world datasets (ScanNet, Matterport3D, KITTI), quantitatively on 3D-EPN shape completion dataset, and demonstrate realistic completions under varying levels of incompleteness.
authors.name: ADVERSARIAL TRAINING
authors.name: Xuelin Chen
authors.name: Baoquan Chen
authors.name: Niloy J. Mitra
references.authors: Panos Achlioptas Olga Diamanti Ioannis Mitliagkas Leonidas Guibas
references.title: Learning representations and generative models for 3d point clouds
references.venue: In International Conference on Machine Learning (ICML),
references.authors: Adrian Bulat Jing Yang Georgios Tzimiropoulos
references.title: To learn image super-resolution, use a gan to learn how to do image degradation first
references.venue: In European Conference on Computer Vision (ECCV),
references.authors: Angel Chang Angela Dai Thomas Funkhouser Maciej Halber Matthias Niessner Manolis Savva Shuran Song Andy Zeng Yinda Zhang
references.title: Matterport3d: Learning from rgb-d data in indoor environments
references.venue: arXiv preprint arXiv:1709.06158,
references.authors: Brian Curless Marc Levoy
references.title: A volumetric method for building complex models from range
references.authors: Angela Dai Angel X. Chang Manolis Savva Maciej Halber Thomas Funkhouser Matthias Nießner
references.title: Scannet: Richly-annotated 3d reconstructions of indoor scenes
references.venue: In Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Angela Dai Charles Ruizhongtai Qi Matthias Nießner
references.title: Shape completion using 3d-encoderpredictor cnns and shape synthesis
references.venue: In International Conference on Computer Vision (ICCV),
references.authors: Angela Dai Daniel Ritchie Martin Bokeloh Scott Reed Jürgen Sturm Matthias Nießner
references.title: Scancomplete: Large-scale scene completion and semantic segmentation for 3d scans
references.venue: In Conference on Computer Vision and Pattern Recognition
references.authors: Andreas Geiger Philip Lenz Raquel Urtasun
references.title: Are we ready for autonomous driving? the kitti vision benchmark suite
references.venue: In Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville Yoshua Bengio
references.title: Generative adversarial nets
references.venue: Advances in Neural Information Processing Systems,
references.authors: Paul Guerrero Yanir Kleiman Maks Ovsjanikov Niloy J Mitra
references.title: Pcpnet learning local shape properties from raw point clouds
references.venue: In Computer Graphics Forum,
references.authors: Swaminathan Gurumurthy Shubham Agrawal
references.title: High fidelity semantic shape completion for point clouds using latent optimization
references.venue: IEEE Winter Conference on Applications of Computer Vision (WACV),
references.authors: Xiaoguang Han Zhen Li Haibin Huang Evangelos Kalogerakis Yizhou Yu
references.title: High-resolution shape completion using deep neural networks for global structure and local geometry inference
references.venue: In International Conference on Computer Vision (ICCV),
references.authors: Satoshi Iizuka Edgar Simo-Serra Hiroshi Ishikawa
references.title: Globally and locally consistent image completion
references.venue: ACM Transactions on Graphics (TOG),
references.authors: Christian Ledig Lucas Theis Ferenc Huszár Jose Caballero Andrew Cunningham Alejandro Acosta Andrew Aitken Alykhan Tejani Johannes Totz Zehan Wang
references.title: Photo-realistic single image super-resolution using a generative adversarial network
references.venue: In Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Jiaxin Li Ben M Chen Gim Hee Lee
references.title: So-net: Self-organizing network for point cloud analysis
references.venue: In Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Yangyan Li Rui Bu Mingchao Sun Wei Wu Xinhan Di Baoquan Chen
references.title: Pointcnn: Convolution on x-transformed points
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Xudong Mao Qing Li Haoran Xie Raymond Y.K. Lau Zhen Wang
references.title: Multi-class generative adversarial networks with the L2 loss function
references.authors: Xudong Mao Qing Li Haoran Xie Raymond YK Lau Zhen Wang Stephen Paul Smolley
references.title: Least squares generative adversarial networks
references.venue: In International Conference on Computer Vision (ICCV),
references.authors: Seong-Jin Park Hyeongseok Son Sunghyun Cho Ki-Sang Hong Seungyong Lee
references.title: Srfeat: Single image super-resolution with feature discrimination
references.venue: In European Conference on Computer Vision (ECCV),
references.authors: Charles R Qi Hao Su Kaichun Mo Leonidas J Guibas
references.title: Pointnet: Deep learning on point sets for 3d classification and segmentation
references.venue: In Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Charles R Qi Wei Liu Chenxia Wu Hao Su Leonidas J Guibas
references.title: Frustum pointnets for 3d object detection from rgb-d data
references.venue: In Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Charles Ruizhongtai Qi Li Yi Hao Su Leonidas J Guibas
references.title: Pointnet++: Deep hierarchical feature learning on point sets in a metric space
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Abhishek Sharma Oliver Grau Mario Fritz
references.title: Vconv-dae: Deep volumetric shape learning without object labels
references.venue: In European Conference on Computer Vision (ECCV),
references.authors: Shuran Song Fisher Yu Andy Zeng Angel X Chang Manolis Savva Thomas Funkhouser
references.title: Semantic scene completion from a single depth image
references.venue: Conference on Computer Vision and Pattern Recognition
references.authors: David Stutz Andreas Geiger
references.title: Learning 3d shape completion from laser scan data with weak supervision
references.venue: In Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Hang Su Varun Jampani Deqing Sun Subhransu Maji Evangelos Kalogerakis Ming-Hsuan Yang Jan Kautz
references.title: Splatnet: Sparse lattice networks for point cloud processing
references.venue: In Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Duc Thanh Nguyen Binh-Son Hua Khoi Tran Quang-Hieu Pham Sai-Kit Yeung
references.title: A field model for repairing 3d shapes
references.venue: In Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Weiyue Wang Qiangui Huang Suya You Chao Yang Ulrich Neumann
references.title: Shape inpainting using 3d generative adversarial network and recurrent convolutional networks
references.venue: In International Conference on Computer Vision (ICCV),
references.authors: Xintao Wang Ke Yu Shixiang Wu Jinjin Gu Yihao Liu Chao Dong Yu Qiao Chen Change Loy
references.title: Esrgan: Enhanced super-resolution generative adversarial networks
references.venue: In European Conference on Computer Vision (ECCV),
references.authors: Bo Yang Stefano Rosa Andrew Markham Niki Trigoni Hongkai Wen
references.title: 3d object dense reconstruction from a single depth view
references.venue: arXiv preprint arXiv:1802.00411,
references.authors: Raymond A Yeh Chen Chen Teck Yian Lim Alexander G Schwing Mark Hasegawa-Johnson Minh N Do
references.title: Semantic image inpainting with deep generative models
references.venue: In Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Kangxue Yin Hui Huang Daniel Cohen-Or Hao Zhang
references.title: P2p-net: bidirectional point displacement net for shape transform
references.venue: ACM Transactions on Graphics (TOG),
references.authors: Lequan Yu Xianzhi Li Chi-Wing Fu Daniel Cohen-Or Pheng-Ann Heng
references.title: Ec-net: an edgeaware point set consolidation network
references.venue: In European Conference on Computer Vision (ECCV),
references.authors: Lequan Yu Xianzhi Li Chi-Wing Fu Daniel Cohen-Or Pheng-Ann Heng
references.title: Pu-net: Point cloud upsampling network
references.venue: In Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Wentao Yuan Tejas Khot David Held Christoph Mertz Martial Hebert
references.title: Pcn: Point completion network
references.venue: In 2018 International Conference on 3D Vision (3DV),
sections.heading: 1 INTRODUCTION
sections.text: Robust, efficient, and scalable solutions now exist for easily scanning large environments and workspaces (Dai et al., 2017a; Chang et al., 2017). The resultant scans, however, are often partial and have to be completed (i.e., missing parts have to be hallucinated and filled in) before they can be used in downstream applications, e.g., virtual walk-through, path planning.
  The most popular data-driven scan completion methods rely on paired supervision data, i.e., for each incomplete training scan, a corresponding complete data (e.g., voxels, point sets, signed distance fields) is required. One way to establish such a shape completion network is then to train a suitably designed encoder-decoder architecture (Dai et al., 2017b; 2018). The required paired training data is obtained by virtually scanning 3D objects (e.g., SunCG Song et al. (2017), ShapeNet Chang et al. (2015) datasets) to simulate occlusion effects. Such approaches, however, are unsuited for real scans where large volumes of paired supervision data remain difficult to collect. Additionally, when data distributions from virtual scans do not match those from real scans, completion networks trained on synthetic-partial and synthetic-complete data do not sufficiently generalize to real (partial) scans. To the best of our knowledge, no point-based unpaired method exists that learns to translate noisy and incomplete point cloud from raw scans to clean and complete point sets.
  We propose an unpaired point-based scan completion method that can be trained without requiring explicit correspondence between partial point sets (e.g., raw scans) and example complete shape models (e.g., synthetic models). Note that the network does not require explicit examples of real complete scans and hence existing (unpaired) large-scale real 3D scan (e.g., Dai et al. (2017a); Chang et al. (2017)) and virtual 3D object repositories (e.g., Song et al. (2017); Chang et al. (2015)) can directly be leveraged as training data. Figure 1 shows example scan completions. As we show in Table 1, unlike methods requiring paired supervision, our method continues to perform well even if the data distributions of synthetic complete scans and real partial scans differ.
  We achieve this by designing a generative adversarial network (GAN) wherein a generator, i.e., an adaptation network, transforms the input into a suitable latent representation such that a discriminator cannot differentiate between the transformed latent variables and the latent variables obtained from training data (i.e., complete shape models). Intuitively, the generator is responsible for the key task of mapping raw partial point sets into clean and complete point sets, and the process is regu-
  larized by working in two different latent spaces that have separately learned manifolds of scanned and synthetic object data.
  We demonstrate our method on several publicly available real-world scan datasets namely (i) ScanNet (Dai et al., 2017a) chairs and tables; (ii) Matterport3D (Chang et al., 2017) chairs and tables; and (iii) KITTI (Geiger et al., 2012) cars. In absence of completion ground truth, we cannot directly compute accuracy for the completed scans, and instead compare using plausibility scores. Further, in order to quantitatively evaluate the performance of the network, we report numbers on a synthetic dataset (Dai et al., 2017b) where completed versions are available. Finally, we compare our method against baseline methods to demonstrate the advantages of the proposed unpaired scan completion framework.
sections.heading: 2 RELATED WORK
sections.text: Shape Completion. Many deep neural networks have been proposed to address the shape completion challenge. Inspired by CNN-based 2D image completion networks, 3D convolutional neural networks applied on voxelized inputs have been widely adopted for 3D shape completion task (Dai et al., 2018; 2017b; Sharma et al., 2016; Han et al., 2017; Thanh Nguyen et al., 2016; Yang et al., 2018; Wang et al., 2017). As quantizing shapes to voxel grids lead to geometric information loss, recent approaches (Yuan et al., 2018; Yu et al., 2018b; Achlioptas et al., 2018) operate directly on point sets to fill in missing parts. These works, however, require supervision in the form of partialcomplete paired data for training deep neural networks to directly regress partial input to their ground truth counterparts. Since paired ground truth of real-world data is rarely available such training data is generated using virtual scanning. While the methods work well on synthetic test data, they do not generalize easily to real scans arising from hard-to-model acquisition processes.
  Realizing the gap between synthetically-generated data and real-world data, Stutz & Geiger (2018) proposed to directly work on voxelized real-world data. They also work in a latent space created for clean and complete data but measure reconstruction loss using a maximum likelihood estimator. Instead, we propose a GAN setup to learn a mapping between latent spaces respectively arising from partial real and synthetic complete data. Further, by measuring loss using Hausdorff distance on point clouds, we directly work with point sets instead of voxelized input.
  Generative Adversarial Network. Since its introduction, GAN (Goodfellow et al., 2014) has been used for a variety of generative tasks. In 2D image domain, researchers have utilized adversarial training to recover richer information from low-resolution images or corrupted images (Ledig et al., 2017; Wang et al., 2018; Mao et al., 2017; Park et al., 2018; Bulat et al., 2018; Yeh et al., 2017; Iizuka et al., 2017). In 3D context, Yang et al. (2018); Wang et al. (2017) combine 3D-CNN and generative adversarial training to complete shapes under the supervision of ground truth data. Gurumurthy & Agrawal (2019) treats the point cloud completion task as denoising AE problem, utilizing adversarial training to optimize on the AE latent space. We also leverage the power of GAN for reasoning the missing part of partial point cloud scanning. However, our method is designed to work with unpaired data, and thus can directly be applied to real-world scans even when real-world and synthetic data distributions differ. Intuitively, our GAN-based approach directly learns a translation mapping between these two different distributions.
  Deep Learning on Point clouds. Our method is built upon recent advances in deep neural networks for point clouds. PointNet Qi et al. (2017a), the pioneering work on this topic, takes an input point set through point-wise MLP layers followed by a symmetric and permutation-invariant func-
  tion to produce a compact global feature, which can then be used for a diverse set of tasks (e.g., classification, segmentation). Although many improvements to PointNet have been proposed (Su et al., 2018; Li et al., 2018b; Qi et al., 2017b; Li et al., 2018a; Zaheer et al., 2017), the simplicity and effectiveness of PointNet and its extension PointNet++ make them popular for many other analysis tasks (Yu et al., 2018a; Yin et al., 2018; Yu et al., 2018b; Guerrero et al., 2018).
  In the context of synthesis, Achlioptas et al. (2018) proposed an autoencoder network, using a PointNet-based backbone, to learn compact representations of point clouds. By working in a reduced latent space produced by the autoencoder, they report significant advantages in training GANs, instead of having a generator producing raw point clouds. Inspired by this work, we design a GAN to translate between two different latent spaces to perform unpaired shape completion on real scans.
sections.heading: 3 METHOD
sections.text: Given a noisy and partial point set S = {si} as input, our goal is to produce a clean and complete point setR = {ri} as output. Note that although the two sets have the same number of points, there is no explicit correspondence between the sets S and R. Further, we assume access to clean and complete point sets for shapes for the object classes. We achieve unpaired completion by learning two class-specific point set manifolds, Xr for the scanned inputs, and Xc for clean and complete shapes. Solving the shape completion problem then amounts to learning a mapping Xr → Xc between the respective latent spaces. We train a generator Gθ : Xr → Xc to perform the mapping. Note that we do not require the noise characteristics in the two data distributions, i.e., real and synthetic, to be the same. In absence of paired training data, we score the generated output by setting up a min-max game where the generator is trained to fool a discriminator Fχ, whose goal is to differentiate between encoded clean and complete shapes, and mapped encodings of the raw and partial inputs. Figure 2 shows the setup of the proposed scan completion network. The latent space encoder-decoders, the mapping generator, and the discriminator are all trained as detailed next.
sections.heading: 3.1 LEARNING LATENT SPACES FOR POINT SETS
sections.text: The latent space of a given set of point sets is obtained by training an autoencoder, which encodes the given input to a low-dimension latent feature and then decodes to reconstruct the original input. We work directly on the point sets via these learned latent spaces instead of quantizing them to voxel grids or signed distance fields.
  For point sets coming from the clean and complete point sets P , we learn an encoder network Ecη that maps P from the original parameter space R3N , defined by concatenating the coordinates of the N (2048 in all our experiments) points, to a lower-dimensional latent space Xc. A decoder network Dcφ performs the inverse transformation back to R3N giving us a reconstructed point set P̃ with also N points. The encoder-decoders are trained with reconstruction loss,
  LEMD(η, φ) = EP∼pcompleted(P, Dcφ(Ecη(P))), (1) where P ∼ pcomplete denotes point set samples drawn from the set of clean and complete point sets, d(X1, X2) is the Earth Mover’s Distance (EMD) between point sets X1, X2, and (η, φ) are the
  input point set completion without HL completion with HL
  Figure 3: Effect of unpaired scan completion without (Equation 5) and with HL term (Equation 6). Without the HL term, the network produces a clean point set for a complete chair, that is different in shape from the input. With the HL term, the network produces a clean point set that matches the input.
  learnable parameters of the encoder and decoder networks, respectively. Once trained, the weights of both networks are held fixed and the latent code z = Ecη(X), z ∈ Xc for a clean and complete point setX provides a compact representation for subsequent training and implicitly captures the manifold of clean and complete data. The architecture of the encoder and decoder is similar to Achlioptas et al. (2018); Qi et al. (2017a): using a 5-layer MLP to lift individual points to a deeper feature space, followed by a symmetric function to maintain permutation invariance. This results in a kdimensional latent code that describes the entire point cloud (k=128 in all our experiments). More details of the network architecture can be found in the appendix.
  As for the point set coming from the noisy-partial point sets S, one can also train another encoder Erγ : S → Xr and decoder Drψ : Xr → S̃ pair that provides a latent parameterization Xr for the noisy-partial point sets, with the definition of the reconstruction loss as,
  LEMD(γ, ψ) = ES∼prawd(S, Drψ(Erγ(S))), (2) where S ∼ praw denotes point set samples drawn from the set of noisy and partial point sets. Although, in experiments, the latent space of this autoencoder trained on noisy-partial point sets works considerably well as the noisy-partial point set manifold, we found that using the latent space produced by feeding noisy-partial point sets to the autoencoder trained on clean and complete point sets yields slightly better results. Hence, unless specified, we set γ = η and ψ = φ in our experiments. The comparison of different choices to obtain the latent space for noisy-partial point sets is also presented in Section 4. Next, we will describe the GAN setup to learn a mapping between the latent spaces of raw noisy-partial and synthetic clean-complete point sets, i.e., Xr → Xc.
sections.heading: 3.2 LEARNING A MAPPING BETWEEN LATENT SPACES
sections.text: We set up a min-max game between a generator and a discriminator to perform the mapping between the latent spaces. The generator Gθ is trained to perform the mapping Xr → Xc such that the discriminator fails to reliably tell if the latent variable comes from original Xc or the remapped Xr.
  The latent representation of a noisy and partial scan zr = Erγ(S) is mapped by the generator to z̃c = Gθ(zr). Then, the task of the discriminator Fχ is to distinguish between latent representations z̃c and zc = Ecη(P). We train the mapping function using a GAN. Given training examples of clean latent variables zc and remapped-noisy latent variables z̃c, we seek to optimize the following adversarial loss over the mapping generator Gθ and a discriminator Fχ,
  min θ max χ
  Ex∼pclean-complete [ log ( Fχ ( Ecη(x) ))] + Ey∼pnoisy-partial [ log ( 1− Fχ ( Gθ(E r γ(y) ))] . (3)
  In our experiments, we found the least square GAN Mao et al. (2016) to be easier to train and hence minimize both the discriminator and generator losses defined as,
  LF (χ) = Ex∼pclean-complete [ Fχ ( Ecη(x) ) − 1 ]2 + Ey∼pnoisy-partial [ Fχ ( Gθ(E r γ(y)) )]2 (4)
  LG(θ) = Ey∼pnoisy-partial [ Fχ ( Gθ(E r γ(y)) ) − 1 ]2 . (5)
  The above setup encourages the generator to perform the mapping Xr → Xc resulting in Dcψ(z̃c) to be a clean and complete point cloud R. However, the generator is free to map a noisy latent vector to any point on the manifold of valid shapes in Xc, including shapes that are far from the original partial scan S. As shown in Figure 3, the result is a complete and clean point cloud that can be dissimilar in shape to the partial scanned input. To prevent this, we add a reconstruction loss term Lrecon to the generator loss:
  LG(θ) = αEy∼pnoisy-partial [ Fχ ( Gθ(E r γ(y)) ) − 1 ]2 + βLHLrecon(S, Dcψ(Gθ(Erγ(S)))), (6)
  whereLHLrecon denotes the Hausdorff distance loss 1 (HL) from the partial input point set to the completion point set, which encourages the predicted completion point set to match the input only partially. Note that, it is crucial to use HL as Lrecon, since the partial input can only provide partial supervision when no ground truth complete point set is available. In contrast, using EMD as Lrecon forces the network to reconstruct the overall partial input leading to worse completion results. The comparison of these design choices is presented in Section 4. Unless specified, we set the trade-off parameters as α = 0.25 and β = 0.75 in all our experiments.
sections.heading: 4 EXPERIMENTAL EVALUATION
sections.text: We present quantitative and qualitative experimental results on several noisy and partial datasets. First, we present results on real-world datasets, demonstrating the effectiveness of our method on unpaired raw scans. Second, we thoroughly compare our method to various baseline methods on 3D-EPN dataset, which contains simulated partial scans and corresponding ground truth for full evaluation. Finally, we derive a synthetic noisy-partial scan dataset based on ShapeNet, on which we can evaluate the performance degradation of applying supervised methods to test data of different distribution and the performance of our method under varying levels of incompleteness. A set of ablation studies is also included to evaluate our design choices.
  Datasets. (A) Real-world dataset comes from three sources. First, a dataset of ∼550 chairs and ∼550 tables extracted from the ScanNet dataset split into 90%-10% train-test sets. Second, a dataset of 20 chairs and 20 tables extracted from the Matterport3D dataset. Note that we train our method only on the ScanNet training split, and use the trained model to test on the Matterport3D data to evaluate generalization to new data sources. Third, a dataset containing cars from the KITTI Velodyne point clouds. (B) 3D-EPN dataset provides simulated partial scans with corresponding ground truth. Scans are represented as Signed Distance Field (SDF). We only use the provided point cloud representations of the training data, instead of using the SDF data which holds richer information. (C) Clean and complete point set dataset contains virtually scanned point sets of ShapeNet models covering 8 categories, namely boat, car, chair, dresser, lamp, plane, sofa, and table. We use this dataset for learning the clean-complete point set manifold in all our experiments. (D) Synthetic dataset provides different incomplete scan distribution and at different levels of incompleteness. Ground truth complete scan counterparts are available for evaluation.
  Evaluation measures. We assess completion quality using the following measures. (A) Accuracy measures the fraction of points in Pcomp that are matched by Pgt, where and Pcomp denote the completion point set and Pgt denote the ground truth point set. Specifically, for each point v ∈ Pcomp, we compute D(v, Pgt) = min{‖ v − q ‖, q ∈ Pgt}. If D(v, Pgt) is within distance threshold = 0.03, we count it as a correct match. The fraction of matched points is reported as the accuracy in percentage. (B) Completeness reports the fraction of points in Pgt that are within distance threshold of any point in Pcomp. (C) F1 score is defined as the harmonic average of the accuracy and the completeness, where F1 reaches its best value at 1 (perfect accuracy and completeness) and worst at 0. (D) Plausibility of the completion is evaluated as the classification accuracy in percentage produced by PointNet++, a SOA point-based classification network. To avoid bias on ShapeNet point clouds, we trained the classification network on the ModelNet40 dataset. We mainly used plausibility score for real-world data completions, where no ground truth data is available for calculating accuracy, completeness, or F1 scores.
  In the following, we show all experimental and evaluation results. We trained separate networks for each category. More details are in the appendix.
sections.heading: 4.1 EXPERIMENTAL RESULTS ON REAL-WORLD DATA
sections.text: Our method works directly on real-world data where no paired data is available. We train and test our network on noisy-partial chairs and tables extracted from the ScanNet dataset. We further test the network trained on ScanNet dataset on chairs and tables extracted from the Matterport3D dataset, to show how well our network can generalize to definitely unseen data. We present qualitative results of our method in Fig 4. Our method consistently produces plausible completions for the ScanNet and Matterport3D data.
  1Unless specified, we use unidirectional Hausdorff distance, from the partial input to its completion.
  In the absence of ground truth completions on real data, we compare our method quantitatively against others based on the plausibility of the results. The left sub-table of Table 1 shows that our method is superior to those supervised methods, namely 3D-EPN and PCN. Directly applying PCN trained on simulated partial data to real-world data leads to completions that have low plausibility, while our method consistently produces results with high plausibility. 3D-EPN trained on simulated partial data failed to complete the real-world partial scans. In Section 4.2 and Section 4.3, we present more in-depth comparisons on 3D-EPN and our synthetic dataset, where the ground truth is available for computing accuracy, completeness, and F1 of the completions.
  Completing the car observations from KITTI is extremely challenging, as each car instance only receives few data points from the Lidar scanner. Fig 4 shows the qualitative results of our method on completing sparse point sets of KITTI cars, we can see that our network can still generate highly plausible cars with such sparse inputs.
  We also use a point-based object part segmentation network (Qi et al., 2017b) to indirectly evaluate our completions of real-world data. Due to the absence of ground truth segmentation, we calculate the approximate segmentation accuracy for each completion. Specifically, for the completion of a chair, we count the predicted segmentation label of each point to be correct as long as the predicted label falls into the set of 4 parts (i.e., seat, back, leg, and armrest) of chair class. Our completion results have much higher approximate segmentation accuracy compared to the real-world raw input (chair: 77.2% vs. 24.8%; table: 96.4% vs. 83.5%; and car: 98.0% vs. 5.2%, as segmentation accuracy on our completions versus on original partial input), indicating high completion quality.
sections.heading: 4.2 COMPARISON WITH BASELINES ON 3D-EPN DATA
sections.text: We compare our method to several baseline methods and present both quantitative and qualitative comparisons on the 3D-EPN test set:
  • Autoencoder (AE), which is trained only with clean and complete point sets. • 3D-EPN (Dai et al., 2017b), a supervised method that requires SDF input and is trained
  with paired data. We convert its Distance Field representation results into surface meshes, from which we can uniformly sample N points for calculating our point-based measures.
sections.heading: 4.3 EFFECT OF DATA DISTRIBUTION DISCREPANCY AND VARYING INCOMPLETENESS
sections.text: Supervised methods assume that simulated partial scans share the same data distribution as the test data. We conduct quantitative experiments to show that applying 3D-EPN and PCN to our synthetic data, which is of different data distribution to its training data and in which the ground truth complete scans are not available for training, lead to performance degradation. The right sub-table of Table 1 shows that our method continues to produce good completions on our synthetic data, as we do not require paired data for training. The visual comparison is presented in the appendix.
  To evaluate our method under different levels of input incompleteness, we conduct experiments on our synthetic data, in which we can control the fraction of missing points. Specifically, we train our network with varying levels of incompleteness by randomizing the amount of missing points during training, and afterwards fix the amount of missing points for testing. Table 3 shows the performance of our method on different classes under increasing amount of incompleteness and the comparison to AE. We can see that AE performance declines dramatically as the incompleteness of the input increases, while our method can still produce completions with high plausibility and F1 score.
sections.heading: 4.4 DIVERSITY OF THE COMPLETION RESULTS
sections.text: Our network is encouraged to partially match the input, alleviating the mode-collapse issue, which often occurs in GAN. Unlike traditional generative model problem, where high diversity in generated results is always better, the diversity in our completion results should match that of the ground truth. Although this can be qualitatively accessed, see Fig. 5 in Appendix, in order to quantitatively quantify the divergence, we compute the Jensen-Shannon Divergence (JSD) between the marginal distribution of ground truth point sets and that of our completions as proposed in Achlioptas et al. (2018). As a reference, we simulate extremely mode-collapsed point cloud sets by repeating a randomly selected point cloud, then report the JSD between ground truth point sets and the simulated extremely mode-collapsed point sets. The JSD scores – lower is better – highlight the diversity of our 3D-EPN completions and the divergence between our diversity and that of the ground truth using the extreme mode-collapse results as reference (the former is ours): 0.06 vs. 0.46 on cars, 0.05 vs. 0.61 on chairs, 0.04 vs. 0.53 on planes and 0.04 vs. 0.59 on tables.
sections.heading: 4.5 ABLATION STUDY
sections.heading: 5 CONCLUSION
sections.text: We presented a point-based unpaired shape completion framework that can be applied directly on raw partial scans to obtain clean and complete point clouds. At the core of the algorithm is an adaptation network acting as a generator that transforms latent code encodings of the raw point scans, and maps them to latent code encodings of clean and complete object scans. The two latent spaces regularize the problem by restricting the transfer problem to respective data manifolds. We extensively evaluated our method on real scans and virtual scans, demonstrating that our approach consistently leads to plausible completions and perform superior to other methods. The work opens up the possibility of generalizing our approach to scene-level scan completions, rather than object-specific completions. Our method shares the same limitations as many of the supervised counterparts: does not produce fine-scale details and assumes input to be canonically oriented. Another interesting future direction will be to combine point- and image-features to apply the completion setup to both geometry and texture details.
sections.heading: 6 ACKNOWLEDGEMENTS
sections.text: We thank all the anonymous reviewers for their insightful comments and feedback. This work is supported in part by grants from National Key R&D Program of China (2019YFF0302900), China Scholarship Council, National Natural Science Foundation of China (No.61602273), ERC Starting Grant, ERC PoC Grant, Google Faculty Award, Royal Society Advanced Newton Fellowship, and gifts from Adobe.
sections.heading: A DETAILS OF DATASETS
sections.text: Clean and Complete Point Sets are obtained by virtually scanning the models from ShapeNet. We use a subset of 8 categories, namely boat, car, chair, dresser, lamp, plane, sofa and table, in our experiments. To generate clean and complete point set of a model, we virtually scan the models by performing ray-intersection test from cameras placed around the model to obtain the dense point set, followed by a down-sampling procedure to obtain a relatively sparser point set of N points. Note that we use the models without any pose and scale augmentation.
  This dataset is used for training to learn the clean-complete point set manifold in all our experiments. The following datasets of different data distributions serve as different noisy-partial input data.
  Real-world Data comes from three sources. The first one is derived from ScanNet dataset which provides many mesh objects that have been pre-segmented from its surrounding environment. For the purpose of training and testing our network, we extract ∼550 chair objects and ∼550 table objects from ScanNet dataset, and manually align them to be consistently orientated with models in ShapeNet dataset. We also split these objects into 90%/10% train/test sets.
  The second one consists of 20 chairs and 20 tables from the Matterport3D dataset, to which the same extraction and alignment as is done in ScanNet dataset is also applied. Note that we train our method only on ScanNet training split, and use the trained model to test on Matterport3D data, to show how our method can generalize to absolutely unseen data. For both ScanNet and Matterport3D datasets, we uniformly sample N points on the surface mesh of each object to obtain the input point sets.
  Last, we extract car observations from the KITTI dataset using the provided ground truth bounding boxes for training and testing our method. We use KITTI Velodyne point clouds from the 3D object detection benchmark and the split of Qi et al. (2018). We filter the observations such that each car observation contains at least 100 points to avoid overly sparse observations.
  3D-EPN Dataset provides partial reconstructions of ShapeNet objects (8 categories) by using volumetric fusion method Curless & Levoy (1996) to integrate depth maps scanned along a virtual scanning trajectory around the model. For each model, a set of trajectories is generated with different levels of incompleteness, reflect the real-world scanning with a hand-held commodity RGB-D sensor. The entire dataset covers 8 categories and a total of 25590 object instances (the test set is composed of 5384 models). Note that, in the original 3D-EPN dataset, the data is represented in Signed Distance Field (SDF) for training data and Distance Field (DF) for test data. As our method works on pure point sets, we only use the point cloud representations of the training data provided by the authors, instead of using the SDF data which holds richer information and is claimed in Dai et al. (2017b) to be crucial for completing partial data.
  Synthetic Data serves the purpose of having another dataset of different incomplete scan distribution and controlling the incompleteness of the input. We use ShapeNet to generate a synthetic dataset, in which we can control the incompleteness of the synthetic partial point sets. For the models in each one of the 4 categories (car, chair, plane, and table), we split them into 90%/10% train/test sets. For each model, from which a clean and complete point set has been scanned (as described earlier in this subsection), we can randomly pick a point and remove its N × r (r ∈ [0, 1)) nearest neighbor points. The parameter r controls the incompleteness of the synthetically-generated input. Furthermore, we add Gaussian noise N (µ, σ2) to each point (µ=0 and σ=0.01 for all our experiments). Last, we duplicate the points in the resulting point sets to generate point sets with an equal number of N points.
sections.heading: B NETWORK ARCHITECTURE DETAILS
sections.text: In this section, we describe the details of the encoder, decoder, generator and discriminator in our network implementation.
  B.1 AE ARCHITECTURE DETAILS
  Encoder consists of 5 1-D convolutional layers which are implemented as 1-D convolutions with ReLU and batch normalization, with kernel size of 1 and stride of 1, to lift the feature of each point to high dimensional feature space independently. In all experiments, we use an encoder with 64,
  128, 128, 256 and k = 128 filters in each of its layers, with k being the latent code size. The output of the last convolutional layer is passed to a feature-wise maximum to produce a k-dimensional latent code.
  Decoder transforms the latent vector using 3 fully connected layers with 256, 256, and N x 3 neurons each, the first two having ReLUs, to reconstruct N x 3 output.
  B.2 GAN ARCHITECTURE DETAILS
  Since the generator and discriminator of GAN directly operate on the latent space, the architecture for them is significantly simpler. Specifically, the generator is comprised of two fully connected layers with 128 and 128 neurons each, to map the latent code of noisy and incomplete point sets to that of clean and complete point sets. The discriminator consists of 3 fully connected layers with 256, 512 and 1 neurons each, to produce a single scalar for each latent code.
sections.heading: C TRAINING DETAILS
sections.text: To make the training of the entire network trackable, we pre-train the AEs used for obtaining the latent spaces. After that we retain the weights of AEs, only the weights of the generator and discriminator are updated through the back-propagation during the GAN training. The following training hyper-parameters are used in all our experiments.
  For training the AE, we use Adam optimizer with an initial learning rate of 0.0005, β1 = 0.9 and a batch size of 200 and train for a maximum of 2000 epochs.
  For training the generator and discriminator on the latent spaces, we use Adam optimizer with an initial learning rate of 0.0001, β1 = 0.5 and a batch size of 24 and train the generator and discriminator alternately for a maximum of 1000 epochs.
sections.heading: D QUALITATIVE RESULTS ON 3D-EPN DATASET
sections.text: We present qualitatively comparisons in Fig 5, where we show the partial input, AE, 3D-EPN, PCN, Ours, Ours+ result and the ground truth point set. We can see that, although our method is not quantitatively the best, our results are very qualitatively plausible, as the generator is restricted to generate point sets from learned clean and complete shape manifolds.
  E VISUAL COMPARISON ON TEST DATA WITH DISTRIBUTION DIFFERENT TO TRAINING DATA
  We present the visual comparison of 3D-EPN, PCN and our method on our synthetic data, which differs from 3D-EPN and PCN training data. In this experiment, the ground truth of our synthetic data is only available for evaluation. In Fig 6, we can see that our method keeps producing high-quality completions, as our method does not require paired data for training hence can still be trained when no ground truth is available. 3D-EPN and PCN produce much worse results as the data distribution of its training data and our synthetic data differ.
  F VARIATIONS OF LEVERAGING GROUND TRUTH SUPERVISION
  To adapt our network for training with ground truth point sets, we first change the reconstruction loss term in the generator loss from HD (Hausdorff Distance) to EMD (Earth Mover’s Distance), as the ground truth point set is complete and thus contains full information for supervising the completion. Note that HD is superior when the ground truth is unavailable for training, as shown in Table 4 of Section 4. Moreover, we present the comparison of different decisions on whether to adopt the adversarial training in our network for training with the ground truth.
sections.heading: G MORE STATISTICS FRO THE BASELINE COMPARISON
sections.text: For the baseline methods comparison on 3D-EPN dataset, we also report the Chamfer distance (CD), Earth Mover’s Distance (EMD) and Hausdorff Distance (HD, maximum of the two directional distances) between the ground truth and the completion in Table 6:
sections.heading: H USER STUDY ON REAL-WORLD DATA COMPLETION
sections.text: We also conducted a user study on the completion results of the real-world scans, where given a partial input users are required to pick the most preferable completion among EPN, PCN and our results. In total, we received 1,000 valid user selections and report the preference (in percentage of the total selections) of each method in the user study. From Fig. 7 we can see that in over half (52%) of the selections, our completion results are selected as the best completion, while PCN completion results are better in 45% of the selections.
  Interestingly, although our method outperforms other methods in the user study, our method clearly does not hold a dominant position in this user study. After a more in-depth analysis of the user selections, we found that users intend to pick the completion in which the partial input is embedded, which can be formulated as the Hausdorff distance from the partial input to the completion, and the supervised method PCN preserves the input point cloud in its completion output as this always minimizes the distance loss. The plausibility of the completion result is usually neglected by users,
  while the plausibility and the Hausdorff distance from the partial input to the completion are both considered as two trade-off terms in the objective function of our completion generator. Fig. 8 gives a typical example, where the PCN completion without clear chair structure is often picked as better completion while our completion tries to trade-off between the HL and the plausibility.
sections.heading: I GENERALIZATION TO UNSEEN CLASSES
sections.text: Our method does generalize to unseen objects from the same category in the test set, which is demonstrated in the experimental results section. However, our method intuitively should not generalize to classes that are not seen during the training, as the autoencoder, which is a fundamental component
  in our network, does not generalize to unseen classes. We present the qualitative results of applying our table completion network on chair and airplane class. We can see that from Fig. 9, on the unseen chair class, which shares similar structure with table, our table completion network can produce some reasonable structures, but is unable to complete with a seat back as the autoencoder does not have such capability; on the unseen airplane class, which is rather dissimilar to table class, our table completion network failed to complete the partial airplanes.