As 3D scanning solutions become increasingly popular, several deep learning setups have been developed for the task of scan completion, i.e., plausibly filling in regions that were missed in the raw scans. These methods, however, largely rely on supervision in the form of paired training data, i.e., partial scans with corresponding desired completed scans. While these methods have been successfully demonstrated on synthetic data, the approaches cannot be directly used on real scans in the absence of suitable paired training data. We develop a first approach that works directly on input point clouds, does not require paired training data, and hence can directly be applied to real scans for scan completion. We evaluate the approach qualitatively on several real-world datasets (ScanNet, Matterport3D, KITTI), quantitatively on 3D-EPN shape completion dataset, and demonstrate realistic completions under varying levels of incompleteness.
This paper proposes a new method for making 3D point clouds by automatically completing 3D scans. It does not require paired data samples for training which makes it possible to train it on real data instead of synthetic data. The authors use a generative adversarial network (GAN) to “generate” complete point clouds from noisy or partial point clouds obtained by 3D scanning. The generator learns to perform mapping from point set manifold of scanned noisy and partial input X_r to manifold of clean shapes X_c. The discriminator tries to tell between encoded clean shapes (synthetic data point clouds) and mappings of noisy input (point clouds from real-life data 3D scans). 
An encoder-decoder network similar to those in Achlioptas et al. (2018) and Qi et al. (2017a) is trained to transform original point clouds to a low-dimensional latent space prior to training the GAN. The authors find that using the encoder-decoder trained on clean shape data even for noisy input yields better results. 
One of the issues of completing noisy and partial scans is that the desired complete scan can have a very different shape compared to the noisy input. The generator can map latent vectors to any points on the target manifold which allows it to generate shapes that are far different from the original inputs. In order not to generate random clean shapes, the authors add a reconstruction loss to the generator which encourages it to preserve the partial shape of the input end reconstruct it in the completed clean shape. The choice of Hausdorff distance for reconstruction loss is sound and the ablation study confirms it. 
The authors perform rather extensive experimental evaluation of their proposed method. They perform qualitative and quantitative analysis on several datasets, both real-life and synthetic ones. The proposed method outperforms existing methods in real-life data scenario. On the synthetic dataset (3D-EPN), the quantitative results are not as good as those of PCN (which is a supervised method, unlike the proposed method), but the qualitative results look plausible and comparable to PCN results. In terms of plausibility score, the proposed method outperforms existing methods in all experiments, which is probably thanks to its objective - map the input into the latent space of clean and complete shapes. However, plausible looking point clouds do not necessarily have to precisely match the input, which is the objective of 3D scan point cloud completion. 
The ablation study also confirms the effectiveness of individual parts in the proposed method. 
The main contribution of this paper is training with unpaired data, which enables training on real-life data, leading to better results on real-life scans. While I understand the difficulties, I believe it would be better to try to focus more on real-life data in the evaluation. There is only one experiment with quantitative analysis on real-life data in the paper. Supporting that with a visual Turing test would have been great. 
 
Questions raised:
In 4.3, you say that “ground truth complete scans are not available for training.” How do you then train the supervised methods? Are they trained on 3D-EPN? In that case, is your proposed method also trained on 3D-EPN? If your method is the only one trained on your synthetic data (dataset D), which is also used for testing, then I do not think you could claim that your method is better at dealing with different data distributions. Please make clear in the comments what data you use for training in this experiment. 
The meaning of section 4.4 is not very clear to me. If my understanding is correct, when completing noisy or partial point clouds, low diversity in results is better than high diversity because it is a task of repairing the input, which only has one correct result. Are you trying to say in this section that your method yields more consistent results with lower diversity than the method from previous work? If that is the case, you should consider rewriting that part to make it clearer to readers. 
The PCN paper (Yuan et al., 2018) shows that PCN can generalize and complete point clouds of objects unseen during training very well. Unfortunately, this paper does not discuss performance on objects of unseen classes. Were such experiments considered? It would be beneficial to do a qualitative analysis of performance on unseen objects. 
The description of the left table in Table 1 first says that it shows performance on real-life scans, but performance on synthetic data also appears there. Shouldn’t it rather say that it is plausibility comparison on real-life scans and synthetic data?
Summary:
The proposed method is easy to understand, exploits recent progress in generative models, and allows training on real-life scans as it does not require paired training data. 
The paper does not contain much algorithmical novelty and mostly combines existing methods to solve the problem of obtaining clean and complete point clouds from real 3D scans. However, the ablation study shows that adding a good reconstruction loss to the generator is crucial for the performance. 
The main strengths of this paper are extensive experimental evaluation using both quantitative and qualitative analysis, and significant improvement in performance on real-life data over previous works. 
The main weakness is limited novelty in terms of the techniques used in the proposed method. 
Additional comments that do not affect the review decision:
Citations in the text have to be revised and correctly put into parentheses where necessary. E.g., on page 2: “Since its introduction, GAN Goodfellow et al. (2014) has been used...” should be rewritten to “Since its introduction, GAN (Goodfellow et al., 2014) has been used...”
Table 2 is the only table where the best results are not highlighted by bold text. It is also the only table where the proposed method is not the best performing method as it loses to PCN. I apologize if it is just a pure coincidence but it seems as if the authors did not want to draw attention to the fact that their proposed method loses to an existing method in that experiment. I believe that you should be fair and highlight best results in all tables.