summaries: This paper introduces a supervised neural network predicting a wireframe structure from a 3D point cloud. The network takes a raw unordered 3D point cloud as input, processes it using FCGF architecture, and predicts three types of information: vertex existence in each patch, vertex location, and edge existence for each pair of vertices. In the experiments, the network is evaluated with two datasets, a subset of the ABC dataset and a set of 3D models from Google 3D warehouse. Also, it is compared with the baseline methods using four evaluation metrics, which are created to assess the accuracy of the predicted vertices, edges, and the overall wireframe graph structure. The results demonstrate the outperformance of the proposed method quantitatively and qualitatively. This paper presents a deep architecture to extract a wireframe model from a 3D point cloud. This is a problem of high interest, and the author claim that the approach they present is the first one to address this task, which is true to the best of my knowledge. Since both the approach and the evaluation are sound, this alone seem to warrant publication. There are however several weaknesses in the paper, and my accept recommendation is conditional to clear answer on each of them: This paper introduces PC2WF, a neural network that turns 3D point clouds into a wireframe model. PC2WF encodes each point into a feature vector and uses them to predict the candidate corners. After that, line proposals are generated by connecting pairs of corners, and the point features along each line are pooled into its confidence value. By pruning the proposed lines, PC2WF generates the final wireframe represesntation.
abstractText: We introduce PC2WF, the first end-to-end trainable deep network architecture to convert a 3D point cloud into a wireframe model. The network takes as input an unordered set of 3D points sampled from the surface of some object, and outputs a wireframe of that object, i.e., a sparse set of corner points linked by line segments. Recovering the wireframe is a challenging task, where the numbers of both vertices and edges are different for every instance, and a-priori unknown. Our architecture gradually builds up the model: It starts by encoding the points into feature vectors. Based on those features, it identifies a pool of candidate vertices, then prunes those candidates to a final set of corner vertices and refines their locations. Next, the corners are linked with an exhaustive set of candidate edges, which is again pruned to obtain the final wireframe. All steps are trainable, and errors can be backpropagated through the entire sequence. We validate the proposed model on a publicly available synthetic dataset, for which the ground truth wireframes are accessible, as well as on a new real-world dataset. Our model produces wireframe abstractions of good quality and outperforms several baselines.
authors.name: Yujia Liu
authors.name: Stefano D’Aronco
authors.name: Konrad Schindler
authors.name: Jan Dirk Wegner
references.authors: Syeda Mariam Ahmed Yan Zhi Tan Chee Meng Chew Abdullah Al Mamun Fook Seng Wong
references.title: Edge and corner detection for unorganized 3d point clouds with application to robotic welding
references.venue: In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS),
references.authors: Dena Bazazian Josep R Casas Javier Ruiz-Hidalgo
references.title: Fast and robust edge extraction in unorganized point clouds
references.venue: In Digital Image Computing: Techniques and Applications (DICTA),
references.authors: Siheng Chen Dong Tian Chen Feng Anthony Vetro Jelena Kovačević
references.title: Fast resampling of threedimensional point clouds via graphs
references.venue: IEEE Transactions on Signal Processing,
references.authors: Otfried Cheong Joachim Gudmundsson Hyo-Sil Kim Daria Schymura Fabian Stehn
references.title: Measuring the similarity of geometric graphs
references.venue: In International Symposium on Experimental Algorithms,
references.authors: Christopher Choy JunYoung Gwak Silvio Savarese
references.title: 4d spatio-temporal convnets: Minkowski convolutional neural networks
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Christopher Choy Jaesik Park Vladlen Koltun
references.title: Fully convolutional geometric features
references.venue: In IEEE International Conference on Computer Vision (ICCV),
references.authors: Inge Coudron Steven Puttemans Toon Goedemé
references.title: Polygonal reconstruction of building interiors from cluttered pointclouds
references.venue: In European Conference on Computer Vision (ECCV),
references.authors: Kris Demarsin Denis Vanderstraeten Tim Volodine Dirk Roose
references.title: Detection of closed sharp edges in point clouds using normal estimation and graph theory
references.venue: Computer-Aided Design,
references.authors: Tao Du Jeevana Priya Inala Yewen Pu Andrew Spielberg Adriana Schulz Daniela Rus Armando Solar-Lezama Wojciech Matusik
references.title: InverseCSG: Automatic conversion of 3d models to CSG trees
references.venue: ACM Transactions on Graphics (TOG),
references.authors: Alexandre Durupt Sebastien Remy Guillaume Ducellier Benoît Eynard
references.title: From a 3d point cloud to an engineering CAD model: a knowledge-product-based approach for reverse engineering
references.venue: Virtual and Physical Prototyping,
references.authors: Hao Fang Florent Lafarge Mathieu Desbrun
references.title: Planar shape detection at structural scales
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition
references.authors: Przemysław Głomb
references.title: Detection of interest points on 3d data: Extending the Harris operator
references.venue: In Computer Recognition Systems
references.authors: Diego Gonzalez-Aguilera Susana Del Pozo Gemma Lopez Pablo
references.title: Rodriguez-Gonzalvez. From point cloud to CAD models: Laser and optics geotechnology for the design of electrical substations
references.venue: Optics & Laser Technology,
references.authors: Timo Hackel Jan Dirk Wegner Konrad Schindler
references.title: Contour detection in unstructured 3d point clouds
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition
references.authors: Timo Hackel Jan Dirk Wegner Konrad Schindler
references.title: Joint classification and contour extraction of large 3d point clouds
references.venue: ISPRS Journal of Photogrammetry and Remote Sensing,
references.authors: Christopher G Harris Mike Stephens
references.title: A combined corner and edge detector
references.venue: In Alvey Vision Conference,
references.authors: Kun Huang Shenghua Gao
references.title: Wireframe parsing with guidance of distance map
references.venue: IEEE Access,
references.authors: Kun Huang Yifan Wang Zihan Zhou Tianjiao Ding Shenghua Gao Yi Ma
references.title: Learning to parse wireframes in images of man-made environments
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition
references.authors: Jaehoon Jung Sungchul Hong Sanghyun Yoon Jeonghyun Kim Joon Heo
references.title: Automated 3d wireframe modeling of indoor structures from point clouds using constrained least-squares adjustment for as-built BIM
references.venue: Journal of Computing in Civil Engineering,
references.authors: Sebastian Koch Albert Matveev Zhongshi Jiang Francis Williams Alexey Artemov Evgeny Burnaev Marc Alexa Denis Zorin Daniele Panozzo
references.title: ABC: A big CAD model dataset for geometric deep learning
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Jun Li Kai Xu Siddhartha Chaudhuri Ersin Yumer Hao Zhang Leonidas Guibas
references.title: GRASS: Generative recursive autoencoders for shape structures
references.venue: ACM Transactions on Graphics (TOG),
references.authors: Lingxiao Li Minhyuk Sung Anastasia Dubrovina Li Yi Leonidas J Guibas
references.title: Supervised fitting of geometric primitives to 3d point clouds
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition
references.authors: Yan-qin Li Gui-zhong Xie Fan-nian Meng De-hai Zhang
references.title: Using the point cloud data to reconstructing CAD model by 3d geometric modeling method in reverse engineering
references.venue: In International Conference on Manufacturing Engineering and Intelligent Materials (ICMEIM),
references.authors: Hong-Wei Lin Chiew-Lan Tai Guo-Jin Wang
references.title: A mesh reconstruction algorithm driven by an intrinsic property of a point cloud
references.venue: Computer-Aided Design,
references.authors: Liangliang Nan Peter Wonka
references.title: Polyfit: Polygonal surface reconstruction from point clouds
references.venue: In IEEE International Conference on Computer Vision (ICCV),
references.authors: Fabio Remondino
references.title: From point cloud to surface: the modeling and visualization problem. International Archives of the Photogrammetry
references.venue: Remote Sensing and Spatial Information Sciences,
references.authors: Shaoqing Ren Kaiming He Ross Girshick Jian Sun
references.title: Faster R-CNN: Towards real-time object detection with region proposal networks
references.venue: In Advances in Neural Information Processing Systems (NeurIPS),
references.authors: Ruwen Schnabel Roland Wahl Reinhard Klein
references.title: Efficient RANSAC for point-cloud shape detection
references.venue: Computer Graphics Forum,
references.authors: Ruwen Schnabel Raoul Wessel Roland Wahl Reinhard Klein
references.title: Shape recognition in 3d pointclouds
references.venue: In International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision (WSCG),
references.authors: Gopal Sharma Rishabh Goyal Difan Liu Evangelos Kalogerakis Subhransu Maji
references.title: CSGNet: Neural shape parser for constructive solid geometry
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition
references.authors: Ivan Sipiran Benjamin Bustos
references.title: A robust 3d interest points detector based on harris operator
references.venue: In Eurographics Conference on 3D Object Retrieval
references.authors: Ivan Sipiran Benjamin Bustos
references.title: Harris 3d: a robust extension of the Harris operator for interest point detection on 3d meshes
references.venue: The Visual Computer,
references.authors: Stephen M Smith J Michael Brady
references.title: SUSAN -— a new approach to low level image processing
references.venue: International Journal of Computer Vision,
references.authors: Shubham Tulsiani Hao Su Leonidas J Guibas Alexei A Efros Jitendra Malik
references.title: Learning shape abstractions by assembling volumetric primitives
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition
references.authors: Nicolas Walter Olivier Aubreton Yohan D Fougerolle Olivier Laligant. Susan
references.title: 3d operator, principal saliency degrees and directions extraction and a brief study on the robustness to noise
references.venue: In IEEE International Conference on Image Processing (ICIP),
references.authors: Peng-Shuai Wang Yang Liu Yu-Xiao Guo Chun-Yu Sun Xin Tong
references.title: O-CNN: Octree-based convolutional neural networks for 3d shape analysis
references.venue: ACM Transactions on Graphics (TOG),
references.authors: Xiaogang Wang Yuelang Xu Kai Xu Andrea Tagliasacchi Bin Zhou Ali Mahdavi-Amiri Hao Zhang
references.title: Pie-net: Parametric inference of point cloud
references.venue: edges. Conference on Neural Information Processing Systems (NeurIPS),
references.authors: Shaobo Xia Ruisheng Wang
references.title: A fast edge extraction method for mobile LiDAR point clouds
references.venue: IEEE Geoscience and Remote Sensing Letters,
references.authors: Nan Xue Tianfu Wu Song Bai Fu-Dong Wang Gui-Song Xia Liangpei Zhang Philip H.S. Torr
references.title: Holistically-attracted wireframe parsing
references.venue: arXiv preprint arXiv:2003.01663,
references.authors: Bisheng Yang Yufu Zang
references.title: Automated registration of dense terrestrial laser-scanning point clouds using curves
references.venue: ISPRS Journal of Photogrammetry and Remote Sensing,
references.authors: Lequan Yu Xianzhi Li Chi-Wing Fu Daniel Cohen-Or Pheng-Ann Heng
references.title: EC-Net: an edgeaware point set consolidation network
references.venue: In European Conference on Computer Vision (ECCV),
references.authors: Yichao Zhou Haozhi Qi Yi Ma
references.title: End-to-end wireframe parsing
references.venue: In IEEE International Conference on Computer Vision (ICCV),
references.authors: Yichao Zhou Haozhi Qi Yuexiang Zhai Qi Sun Zhili Chen Li-Yi Wei Yi Ma
references.title: Learning to reconstruct 3d Manhattan wireframes from a single image
references.venue: In IEEE International Conference on Computer Vision (ICCV),
references.authors: Chuhang Zou Ersin Yumer Jimei Yang Duygu Ceylan Derek Hoiem
references.title: 3D-PRNN: Generating shape primitives with recurrent neural networks
references.venue: In IEEE International Conference on Computer Vision (ICCV),
references.authors: Chuhang Zou Alex Colburn Qi Shan Derek Hoiem
references.title: LayoutNet: Reconstructing the 3d room layout from a single RGB image
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition
sections.heading: 1 INTRODUCTION
sections.text: Many practical 3D sensing systems, like stereo cameras or laser scanners, produce unstructured 3D point clouds. That choice of output format is really just a "smallest common denominator", the least committal representation that can be reliably generated with low-level signal processing. Most users would prefer a more efficient and more intuitive representation that describes the scanned object’s geometry as a compact collection of geometric primitives, together with their topological relations. Specifically, many man-made objects are (approximately) polyhedral and can be described by corners, straight edges and/or planar surfaces. Roughly speaking there are two ways to abstract a point cloud into a polyhedral model: either find the planar surfaces and intersect them to find the edges and corners, e.g., Schnabel et al. (2007); Fang et al. (2018); Coudron et al. (2018); or directly find the salient corner and/or edges, e.g., Jung et al. (2016); Hackel et al. (2016).
  A wireframe is a graph representation of an object’s shape, where vertices correspond to corner points (with high Gaussian curvature) that are linked by edges (with high principal curvature). Wireframes are a good match for polyhedral structures like mechanical parts, furniture or building interiors. In particular, since wireframes focus on the edge structure, they are best suited for piece-wise smooth objects with few pronounced crease edges (whereas they are less suitable for smooth objects without defined edges or for very rough ones with edges everywhere). We emphasise that wireframes are not only a "compression technique" to save storage. Their biggest advantage in many applications is that they are easy to manipulate and edit, automatically or interactively in CAD software, because they make the salient contours and their connectivity explicit. Reconstructed wireframes can drive and help to create 3D CAD models for manufacturing parts, metrology, quality inspection, as well as visualisation, animation, and rendering.
  Inferring the wireframe from a noisy point cloud is a challenging task. We can think of the process as a sequence of steps: find the corners, localise them accurately (as they are not contained in the point cloud), and link them with the appropriate edges. However, these steps are intricately correlated. For example, corner detection should "know" about the subsequent edge detection: curvature is affected by noise (as any user of an interest point detector can testify), so to qualify as a corner
  a 3D location should also be the plausible meeting point of multiple, non-collinear edges. Here we introduce PC2WF, an end-to-end trainable architecture for point cloud to wireframe conversion. PC2WF is composed of a sequence of feed-forward blocks, see Fig. 1 for the individual steps of the pipeline. First a "backbone" block extracts a latent feature encoding for each point. The next block identifies local neighbourhoods ("patches") that contain corners and regresses their 3D locations. The final block links the corners with an overcomplete set of plausible candidate edges, collects edge features by pooling point features along each edge, and uses those to prune the candidate set to a final set of wireframe edges. All stages of the architecture that have trainable parameters support back-propagation, such that the model can be learned end-to-end. We validate PC2WF with models from the publicly available ABC dataset of CAD models, and with a new furniture dataset collected from the web. The experiments highlight the benefit of explicit, integrated wireframe prediction: on the one hand, the detected corners are more complete and more accurate than with a generic corner detector; on the other hand, the reconstructed edges are more complete and more accurate than those found with existing (point-wise) edge detection methods.
sections.heading: 2 RELATED WORK
sections.text: Relatively little work exists on the direct reconstruction of 3D wireframes. We review related work for the subtasks of corner and edge detection, as well as works that reconstruct wireframes in 2D or polyhedral surface models and geometric primitives in 3D.
  Corner Detection. Corner points play a fundamental role in both 2D and 3D computer vision. A 3D corner is the intersection point of ≥3 (locally) non-collinear surfaces. Corners have a well-defined location and a local neighbourhood with non-trivial shape (e.g., for matching). Collectively the set of corners often is sufficient to tightly constrain the scale and shape of an object. Typical local methods for corner detection are based on the 2nd-moment matrix of the points (resp. normals) in a local neighbourhood (Głomb, 2009; Sipiran & Bustos, 2010; 2011), a straight-forward generalisation of the 2D Harris operator first proposed for 2D images (Harris et al., 1988). Also other 2D corner detectors like Susan (Smith & Brady, 1997) have been generalised to 3D (Walter et al., 2009).
  Point-level Edge Point Detection. A large body of work is dedicated to the detection of 3D points that lie on or near edges. The main edge feature is high local curvature (Yang & Zang, 2014). Related features, such as symmetry (Ahmed et al., 2018), change of normals (Demarsin et al., 2007), and eigenvalues (Bazazian et al., 2015; Xia & Wang, 2017) can also be used for edge detection. Hackel et al. (2016; 2017) use a combination of several features as input for a binary edge-point classifier. After linking the points into an adjacency graph, graph filtering can be used to preserve only edge points (Chen et al., 2017). EC-Net (Yu et al., 2018) is the first method we know of that detects edge points in 3D point clouds with deep learning.
  Learning Structured Models. Many methods have been proposed to abstract point clouds into more structured 3D models, e.g., triangle meshes (Lin et al., 2004; Remondino, 2003; Lin et al., 2004), polygonal surfaces (Fang et al., 2018; Coudron et al., 2018), simple parametric surfaces (Schnabel et al., 2007; 2008; Li et al., 2019). From the user perspective, the ultimate goal would be to reverseengineer a CAD model (Gonzalez-Aguilera et al., 2012; Li et al., 2017b; Durupt et al., 2008). Several works have investigated wireframe parsing in 2D images (Huang et al., 2018; Huang & Gao, 2019). LCNN (Zhou et al., 2019a) is an end-to-end trainable system that directly outputs a vectorised wireframe. The current state-of-the-art appears to be the improved LCNN of Xue et al. (2020), which uses a holistic "attraction field map" to characterise line segments. Zou et al. (2018) estimate the 3D layout of an indoor scene from a single perspective or panoramic image with an encoder-decoder architecture, while Zhou et al. (2019b) obtain a compact 3D wireframe representation from a single image by exploiting global structural regularities. Jung et al. (2016) reconstruct 3D wireframes of building interiors under a Manhattan-world assumption. Edges are detected heuristically by projecting the points onto 2D occupancy grids, then refined via least-squares fitting under parallelism/orthogonality constraints. Wireframe extraction in 2D images is simplified by the regular pixel grid. For irregular 3D points the task is more challenging and there has been less work.
  Another related direction of research is 3D shape decomposition into sets of simple, parameterised primitives (Tulsiani et al., 2017; Li et al., 2017a; Zou et al., 2017; Sharma et al., 2018; Du et al., 2018). Most approaches use a coarse volumetric occupancy grid as input for the 3D data, which
  works well for shape primitive fitting but only allows for rather inaccurate localisation of edges and vertices. Li et al. (2019) propose a method for fitting 3D primitives directly to raw point clouds. A limiting factor of their approach is that only a fixed maximum number of primitive instances can be detected. This upper bound is hard-coded in the architecture and prevents the network from fitting more complex structures. Nan & Wonka (2017) propose Polyfit to extract polygonal surfaces from a point cloud. Polyfit first selects an exhaustive set of candidate surfaces from the raw point cloud using RANSAC, and then refines the surfaces. Similarly to our PC2WF, Polyfit works best with objects made of straight edges and planar surfaces. The concurrent work Wang et al. (2020) proposed a deep neural network that is trained to identify a collection of parametric edges.
  In summary, most 3D methods are either hand-crafted for restricted, schematic settings, or only learn to label individual "edge points". Here we aim for an end-to-end learnable model that directly maps a 3D point cloud to a vectorised wireframe.
sections.heading: 3 METHOD
sections.text: Our full architecture is depicted in Fig. 1. In the following we describe each block and, where necessary, discuss training and inference procedures. We denote a point cloud of N 3D points with P = {pi}Ni=1 where pi ∈ R3 represents the 3D coordinates of point i. The 3D wireframe model of the point cloud P is denoted byW = (V, E) and it is defined, analogously to an undirected graph, by a set of vertices V which are connected by a set of edges E ⊆ V × V . As the vertices represent points in the 3D space we can think of them as being described by 3D coordinates vi ∈ V ⊂ R3. Overall Pipeline. Our feed-forward pipeline consists of four blocks: (1) a convolutional backbone that extracts per-point features; (2) a vertex detector that examines patches of the point cloud and decides which ones contain vertices; (3) a vertex localiser that outputs the location of the vertex within a patch; (4) an edge detector that instantiates candidate edges and prunes them to a wireframe.
  Backbone. PC2WF first extracts a feature vector per 3D point that encodes its geometric context. Our pipeline is compatible with any (hand-coded or learned) function that outputs a fixed-size descriptor vector per point. We use Fully Convolutional Geometric Features (FCGF, Choy et al., 2019b), which are compact and capture broad spatial context. The backbone operates on sparse tensors (Choy et al., 2019a) and efficiently computes 32-dimensional features in a single, 3D fully-convolutional pass.
  Vertex Detector. This block performs binary classification of local patches (neighbourhoods in the point cloud) into those (few) that contain a corner and those that do not. Only patches detected to contain a corner are passed on to the subsequent block. While one could argue that detecting the presence of a vertex and inferring its 3D coordinates are two facets of the same problem, we found it advantageous to separate them. Detecting first whether a patch is likely to contain a corner, then constructing its 3D coordinates reduces the computational cost, as we discard patches without corners early. Empirically it also reduces false positives and increases localisation accuracy.
  Our patch generation is similar to EC-Net (Yu et al., 2018). A patch contains a fixed number M of points, chosen by picking a patch centroid and finding the closest (M − 1) points in terms of geodesic distance on the underlying surface. The geodesic distance avoids "shortcuts" in regions with thin structures or high curvature. To approximate the geodesic distance, the entire point cloud is linked into a k-nearest neighbour graph, with edge weights proportional to Euclidean distance. Point-to-point distances are then found as shortest paths on that graph. During training we randomly draw positive samples that contain a corner and negative ones that do not. To increase the diversity of positive samples, the same corner can be included in multiple patches with different centroids. For the negative samples, we ensure to draw not only patches on flat surfaces, but also several patches near edges, by doing this negative samples with strong curvature are well represented. For inference, patch seeds are drawn using farthest point sampling. The vertex detector takes as input the (indexed) features of the M points in the patch, max-pooled per dimension to achieve permutation invariance, and outputs a scalar vertex/no_vertex probability. Its loss function Lpat is binary cross-entropy.
  Vertex Localiser. The vertex detector passes on a set of P patches predicted to contain a corner. The M points pi of such a patch form the input for the localisation block. The backbone features fi ∈ RC and the 3D coordinates pi ∈ R3 of all M points are concatenated and fed into a multi-layer perceptron with fully connected layers, ReLU activations and batch normalisation. The MLP outputs a vector of M scalar weights wi, where ∑ i wi = 1, such that the predicted vertex location is the
  weighted mean of the input points, v = ∑ wipi. The prediction vj is supervised by a regression loss
  Lvert = 1P ∑P
  i ‖vi − vGTi ‖22, summed over all P corner patches. Predicting weights wi rather than directly the coordinates x̃ safeguards against extrapolation errors on unusual patches. We note that the localiser not only improves the corner point accuracy but also improves efficiency, by making it possible to use farthest point sampling rather than testing every input point.
  Edge Detector. The edge detection block serves to determine which pairs of (predicted) vertices are connected by an edge. Candidate edges eij are found by linking two vertices vi and vj , see below. To obtain a descriptor for an edge, we sample theNs equally spaced query locationsQ = {qk}s−1k=0 along the line segment between the two vertices, including the endpoints: q0 = ṽi, qs−1 = ṽj . For each qk we find the nearest 3D input point pNN(k) (by Euclidean distance) and retrieve its backbone encoding fNN(k). This yields an (Ns × C) edge descriptor, which is max-pooled along its first dimension, inspired by the ROI pooling in Faster-RCNN (Ren et al., 2015) and LOI pooling in LCNN (Zhou et al., 2019a). The resulting ( 1strideNs × C) descriptor is flattened and fed through an MLP with two fully connected layers, which returns a scalar edge/no_edge score.
  During training we select a number of vertex pairs from both the ground truth vertex list {vi}Gi=1 and the predicted vertex list {ṽi}Ki=1, and draw positive and negative sets of edge samples, E+, E−. We empirically found that evaluating all vertex pairs contained in E+, E− during training leads to a large imbalance between positive and negative samples. Similar to Zhou et al. (2019a), we thus draw input edges for the edge detector from the following sets:
  (a) Positive edges between ground truth vertices: This set comprises all true edges in the ground truth wireframe Egt+ = E (b) Negative edges between ground truth vertices: Two situations are relevant: "spurious edges", i.e., connections between two ground truth vertices that do not share an edge (but lie on the same surface, such that there are nearby points to collect features). And "inaccurate edges" where one of the endpoints is not a ground truth vertex, but not far from one (to cover difficult cases not far from a correct edge). Formally:
  Egt- ={evi,vj =(vi, vj) | vi, vj ∈ V, eij /∈ E ,∀p on evi,vj ,∃p′ ∈ P, ‖p− p′‖2 < } ∪ {ei,pk =(vi, pk) | (vi, vj) ∈ E , pk ∈ P, 1 ≤ ‖vj − pk‖2 ≤ 2}
  (c) Positive edges between predicted vertices: Connections between two "correct" predicted vertices that have both been verified to coincide with a ground truth wireframe vertex up to a small threshold +. Formally:
  Epred+ = {eṽi,ṽj = (ṽi, ṽj) | min ‖ṽi − vk‖2 < +,min ‖ṽj − vl‖2 < +, (vk, vl) ∈ Egt+} (d) Negative edges between predicted vertices: These are (i) "wrong links" as in Egt-, between "correctly" predicted vertices; and (ii) "hard negatives" where exactly one of the two vertices is close
  to a ground truth wireframe vertex, to cover "near misses". Epred- ={eṽi,ṽj =(ṽi, ṽj) | min ‖ṽi − vk‖2 < −,min ‖ṽj − vl‖2 < −, (vk, vl) ∈ Egt−} ∪ {eṽi,ṽj =(ṽi, ṽj) | 1 ≤ ‖ṽi − vk‖2 ≤ 2, ‖ṽj − vl‖2 ≤ 2, vk, vl ∈ V} Fig. 2 visually depicts the different edge sets. To train the edge detector we randomly sample positive examples from E+ = Egt+ ∪ Epred+ and negative examples from E− = Egt- ∪ Epred-. As loss function Ledge we use the balanced binary cross entropy. During inference, we go through the fully connected graph of edge candidates connecting any two (predicted) vertices. Candidates with a too high average distance to the input points (i.e., not lying on any object surface) are discarded. All others are fed into the edge detector for verification.
  End-to-end training. When training the complete network, we minimise the joint loss function with balancing weights α and β: L = Lpat + αLvert + βLedge. Non-Maximum Suppression (NMS) is a standard post-processing step for detection tasks, to suppress redundant, overlapping predictions. We use it (i) to prevent duplicate predictions of vertices that are very close to each other, and (ii) to prune redundant edges.
sections.heading: 4 EXPERIMENTS
sections.text: We could not find any publicly available dataset of point clouds with polyhedral wireframe annotations. Thus, we collect our own synthetic datasets: a subset from ABC (Koch et al., 2019) and a Furniture dataset. ABC is a collection of one million high-quality CAD models of mechanical parts with explicitly parameterised curves, surfaces and various geometric features, from which ground truth wireframes can be generated. We select a subset of models with straight edges. It consists of 3,000 samples, which we randomly split into 2,000 for training, 500 for validation, and 500 for testing. For the furniture dataset we have collected 250 models from Google 3DWarehouse for the categories bed, table, chair/sofa, stairs, monitor and render ground truth wireframes. Examples are shown in the supplementary material. To generate point clouds, we use the virtual scanner of Wang et al. (2017). We place 14 virtual cameras on the face centres of the truncated bounding cubes of each object, uniformly shoot 16,000 parallel rays towards the object from each direction, and compute their intersections with the surfaces. Finally, we jitter all points by adding Gaussian noiseN (0, 0.01), truncated to [−0.02, 0.02]. The number N of points varies between ≈ 100k−200k for different virtually scanned point clouds. For details, see the supplementary material.
sections.heading: 4.1 EVALUATION MEASURES
sections.text: We first separately evaluate the vertex detection and edge detection performance of our architecture, then assess the reconstructed wireframes in terms of structural average precision and of a newly designed wireframe edit distance.
  Mean Average precision mAPv for vertices. To generate high quality wireframes, the predicted vertex positions must be accurate. This metric evaluates the geometric quality of vertices, regardless of edge links. A predicted vertex is considered a true positive if the distance to the nearest ground truth vertex is below a threshold ηv. For a given ηv we generate precision-recall curves by varying
  the detection threshold. Average precision (APv) is defined as the area under this curve, mAPv is averaged over different thresholds ηv .
  Point-wise precision and recall for edges. We do not have a baseline that directly extracts 3D wireframe edges. Rather, existing methods label individual input points as "lying on an edge", and produce corresponding precision-recall curves (Hackel et al., 2016; Yu et al., 2018). To use this metric, we label a 3D point as "edge point" if its distance to the nearest wireframe edge is below the average point-to-point distance. As for APv, we can now check whether a predicted "edge point" is correct up to ηe, and compute APe and mean average precision (mAPe) over different ηe.
  Structural average precision for wireframes. The above two evaluation metrics looks at vertices and edges separately, in terms of geometry. In order to comprehensively evaluate the generated wireframe, we also need to take into account the connectivity. The structural average precision (sAP, Zhou et al., 2019a) metric, inspired by mean average precision, can be used to assess the overall prediction. This metric checks if the two end points (ṽi, ṽj) of a detected edge both match the end points (vk, vl) of a ground truth edge up to a threshold ηw, i.e., minvk∈V ‖ṽi − vk‖2 +minvl∈V ‖ṽj − vl‖2 < ηw, (vk, vl) ∈ E . We report sAP for different ηw and mean sAP (msAP).
  Wireframe edit distance (WED). In order to topologically compare wireframes, we have designed a quality measure based on edit distance. We consider wireframes as geometric graphs drawn in 3D space. Distance measures commonly used in computational geometry and graph theory, such as Hausdorff distance and Frechet distance, resistance distance, or geodesic distance, do not seem to apply to our situation. Inspired by Cheong et al. (2009), we propose a wireframe edit distance (WED). It is a variant of the graph edit distance (GED), which measures how many elementary graph edit operators (vertex insertion/deletion, edge insertion/deletion, etc.) are needed to transform one graph to another. Instead of allowing arbitrary sequences of operations, which makes computing the GED NP-hard, we fix their order, see Fig. 3: (1) vertex matching; (2) vertex translation; (3) vertex insertion; (4) edge deletion; (5) edge insertion. Let the ground truth wireframeWgt = (Vgt, Egt) and the predicted wireframeWo = (Vo, Eo). First, we match each vertex voi inWo to the nearest ground truth vertex vgtu inWgt, then translate them at a cost of cv · ‖voi vgtu ‖. New vertices are inserted where a ground truth vertex has no match (at no cost, as cost will accrue during edge insertion). After all vertex operations, we perform edge deletion and insertion based on ground truth edges at a cost of ce · ‖e‖, where ‖e‖ is the length of deleted/inserted edges. The final WED is the sum of all cost terms.
sections.heading: 4.2 RESULTS AND COMPARISONS
sections.text: Vertex detection. We validate the PC2WF vertex detection module and compare to a Harris3D baseline (with subsequent NMS to guarantee a fair comparison) on the ABC dataset, Tab. 1. Exemplary visual results and precision-recall curves are shown in Fig. 4 and Fig.5. Harris3D often fails to accurately localise a corner point, and on certain thin structures it returns many false positives. The patch size M used in our vertex detector has an impact on performance. Patch sizes between 20 and 50 achieve the best performance, where the precision stays at a high level over a large recall range. Patch size 1 leads to low precision (although still better than Harris3D), confirming that per-point features, even when computed from a larger receptive field, are affected by noise and insufficient to find corners with acceptable false positive ratio. We refer the interested reader to the supplementary material for a more exhaustive ablation study.
  Edge prediction. Due to the limited number of methods able to extract wireframes from 3D point clouds, we compare PC2WF to recent algorithms that detect edge points, such as FREE (Bazazian et al., 2015), ECD (Ahmed et al., 2018), FEEM (Xia & Wang, 2017), Contour Detector (Hackel et al., 2016) and EC-Net (Yu et al., 2018). We also compare with Polyfit (Nan & Wonka, 2017), which is the only method able to reconstruct a vectorial representation similar to ours from a point cloud. Since Polyfit extracts planar surfaces of the object, we must convert that representation into a wireframe, which amounts to identifying the plane intersections. Precision-recall curves for all methods are shown in Fig. 6. Visual results are shown in Fig. 8. PC2WF outperforms all baselines based on hand-crafted features like FREE, ECD, FEEM and Contour Detector, as well as the deep learning method EC-Net, across all quality measures (Tab. 2). A visual comparison between the ground truth, PC2WF, EC-Net Yu et al. (2018), and Polyfit is shown in Fig. 9. One can clearly see the advantage of predicting vector edges rather than "edge points", as they lead to a much sharper result. The inevitable blur of edge points across the actual contour line, and their irregular density, make vectorisation a non-trivial problem. Polyfit and PC2WF show comparable performance for this specific point cloud. PC2WF failed to localise some corners and missed some small edges, whereas Polyfit failed to reconstruct the lower part of the armchair.
  Wireframe Reconstruction. We visualise PC2WF’s output wireframes in Fig. 10. Structural precision-recall curves with various patch sizes are shown in Fig. 7. For additional examples of output predictions, including cases with curved edges, please refer to the supplementary material. The results are aligned with the ones of the vertex module. Moderate patch sizes between 20 and 50 achieve the best performance. Tab. 3 shows the structural average precision with patch size 50, which exceeds the one of Polyfit. The results in terms of the wireframe edit distance are shown in Tab. 4 for PC2WF and Polyfit. The average number of predicted vertices (21.7) and ground truth vertices (20.1) per object are close, such that only few vertex insertions are needed in most cases. Although all predicted vertices need some editing, the WEDv caused by those edits is small (0.2427), meaning that the predictions are accurate. On the contrary, the 2.6 edges that need to be edited on
  0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Recall
  0.0
  0.1
  0.2
  0.3
  0.4
  0.5
  0.6
  0.7
  0.8
  0.9
  1.0
  Pr ec
  isi on
  f=0.2 f=0.3 f=0.4 f=0.5
  f=0.5
  f=0.6
  f=0.7
  f=0.8
  Patch size=1 Patch size=20 Patch size=50 Patch size=100 Harris3D
  Figure 5: Vertex prediction accuracy on ABC dataset. 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Recall
  0.0
  0.1
  0.2
  0.3
  0.4
  0.5
  0.6
  0.7
  0.8
  0.9
  1.0
  Pr ec
  isi on
  f=0.2 f=0.3 f=0.4 f=0.5
  f=0.5
  f=0.6
  f=0.7
  f=0.8
  FREE ECD FEEM Contour EC-NET PC2WF
  Figure 6: Edge detection accuracy on ABC dataset. 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Recall
  0.0
  0.1
  0.2
  0.3
  0.4
  0.5
  0.6
  0.7
  0.8
  0.9
  1.0
  Pr ec
  isi on
  f=0.2 f=0.3 f=0.4 f=0.5
  f=0.5
  f=0.6
  f=0.7
  f=0.8
  Patch size=1 Patch size=20 Patch size=50 Patch size=100
  Figure 7: Wireframe accuracy on ABC dataset.
  average increase WEDe a lot more (1.4142) and dominate the WED, i.e., the inserted/deleted edges have significant length.
sections.heading: 5 CONCLUSION
sections.text: We have proposed PC2WF, an end-to-end trainable deep architecture to extract a vectorised wireframe from a raw 3D point cloud. The method achieves very promising results for man-made, polyhedral
  objects, going one step further than low-level corner or edge detectors, while at the same time outperforming them on the isolated vertex and edge detection tasks. We see our method as one further step from robust, but redundant low-level vision to compact, editable 3D geometry. As future work we aim to address the biggest limitation of our current method which is the inability to handle wireframes with strongly curved edges.
title: RAW POINT CLOUDS