summaries: The paper builds a memory and computation-efficient version of the KPConv model, which is a point cloud processing model. The paper uses many techniques to achieve this including, depthwise kernels, attention over kernels, and also neural architecture search (NAS). The paper conducts experiments to demonstrate the effectiveness of their techniques. This paper aims to accelerate the inference of 3D point cloud neural networks. The authors first borrow a few designs from efficient 2D neural networks: depthwise convolution and inverted residual bottleneck. They then propose to reweight the weights of kernel points with attention to boost its representation power. Finally, the authors adopt the predictor-based neural architecture search to explore the best model under a resource constraint. They have evaluated their proposed solution on the small-scale ModelNet40 dataset and the large-scale SemanticKITTI dataset and have achieved reasonably good empirical performance. This paper addresses point-based methods for classification and semantic segmentation in 3D applications. It aims to improve the computational efficiency of this kind of methods to make it better fit the applications of limited resources such as mobile scenarios. To achieve this, this paper conducts the following works. First, it develops a mobile attention kernel point convolution (MAKPConv) scheme to improve the performance of existing kernel point convolution. Second, it utilizes neural architecture search (NAS) technique to design the MAKPConv-based network. In this part, this paper proposes a wide & deep neural predictor for the NAS process. Experimental study is conducted on benchmark datasets and tasks to demonstrate that the resulted deep networks can achieve higher computational efficiency and improved performance. Ablation study is also conducted to illustrate the key components in this work.  This paper aims to address the point cloud analysis task by:  1) improving the existing KPConv operation via considering the kernel relationship;  2) designing the networks via a predictor-based NAS approach.  The improved KPConv operation. i.e., MAKPConv, can model the local structure efficiently, and the searched network has fewer parameters while performing better than the baseline on two datasets, including ModelNet40 for classification and SemanticKitti for segmentation.
abstractText: Kernel Point Convolution (KPConv) achieves cutting-edge performance on 3D point cloud applications. Unfortunately, the large size of KPConv network limits its usage in mobile scenarios. In addition, we observe that KPConv ignores the kernel relationship and treats each kernel point equally when formulating neighbor-kernel correlation via Euclidean distance. This leads to a weak representation power. To mitigate the above issues, we propose a module named Mobile Attention Kernel Point Convolution (MAKPConv) to improve the efficiency and quality of KPConv. MAKPConv employs a depthwise kernel to reduce resource consumption, and re-calibrates the contribution of kernel points towards each neighbor point via Neighbor-Kernel attention to improve representation power. Furthermore, we capitalize Inverted Residual Bottleneck (IRB) to craft a design space and employ a predictor-based Neural Architecture Search (NAS) approach to automate the design of efficient 3D networks based on MAKPConv. To fully exploit the immense design space via an accurate predictor, we identify the importance of carrying feature engineering on searchable features to improve neural architecture representations, and propose a Wide & Deep Predictor to unify dense and sparse neural architecture representations for lower error in performance prediction. Experimental evaluations show that our NAS-crafted MAKPConv network uses 96% fewer parameters on 3D point cloud classification and segmentation benchmarks with better performance. Compared with state-of-the-art NAScrafted model SPVNAS, our NAS-crafted MAKPConv network achieves ∼1% better mIOU with 83% fewer parameters and 78% fewer Multiply-Accumulates.
references.authors: Iro Armeni Ozan Sener Amir R Zamir Helen Jiang Ioannis Brilakis Martin Fischer Silvio Savarese
references.title: 3d semantic parsing of large-scale indoor spaces
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Matan Atzmon Haggai Maron Yaron Lipman
references.title: Point convolutional neural networks by extension operators
references.venue: arXiv preprint arXiv:1803.10091,
references.authors: Jens Behley Martin Garbade Andres Milioto Jan Quenzel Sven Behnke Cyrill Stachniss Jurgen Gall
references.title: Semantickitti: A dataset for semantic scene understanding of lidar sequences
references.venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,
references.authors: Holger Caesar Varun Bankiti Alex H. Lang Sourabh Vora Venice Erin Liong Qiang Xu Anush Krishnan Yu Pan Giancarlo Baldan Oscar Beijbom
references.title: nuscenes: A multimodal dataset for autonomous driving
references.venue: arXiv preprint arXiv:1903.11027,
references.authors: Tianqi Chen Carlos Guestrin
references.title: Xgboost: A scalable tree boosting system
references.venue: In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining,
references.authors: Heng-Tze Cheng Levent Koc Jeremiah Harmsen Tal Shaked Tushar Chandra Hrishi Aradhye Glen Anderson Greg Corrado Wei Chai Mustafa Ispir
references.title: Wide & deep learning for recommender systems
references.venue: In Proceedings of the 1st workshop on deep learning for recommender systems,
references.authors: Christopher Choy JunYoung Gwak Silvio Savarese
references.title: 4d spatio-temporal convnets: Minkowski convolutional neural networks
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Tiago Cortinhal George Tzelepis Eren Erdal Aksoy
references.title: Salsanext: fast, uncertainty-aware semantic segmentation of lidar point clouds for autonomous driving
references.venue: arXiv preprint arXiv:2003.03653,
references.authors: Xiaoliang Dai Alvin Wan Peizhao Zhang Bichen Wu Zijian He Zhen Wei Kan Chen Yuandong Tian Matthew Yu Peter Vajda
references.title: Fbnetv3: Joint architecture-recipe search using predictor pretraining
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Tony Duan Avati Anand Daisy Yi Ding Khanh K Thai Sanjay Basu Andrew Ng Alejandro Schuler
references.title: Ngboost: Natural gradient boosting for probabilistic prediction
references.venue: In International Conference on Machine Learning,
references.authors: Łukasz Dudziak Thomas Chau Mohamed S Abdelfattah Royson Lee Hyeji Kim Nicholas D Lane
references.title: Brp-nas: Prediction-based nas using gcns
references.venue: arXiv preprint arXiv:2007.08668,
references.authors: Siqi Fan Qiulei Dong Fenghua Zhu Yisheng Lv Peijun Ye Fei-Yue Wang
references.title: Scf-net: Learning spatial contextual features for large-scale point cloud segmentation
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Andreas Geiger Philip Lenz Christoph Stiller Raquel Urtasun
references.title: Vision meets robotics: The kitti dataset
references.venue: International Journal of Robotics Research (IJRR),
references.authors: Ankit Goyal Hei Law Bowei Liu Alejandro Newell Jia Deng
references.title: Revisiting point cloud shape classification with a simple and effective baseline
references.venue: arXiv preprint arXiv:2106.05304,
references.authors: Jie Hu Li Shen Gang Sun
references.title: Squeeze-and-excitation networks
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Qingyong Hu Bo Yang Linhai Xie Stefano Rosa Yulan Guo Zhihua Wang Niki Trigoni Andrew Markham
references.title: Randla-net: Efficient semantic segmentation of large-scale point clouds
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Zeyu Hu Mingmin Zhen Xuyang Bai Hongbo Fu Chiew-lan Tai
references.title: Jsenet: Joint semantic segmentation and edge detection network for 3d point clouds
references.venue: In Computer Vision–ECCV 2020: 16th European Conference,
references.authors: Guolin Ke Qi Meng Thomas Finley Taifeng Wang Wei Chen Weidong Ma Qiwei Ye TieYan Liu
references.title: Lightgbm: A highly efficient gradient boosting decision tree
references.venue: Advances in neural information processing systems,
references.authors: Itai Lang Asaf Manor Shai Avidan
references.title: Samplenet: Differentiable point cloud sampling
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Huan Lei Naveed Akhtar Ajmal Mian
references.title: Seggcn: Efficient 3d point cloud segmentation with fuzzy spherical kernel
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Huan Lei Naveed Akhtar Ajmal Mian
references.title: Spherical kernel for efficient graph convolution on 3d point clouds
references.venue: IEEE transactions on pattern analysis and machine intelligence,
references.authors: Yiqun Lin Zizheng Yan Haibin Huang Dong Du Ligang Liu Shuguang Cui Xiaoguang Han
references.title: Fpconv: Learning local flattening for point convolution
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Yongcheng Liu Bin Fan Shiming Xiang Chunhong Pan
references.title: Relation-shape convolutional neural network for point cloud analysis
references.authors: Yuqiao Liu Yehui Tang Yanan Sun
references.title: Homogeneous architecture augmentation for neural predictor
references.venue: arXiv e-prints, pp
references.authors: Ilya Loshchilov Frank Hutter
references.title: Sgdr: Stochastic gradient descent with warm restarts
references.venue: arXiv preprint arXiv:1608.03983,
references.authors: Maxim Naumov Dheevatsa Mudigere Hao-Jun Michael Shi Jianyu Huang Narayanan Sundaraman Jongsoo Park Xiaodong Wang Udit Gupta Carole-Jean Wu Alisson G Azzolini
references.title: Deep learning recommendation model for personalization and recommendation systems
references.authors: Charles R Qi Hao Su Kaichun Mo Leonidas J Guibas
references.title: Pointnet: Deep learning on point sets for 3d classification and segmentation
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Charles Ruizhongtai Qi Li Yi Hao Su Leonidas J Guibas
references.title: Pointnet++: Deep hierarchical feature learning on point sets in a metric space
references.venue: In Advances in neural information processing systems,
references.authors: Shi Qiu Saeed Anwar Nick Barnes
references.title: Semantic segmentation for real point cloud scenes via bilateral augmentation and adaptive fusion
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Haoxi Ran Wei Zhuo Jun Liu Li Lu
references.title: Learning inner-group relations on point clouds
references.venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV),
references.authors: Esteban Real Alok Aggarwal Yanping Huang Quoc V Le
references.title: Regularized evolution for image classifier architecture search
references.venue: In Proceedings of the aaai conference on artificial intelligence,
references.authors: Mark Sandler Andrew Howard Menglong Zhu Andrey Zhmoginov Liang-Chieh Chen
references.title: Mobilenetv2: Inverted residuals and linear bottlenecks
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: John Schulman Filip Wolski Prafulla Dhariwal Alec Radford Oleg Klimov
references.title: Proximal policy optimization algorithms
references.venue: arXiv preprint arXiv:1707.06347,
references.authors: Julien Siems Lucas Zimmer Arber Zela Jovita Lukasik Margret Keuper Frank Hutter
references.title: Nasbench-301 and the case for surrogate benchmarks for neural architecture
references.authors: Haotian Tang Zhijian Liu Shengyu Zhao Yujun Lin Ji Lin Hanrui Wang Song Han
references.title: Searching efficient 3d architectures with sparse point-voxel convolution
references.venue: In European Conference on Computer Vision,
references.authors: Hugues Thomas Charles R Qi Jean-Emmanuel Deschaud Beatriz Marcotegui François Goulette Leonidas J Guibas
references.title: Kpconv: Flexible and deformable convolution for point clouds
references.venue: In Proceedings of the IEEE International Conference on Computer Vision,
references.authors: Yue Wang Yongbin Sun Ziwei Liu Sanjay E Sarma Michael M Bronstein Justin M Solomon
references.title: Dynamic graph cnn for learning on point clouds
references.venue: Acm Transactions On Graphics (tog),
references.authors: Wei Wen Hanxiao Liu Yiran Chen Hai Li Gabriel Bender Pieter-Jan Kindermans
references.title: Neural predictor for neural architecture search
references.venue: In European Conference on Computer Vision,
references.authors: Zhirong Wu Shuran Song Aditya Khosla Fisher Yu Linguang Zhang Xiaoou Tang Jianxiong Xiao
references.title: 3d shapenets: A deep representation for volumetric shapes
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Yifan Xu Tianqi Fan Mingye Xu Long Zeng Yu Qiao
references.title: Spidercnn: Deep learning on point sets with parameterized convolutional filters
references.venue: In Proceedings of the European Conference on Computer Vision (ECCV),
references.authors: Xiaoqing Ye Jiamao Li Hexiao Huang Liang Du Xiaolin Zhang
references.title: 3d recurrent neural networks with context fusion for point cloud semantic segmentation
references.venue: In Proceedings of the European Conference on Computer Vision (ECCV),
references.authors: Yizhou Yu Kun Zhou Dong Xu Xiaohan Shi Hujun Bao Baining Guo Heung-Yeung Shum
references.title: Mesh editing with poisson-based gradient field manipulation
references.venue: In ACM SIGGRAPH 2004 Papers,
references.authors: Chris Zhang Wenjie Luo Raquel Urtasun
references.title: Efficient convolutions for real-time semantic segmentation of 3d point clouds
references.venue: In 2018 International Conference on 3D Vision (3DV),
references.authors: Yang Zhang Zixiang Zhou Philip David Xiangyu Yue Zerong Xi Boqing Gong Hassan Foroosh
references.title: Polarnet: An improved grid representation for online lidar point clouds semantic segmentation
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Hui Zhou Xinge Zhu Xiao Song Yuexin Ma Zhe Wang Hongsheng Li Dahua Lin
references.title: Cylinder3d: An effective 3d framework for driving-scene lidar semantic segmentation
references.authors: Zixiang Zhou Yang Zhang Hassan Foroosh
references.title: Panoptic-polarnet: Proposal-free lidar point cloud panoptic segmentation
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Barret Zoph Quoc V Le
references.title: Neural architecture search with reinforcement learning
references.venue: arXiv preprint arXiv:1611.01578,
sections.heading: 1 INTRODUCTION
sections.text: 3D applications such as object recognition (Wu et al., 2015), shape segmentation (Yu et al., 2004), and indoor-scene segmentation (Armeni et al., 2016) are growing popularity thanks to the rise of 3D acquisition technologies and the increasing amount of geometric data collected from 3D sensors. Deep learning models are becoming one of the most common and promising ways to represent 3D data, especially 3D point cloud data, thanks to its high performance (Zhou et al., 2020; Zhao et al., 2020) and good efficiency (Cortinhal et al., 2020; Hu et al., 2020a).
  Starting from PointNet (Qi et al., 2017a), point-based methods are becoming one of the most popular choices to extract features and make predictions for 3D input point clouds. Such an approach typically learns an individual representation for each data point and aggregates different representations to formulate a signature for all global points, usually via end-to-end point convolutions (Xu et al., 2018; Atzmon et al., 2018). Kernel point convolution (KPConv) (Thomas et al., 2019) is the first work that maps the input points to the influence of kernel points via linear correlation, and then further convolves the mapped features over the kernel points to compute weighted features. Given a point yi, the kernel function g over neighbor points yi in KPConv carries g(yi) =∑ k<K h(yi, x̃k)@Wk to learn representations via K kernel points, with the neighbor-kernel correlation h(yi, xk). However, KPConv results in very large model sizes for 3D point cloud recognition due to inefficient local feature aggregation via point convolution over kernel weights Wk. As a result, KPConv takes up to ∼200G Multiply-Accumulates (MACs) to perform inference on a single scene at the cost of ∼200ms on a TITAN XP GPU. Such resource consumption and inference cost are too high for real-time execution in mobile scenarios. In addition, the above Neighbor-Kernel cor-
  relation function has a strong hidden assumption: the contribution of all kernel points are measured equally towards the neighbor points. As a result of ignoring cross-kernel information from other kernel points to accommodate the joint evolution of different kernel points, such an assumption fails to enhance representation learning on important kernel points and thus yields sub-optimal results.
  In this paper, we propose a novel 3D attention module, named Mobile Attention Kernel Point Convolution (MAKPConv), to address the above issues. MAKPConv employs two key design elements to improve the performance and efficiency. First, we propose a depthwise kernel to split feature extraction on point-wise features from channel-wise features during point convolution. Such design allows significant savings in both parameter count and amount of computation, without performance degradation. Second, we introduce Neighbor-Kernel attention (NK attention) to enable the kernelwise information exchange between different kernel points via learnable attention networks. The joint evolution of all kernel points encourages representation learning on more important kernel points while suppressing less important ones, thus benefits from a better learning adaptability of kernel point contributions towards neighbor point distributions in KPConv networks.
  Inspired by Sandler et al. (2018), we capitalize Inverted Residual Bottleneck (IRB) to encapsulate MAKPConv module into MAKPConv block, and craft MAKPConv design space to explore a wider range of choices in 3D models. As 3D architectures are usually high-dimensional, unstructured and unordered, tailoring a good neural architecture to address such complex point cloud data requires a delicate and precise evaluation protocol. This is difficult to achieve with weight-sharing oneshot NAS (e.g., SPVNAS (Tang et al., 2020)) because co-adaptation between sub-networks may degrade the quality of evaluation and lead to sub-optimal results. Thus, we pioneer the usage of predictor-based Neural Architecture Search (NAS) (Wen et al., 2020) that utilizes a sample-based mechanism to reduce evaluation noise and seek better models on 3D point cloud. Specifically, we train a performance predictor based on exploitable samples, and probe the whole MAKPConv design space to obtain the best NAS model based on predictive performance. We observe that existing predictor-based methods adopt a fixed scheme (e.g., one-hot encoding) to encode different types of features, leading to sub-optimal neural architecture representations with inaccurate priors. This is because real-valued features (e.g., block width) and categorical features (e.g., optional NK attention) have different priors and thus they should not be encoded in the same way. To address this issue, we formulate dense representations for features that have quasi-linear priors on model performance (e.g., depth, width), and formulate sparse representation for features that have nonlinear sophisticated priors on model performance (e.g., optional attention). Intuited by Cheng et al. (2016), we devise a Wide & Deep predictor to interact dense and sparse representations such that the combination of memorization and generalization reduces the prediction error.
  By utilizing the Wide & Deep predictor to perform NAS under the MAKPConv design space, our results on various benchmarks show promising results of the automatically discovered models in terms of both performance and efficiency. In particular, our MAKPConv networks outperform the original KPConv on both classification and semantic segmentation benchmarks with 92%∼93% fewer parameters, thanks to the efficient design of MAKPConv block. Empowered by our Wide & Deep predictor driven NAS, the NAS-crafted networks outperform the state-of-the-art NAS work, SPVNAS, by ∼1 mIOU with 83% smaller size and 78% fewer MACs. We highlight the contribution of this paper as follows:
  • We propose Mobile Attention Kernel Point Convolution (MAKPConv) module that combines depthwise kernel with Neighbor-Kernel attention to achieve efficient and highperforming architectures dedicated to learning on 3D point clouds.
  • We identify the importance of carrying feature engineering on neural architecture representations to improve predictor-based NAS. Specifically, we encode different searchable features with dense and/or sparse representations given their priors, and utilize a Wide & Deep neural predictor to interact these representations for better accuracy prediction.
sections.heading: 2 RELATED WORK
sections.text: 3D Point Convolution Networks. Point convolution networks (Xu et al., 2018; Hu et al., 2020a; Lang et al., 2020) extends convolutional operations towards irregular point set grids to benefit from learnable radial function and spherical harmonics in 3D point clouds. Later work (Thomas et al.,
  2019) performs point convolution over radius neighbors and aggregates convolved results to produce learned representation without additional intermediate operators. Recent work on point convolutions improves point sampling efficiency (Hu et al., 2020a; Lang et al., 2020), local feature aggregations (Zhao et al., 2020; Lin et al., 2020), manual architecture fabrications (Hu et al., 2020b; Lei et al., 2020a), and/or improved training protocols (Goyal et al., 2021) to seek higher model performance and efficiency. However, these works neither simplify the structure of point networks to improve model efficiency, nor specify an accurate neighbor-kernel correlation. Our work extends kernel point convolution (Thomas et al., 2019) and addresses the above caveats: we improve the efficiency of local feature aggregation via a depthwise kernel on point convolution, and further introduce Neighbor-Kernel attention to enhance the representation power of a depthwise kernel.
  Predictor-based Neural Architecture Search. Predictor-based NAS (Wen et al., 2020; Dudziak et al., 2020) is a popularized method that trains a performance predictor on limited exploitable samples. The trained predictor serves as a surrogate model of the ground-truth performance and is utilized to guide architecture search on the full design space. Existing achievements on predictorbased NAS lies in improving sample efficiency (Dudziak et al., 2020) and augmenting architecture samples (Liu et al., 2021), yet they overlook the importance of carrying feature engineering on neural architecture features. Our work identifies the importance of improving neural architecture representation, and assigns different priors to a wide set of searchable features and yield dense representations and sparse representations. Our work further designs a Wide & Deep predictor to leverage and interact these representations for better prediction quality on unseen architectures.
sections.heading: 3 MOBILE ATTENTION KERNEL POINT CONVOLUTION
sections.text: We start with the formulation of KPConv (Thomas et al., 2019). Given a center point X ∈ R3 and its corresponding features F̂ ∈ RDin , a kernel point convolution (KPConv) function carries all N neighbor points Xn = {x′ : ||x′ − x|| < r} ∈ RN×3 and their features F̄ ∈ RN×Din within radius r over a set of K kernel points X̃ ∈ RK×3 and accompanying weights Wk. Specifically, a correlation function: h : RN×Din → RK×Din maps the unordered neighbor point features F̄ to a set of features F on kernel points. Usually, the correlation function is represented as H ∈ RN×K and applied via F = HF̄ . The correlation function is followed by a kernel g : RK×Din → RDout that convolves these features to obtain the aggregated output F ′. Here, Din denotes the dimension of input, and Dout denotes the dimension of output. The original KPConv overlooks an important opportunity to improve the efficiency by reducing the redundancy of kernel g that is due to the simultaneously performed point-wise and channel-wise feature extractions. In addition, the original KPConv adopts a linear function as correlation: it treats the impact of each kernel point equally and ignores the adaptation of neighbor points towards kernel dispositions. For example, the center kernel point usually attracts more centralized neighbor points with a higher density, and edge kernel points perform critical jobs in recognizing boundary neighbor points with a lower density in the radius sampling neighborhood. Thus, the original KPConv may have sub-optimal performance and efficiency on 3D applications.
  To overcome the above limitations, we propose Mobile Attention Kernel Point Convolution (MAKPConv) module as an in-place improvement of KPConv block to seek higher quality and efficiency. Figure 1 provides an overview of MAKPConv. Specifically, we design a depthwise kernel to address the computation cost: A depthwise kernel ĝ carries element-wise multiplication in place of matrix multiplication between aggregated features and kernel weights, thus achieves memory and computation reduction. Considering a depthwise kernel has limited representation power, we introduce Neighbor-Kernel attention (NK attention) as an improved correlation function ĥ to emphasize representation learning on important depthwise kernels and fully utilize the capacity of kernel points. This compensates for the reduced representation power: the joint evolution of kernel points via learnable attention encourages effective representation learning on important depthwise kernels and best exploit their potential. As a result, the fusion of depthwise kernel and NK attention into MAKPConv block has the potential to achieve high-performing yet efficient 3D models.
sections.heading: 3.1 DEPTHWISE KERNEL
sections.text: In original kernel function of KPConv, a regular kernel takes an input F ∈ RK×Din over K kernel points and uses a weight kernel W ∈ RK×Din×Dout to simultaneously perform point-wise transformation and channel-wise transformation. A regular kernel g produces an output F ′ ∈ RDout :
  F ′d = g(F,W ) = ∑ k∈K ∑ d∈Din Fk,i ·Wk,i,d, (1)
  where k is a kernel index, and d is a channel index. A regular kernel leads to K × Din × Dout parameters andDin×Dout×K MACs during inference, and leads to large computation and memory for the deep layers of the 3D architectures. To favor model efficiency, we design a depthwise kernel ĝ that only performs point-wise feature extractions over input point clouds, see Figure 1(a). The proposed depthwise kernel adopts learnable weightsW ∈ RK×Din , extracts point-wise information on each channel, and obtains an output F ′ ∈ RDout as follows:
  F ′d = ĝ(F,W ) = ∑ k∈K Fk,d ·Wk,d, (2)
  where k is a kernel index, and d is a channel index. Here, Dout = Din = D as a depthwise kernel does not operate on channel-wise features. As a result, a depthwise kernel point convolution only leads to K ×Din parameters and only Din ×K MACs during inference, and reduces Dout× parameters and MACs during convolution compared to a regular kernel. The parameter saving from a depthwise kernel is significant when Dout is large, especially in deeper layers of the network.
sections.heading: 3.2 NEIGHBOR-KERNEL ATTENTION
sections.text: Although depthwise kernel ĝ adds to model efficiency, such design lacks representation power to fit complex point cloud structures due to inactive channel-wise communication. To compensate for the loss, we propose Neighbor-Kernel attention (NK attention) ĥ : RN×Din → RK×Din that re-weights the contribution of kernel points by learning a joint representation Ĥ ∈ RN×K on Neighbor-Kernel distances (NK distance) via an attention network, see the lower part of Figure 1. Such representation, when being applied to neighbor point features F̄ , can fully harness the capacity provided by different kernel points, thus provide better aggregated features F for the depthwise kernel.
  NK attention first adopts a linear function to transform NK distance into linear NK correlation, and applies average pooling to obtain global neighborhood correlation. This global information captures rich information over the statistics of the locations of neighbor points and thus gives a highlevel abstraction on the contribution of each kernel point towards the global neighborhood. Then, the global neighborhood correlation is fed into a 2-layer attention network to enable cross-kernel interaction and learn a good representation to quantify the contribution of kernel points, followed by a sigmoid gating to normalize the learned representation. Unlike Squeeze-and-excitation (Hu et al., 2018) , NK attention adopts a full-rank transformation to preserve the information needed towards every kernel with marginal parameter cost. Finally, the learned representation is scaled to the original NK linear correlation, followed by a bypass (i.e., addition) of the original linear NK correlation for easier optimization. As a result, NK attention grasps a richer representation Ĥ that incorporates the joint evolution of kernel points and encourages representation learning on important depthwise kernels, This leads to concrete performance improvement on 3D point-cloud benchmarks.
sections.heading: 4 NERUAL ARCHITECTURE SEARCH WITH WIDE & DEEP PREDICTOR
sections.text: In this section, we introduce a predictor-based Neural Architecture Search (NAS) driven by a novel Wide & Deep predictor, see Figure 2. MAKPConv module provides efficient motifs for processing point-wise features. To incorporate channel-wise feature extraction into MAKPConv module, we exploit the design of Inverted Residual Bottleneck (IRB) (Sandler et al., 2018) and sandwich MAKPConv module with 2 linear transformation to construct MAKPConv block. To design efficient yet high performing 3D models, we utilize MAKPConv block as an architecture motif and exploit predictor-based neural architectures with a wide set of features such as positional organizations, structural settings, and accessories. As formulating neural architecture representations from these features is the key in predictor-based NAS, we identify the importance of carrying engineering on searchable features. Thus, we assign different priors to these features to formulate dense architecture representations and sparse architecture representations, and propose a novel Wide & Deep neural predictor to effectively interact different sets of neural architecture representations for accurate predictions. Wide & Deep predictor trades-off generalization and memorization via dense deep models and sparse wide embeddings, thus reduce prediction MSE on unseen architectures.
sections.heading: 4.1 MAKPCONV BLOCK
sections.text: Inspired by Sandler et al. (2018), we sandwich MAKPConv by 2 linear transformation layers to derive MAKPConv block as a promising architecture motif to carry efficient and high performing 3D point cloud learning. It has a similar structure as Inverted Residual Bottleneck (IRB) and is described by four key parameters: expansion factor E, kernel size K, output width W , and optional choice of NK attention Atn. We follow the positional setting of MobileNetV2 to manually craft a MAKPConv network as a hand-crafted baseline to verify its performance and efficiency. We compare the performance and size between hand-crafted MAKPConv network and KPConv under the native KPConv protocol in Table 1. Compared with KPConv, the hand-crafted MAKPConv achieves on-par Overall Accuracy (OA) with at least 92% smaller size on ModelNet40, and achieves 0.4% higher mIOU with 93% smaller size on SemanticKITTI, even without NK attention. Such advantage is attributed to the efficient depthwise kernel. With NK attention, MAKPConv yields ∼0.4% OA gain on classification and ∼0.5% mIOU gain on semantic segmentation with marginal increase in parameter count. These results suggest that re-weighing the contribution of kernel points via learnable attention network enhances depthwise kernels thus improves performance.
sections.heading: 4.2 MAKPCONV DESIGN SPACE
sections.text: We utilize MAKPConv blocks to construct a flexible network design space. Specifically, we divide the overall 3D network into a stem KPConv for initial feature extraction, a head MLP for carrying fine-grained classification/segmentation, and several searchable stages. In each searchable stage, we jointly search for the optimal configuration of MAKPConv blocks, including the positional organizations, structural settings, and accessories. We elaborate these searchable features below.
  Positional Organizations. The block width and depth of MAKPConv has a direct impact on the latency of MAKPConv models by affecting both computation-bounded operations (e.g., linear transformation) and memory-bounded operations (e.g., indexing). As deeper and wider architectures lead to better performance in 3D point cloud classification and segmentation following the Pareto frontiers, positional organizations have a positive quasi-linear impact towards performance and take continuous values on any positive integers.
  Structural Settings. The expansion ratio affects the inner-structure of MAKPConv block that has a positive quasi-linear correlation with performance. However, co-adaption of expansion factor towards block width may induce challenges when learning a mapping rule to accurately predict architectures in the search space. Structural settings take continuous values on any positive integers.
  Accessories. We mark the search of optional attentions and kernel size as accessories in the MAKPConv search space. Accessories have sophisticated impact on model performance. For example, as kernel point convolution utilize different dispositions for different number of kernels, it is unclear whether the change in the number of kernel points affects the quality of kernel disposition and the model performance. Both accessories can only take categorical values.
  Considering the high variance in 3D point cloud tasks and the robustness to noisy evaluations, we adopt predictor-based NAS to explore MAKPConv design space. However, MAKPConv design space is prohibitively large: a total of 11 stages and 7 searchable features within each stage leads a total search space size of 1.8×1019 possible architectures. Such a large and complex search space calls for a high-performing predictor-based NAS to accurately model the design space.
sections.heading: 4.3 WIDE & DEEP PREDICTOR
sections.text: Neural architecture representation is the key in the performance of predictor-based NAS due to its strong connection with predictor performance. Specifically, better neural architecture representation links searchable features towards priors that are more reflective to the final performance, and thus increases the chance to discover high-performing NAS models.
  To improve neural architecture representations, We focus on carrying feature engineering to distinguish searchable features with different priors. Motivated by the aforementioned relationship between searchable features and the model performance, we assign different priors to a wide set of searchable features and craft dense and sparse neural architecture representations for corresponding features. Specifically, we formulate dense neural architecture representation for searchable features that can take continuous values (e.g., positional organizations, structural settings), and formulate sparse neural architecture representation for searchable features that can only take discrete values (e.g., accessories). With dense and sparse neural architecture representations, we follow the idea of Wide and Deep Models (Cheng et al., 2016) to design a Wide & Deep predictor that trades-off memorization and generalization in performance prediction, see Figure 2. Wide & Deep model adopts a separate MLP to obtain high-level dense outputs based on each of the 2 dense representations, and adopts an embedding table to convert sparse representations to continuous sparse outputs. To interact dense and sparse outputs, we leverage the DotProduct block in DLRM (Naumov et al., 2019) to fuse learned dense representations and sparse representations. Specifically, given a dense representation D ∈ RB×dim and a sparse representation S ∈ RB×N×dim, DotProduct carries cross-representation communication by carrying the following transformation:
  X = [vec−1B,1,dim(D);S], DotProduct(D,S) = Triu(XX T ) (3)
  where dim represents the dimension of both dense and sparse representations,N denotes the number of sparse inputs, and B denotes the batch size. Eq. (3) first vectorizes the dense tensor D to have the same dimension as the sparse tensor S, and extracts the upper triangular matrix upon self matrix factorization. We carry 2 DotProducts on 2 separate pairs of dense and sparse representations to encourage the learning of independent interactions. This is because positional organizations and
sections.heading: 4.4 TRAINING AND EVALUATION OF THE WIDE & DEEP PREDICTOR
sections.text: Architecture Sampling. We randomly sample ∼1K architectures on both ModelNet40 and SemanticKITTI. Following the common NAS pipeline, we split the training dataset of ModelNet40/SemanticKITTI into mini-train and mini-val. We train the sampled architectures on mini-train until convergence, and evaluate these architectures on mini-val to obtain architecture performance.
  Predictor Training. Mean-Squared Error (MSE) is the training objective of the Wide & Deep Predictor on 1K sampled architectures. To ensure a consistent comparison of predictor performance over multiple benchmarks, we carry optimization on predictors with normalized performance ranging from 0 to 1. Since the Wide & Deep Predictor can perform gradient-based optimization, we incorporate predictor pretraining on MACs (Dai et al., 2021) and rank loss (Dudziak et al., 2020).
  We conduct extensive experiments on various top-performing NAS predictors collected from NASBench-301 (Siems et al., 2020). To justify that interacting dense and sparse representations benefits predictor performance, we establish “Wide” as a MLP Predictor that accepts sparse neural architecture representations transformed from categorical encoding of features, and establish “Deep” as a MLP predictor that accepts dense neural architecture representations transformed from realvalued encoding features. These two predictors also leverage rank loss (Dudziak et al., 2020) and predictor pretraining (Dai et al., 2021), thus are utilized as strong baselines for gradient-based NAS predictors to justify the significance of dense-sparse feature interaction. We list the supported training heuristics and compare the performance (i.e., MSE and Kendall τ ) of Wide & Deep Predictor over various NAS predictors under 10 different runtimes. Compared with all non-gradient predictors (Chen & Guestrin, 2016; Ke et al., 2017; Duan et al., 2020), Wide & Deep predictor achieves lower MSE and higher Kendall τ in accuracy prediction, thanks to the potential of ranking-aware training and predictor pretraining via gradient-based optimization. Compared to gradient-based predictors using either dense or sparse features, Wide & Deep predictor achieves better prediction quality thanks to better architecture representation. Such improvement in predictor precision eventually yields the discovery of better NAS models, see the elaborations in Section 6.2.
sections.heading: 5 EXPERIMENTS
sections.text: In this section, we evaluate MAKPConv over classification (ModelNet40 (Wu et al., 2015)) and semantic segmentation (SemanticKITTI (Behley et al., 2019)) benchmark.
sections.heading: 5.1 SEARCH CONFIGURATION
sections.text: MAKPConv Design Space. Each block can choose from kernel size {5, 7, 13} to perform MAKPConv operation. Each block has up to 3 different choices for block depth, block width, and expansion factors. In addition, each block has the choice of an optional NK attention to balance performance and efficiency. We describe other details of the MAKPConv design space in Appendix.
  Regularized Evolution. Following Real et al. (2019), we employ regularized evolution and our Wide & Deep predictor to effectively probe the MAKPConv design space. Given a candidate architecture A with predicted performance P̂ and MACs M , we use S(A) = P̂ − β × logM as our
  search objective, and empirically set β to 0.5 to balance performance and resource consumption on both ModelNet40 and SemanticKITTI. We use a population of 200 and sample size of 150 to carry regularized evolution with 360 rounds over MAKPConv design space to craft NAS models.
sections.heading: 5.2 EVALUATION ON 3D BENCHMARKS
sections.text: We use our Wide & Deep predictor to automate the design of MAKPConv blocks and discover promising models on ModelNet40 and SemanticKITTI 3D applications. To ensure a fair comparison with larger models on SemanticKitti, we apply width scaling (Sandler et al., 2018) on the NAScrafted models and attach a m× suffix to denote the application of a m× width multiplier. We compare the performance and efficiency metrics such as parameter count and MACs.
  ModelNet40. We follow the RSCNN protocol in SimpleView (Goyal et al., 2021) and show the evaluation results on the testing dataset of ModelNet40 in Table 3. For fair comparison, we report the mean accuracy across 5 runs and observe a 0.2% OA variance. Compared to KPConv, the hand-crafted MAKPConv achieves on-par OA with∼92% fewer parameters thanks to the proposed MAKPConv module. Our NAS approach explores more efficient model designs: the NAS-crafted MAKPConv achieves 0.5% higher OA than hand-crafted counterpart while being 55% smaller. Compared with efficient 3D models (Lei et al., 2020b; Goyal et al.,
  2021; Lin et al., 2020; Liu et al., 2019; Wang et al., 2019), NAS-crafted MAKPConv achieves onpar performance with ∼30% fewer parameters. By further scaling up the model, our NAS-crafted MAKPConv outperforms its hand-crafted counterpart by 0.8% OA with a similar parameter count.
  SemanticKITTI. Table 4 shows the mIOU comparison on sequence 08 of the SemanticKITTI dataset. Our hand-crafted MAKPConv outperforms point-based, projection-based and voxel-based methods by at least 0.9%, 1.1% and 1.2% mIOU respectively, with significant parameter and MAC reduction. The NAS-crafted MAKPConv achieves ∼ 1% higher mIOU than the state-of-the-art SPVNAS while saving 83% parameters and 78% MACs. Applying width multiplier on NAS-crafted model further boosts the performance. We elaborate on the latency comparison in Appendix A.7.
sections.heading: 6 ABLATION STUDIES
sections.text: In this section, we first demonstrate how NK attention re-calibrates the contribution of each kernel point by visualizing the relationship between NK attention and NK linear correlation. Then, we discuss the relationship between predictor MSE reduction and quality improvement of the discovered NAS models to demonstrate the importance of predictor design in predictor-based NAS.
sections.heading: 6.1 KERNEL CONTRIBUTION VIA NK ATTENTION
sections.text: Empirical evaluations reflect that NK attention leads to 0.4% OA gain and 0.5% mIOU gain on 3D object classification and semantic segmentation. To verify the source of improvement of MAKPConv over its non-attention counterpart and justify the re-calibration of kernel points, we plot the learned NK attention on 7 kernel points in Figure 3 and identify the contribution of each kernel via the slope. We observe that the learned attention for each kernel point has a strong relationship with kernel dispositions (i.e., 1-4-2 groups among symmetrical axis) in the Octahedron.
  The two vertices that are the most distant from the center point shows the most contribution towards neighbor points. This is because these kernel points are involved in addressing the boundary effects of input point clouds for more robust recognition results. We also observe that the 4 vertices connecting the square plane has the least contribution, illustrating that such high concentration of kernels causes redundancy and suggests a future direction in optimizing kernel location to fully capitalize the kernel points to improve representation power. The center kernel point admits most information from centralized neighbor points, thus co-adapts to global
  neighbor information and yields an moderate contribution towards neighbor points.
sections.heading: 6.2 EFFECTIVENESS OF WIDE & DEEP PREDICTOR
sections.text: We show that even marginal MSE reduction (i.e., 0.07×10−2) leads to significant quality improvement of NAS-crafted models, which justifies the importance of feature and architecture engineering in predictors. We establish random search baseline by selecting the top-5 models from the outcome of 1K random samples, and establish predictor-based NAS baseline by selecting top-5 models using the top-performing Wide NN predictor in Table 5. All models are evaluated on classification and segmentation benchmarks using the same 100-epoch training pipeline. We report the mean and standard deviation of performance on ModelNet40 and SemanticKITTI in Table 5.
  Results demonstrate that: (1) without introducing extra architecture samples, predictor-based NAS can outperform random search by up to 1.4% mIOU on SemanticKITTI. (2) Even with 0.07×10−2 lower MSE, the NAS-crafted models via Wide & Deep predictor can outperform NAS-crafted models via Wide predictor by a large margin (i.e., up to 1.2% mIOU on SemanticKITTI). The above results emphasize the need to improve predictor-based NAS via a delicate neural predictor design.
sections.heading: 7 CONCLUSION
sections.text: In this work, we present MAKPConv module to improve both efficiency and performance of KPConv. MAKPConv employs depthwise KPConv that separates point-wise feature extraction from channel-wise feature extraction to reduce computation cost and memory consumption, and NK attention that re-weighs the contribution of different kernel points via interactions of neighbor-kernel correlation through an learnable attention network. We further incorporate the IRB design to construct MAKPConv block and propose a new predictor-based NAS driven Wide & Deep predictor to automate the model design. Experiments and ablation study over 3D classification and 3D semantic segmentation benchmarks verify the effectiveness of MAKPConv.
sections.heading: 8 REPRODUCIBILITY STATEMENT.
sections.text: We make tremendous efforts to ensure reproducibility of our proposed method. Specifically, we mention the detailed configuration of predictor-based NAS in Section 5.1, introduce the design of hand-crafted MAKPConv design in Appendix A.1/A.2, describe the detailed configuration of Wide & Deep predictor in Appendix A.3, show the transferability of searched architecture in Appendix A.4, cover the training protocols in Appendix A.5, and use Appendix A.6 to justify orthogonal contribution of our improving performance predictor and other line of research in improving search algorithms. In addition, we use Appendix A.7 to give an analysis on the latency breakdown of searched models and points an direction on further improving the efficiency of MAKPConv kernel to seek higher efficiency in 3D point cloud applications. Code that covers other details of this work is available via this link.
sections.heading: A APPENDIX
sections.text: In this appendix, we elaborate the detailed structure of hand-crafted MAKPConv via MAKPConv blocks. We also verify the transferability of NAS discovered architecture on S3DIS (Armeni et al., 2016) indoor scene segmentation benchmark. Furthermore, we discuss detailed hyperparameter settings for training and evaluating 3D architectures on ModelNet40, S3DIS, and SemanticKITTI benchmarks, and visualize the optimization curve in the search process for both vanilla predictorbased NAS (i.e., via Wide Predictor) and our improved predictor-based NAS (i.e., via Wide & Deep predictor). We also discuss the latency breakdown of our searched models and propose some future work in improving the efficiency of MAKPConv series models.
  A.1 HAND-CRAFTED MAKPCONV NETWORK
  Stage Depth Stride In Out E KWidth Width 1 1 1 16 16 1 7 2 2 2 16 24 3 7 3 3 2 24 32 3 7 4 4 2 32 64 3 7 5 3 1 64 96 3 7 6 3 2 96 160 3 7 7 1 1 160 320 3 7 8 1 1/2 416 160 3 7 9 1 1/2 160 96 3 7 10 1 1/2 96 64 3 7 11 1 1/2 64 32 3 7
  serves as a strong baseline to compare with when evaluating the performance of NAS-crafted MAKPConv models.
  A.2 MAKPCONV DESIGN SPACE
  We present the detailed settings of the MAKPConv design space in Table 7. Despite the search of block depth, block width, expansion factor and kernel size, we also incorporate the search of optional attention into MAKPConv design space to trade-off the overhead in Neighbor-Kernel Attention and its performance gain.
  A.3 DESIGN OF WIDE & DEEP PREDICTOR
  We configure the architectural hyperparameters of our wide & deep predictor via intuition without delicate architecture engineering. Specifically, we employ a 3-layer MLP with 64-128-256 layers as the dense architecture to extract dense neural architecture representations for both positional
  organizations and structural settings. The overall architecture that is responsible for processing fused features after dense-sparse interaction is a 2-layer MLP with 256-256 layers. We utilize ReLU as the activation function and apply Dropout of 0.5 before the final regression head to mitigate overfitting. To ensure fair comparison, both the Wide predictor and the Deep predictor also adopt a 2-layer MLP with 256-128 units to keep the same maximum projection dimension as the Wide & Deep predictor.
  A.4 ARCHITECTURE TRANSFERABILITY ON S3DIS
  We further verify the transferability of NAS-crafted MAKPConv architectures on S3DIS to evaluate its performance on indoor scene segmentation. Following the established protocols in 3D point cloud segmentation, we report both 6-fold cross-validation mIOU across all Area 1∼Area 6 splits, see Table 8. By convention, we also report the validation mIOU and mIOU per class on Area 5 to compare with existing methods on 3D point cloud, see Table 9. Though our NAS-crafted model is not optimized in S3DIS dataset, it shows a significant margin on S3DIS 6-fold crossvalidation benchmark compared with existing approaches such as Qiu et al. (2021); Zhao et al. (2020) and achieves 74.5 mIOU, a new state-of-the-art result using only 1.35M parameters. On Area 5 of the S3DIS dataset, we observe that: (1) Our NAS-crafted MAKPConv achieves 1.8% higher mIOU than original KPConv with 11× parameter efficiency. (2) Our NAS-crafted MAKPConv achieves competitive performance over different classes on S3DIS indoor semantic segmentation and demonstrate ∼1% higher mIOU over 9 out of 13 classes compared to existing approaches. Thus, our NAS-crafted models have a good transferability towards alternative benchmarks, and our NAS approach can be further incorporated into other 3D point cloud tasks such as nuScenes (Caesar et al., 2019), KITTI (Geiger et al., 2013) to improve the current state-of-the-art models.
  A.5 HYPERPARAMETER SETTINGS
  The implementation of our work is based upon the official PyTorch repository of KPConv (Thomas et al., 2019). We follow the exact evaluation protocol in this repo to produce our validation results. For training, we demonstrate the detailed hyperparameter settings on 3 different benchmarks (i.e., ModelNet40 (Wu et al., 2015), SemanticKITTI (Behley et al., 2019) and S3DIS (Armeni et al., 2016)) in the following context.
  ModelNet40. A single training batch in ModelNet40 contains 16 sub-sampled point clouds. We adopt a similar training pipeline following the original KPConv (Thomas et al., 2019) paper yet incorporates SimpleView (Goyal et al., 2021) RSCNN training protocol for fair comparison. Specifically, the SimpleView-RSCNN protocol adopts random scaling & translation for data augmentation, employ cross-entropy loss to optimize, and utilize a voting scheme on the best test model to best exploit the potential of searched model. We train our NAS-crafted model for 250 epochs with an initial learning rate of 0.02 and cosine learning rate schedule (Loshchilov & Hutter, 2016). To combat overfitting, we also adopt a dropout rate of 0.5, a L2 weight decay of 3e-4 and employ early stopping to select the best model for testing.
  S3DIS. A single training batch in S3DIS contains 8 sub-sampled point clouds. We use a downsample rate of dl0 = 0.04m following the original KPConv paper (Thomas et al., 2019). Specifically, we
  train our best model for 250 epochs with an initial learning rate of 0.04 and cosine learning rate schedule (Loshchilov & Hutter, 2016). To combat overfitting, We also adopt a L2 weight decay of 3e-4 and the default data augmentation in KPConv paper.
  SemanticKITTI. A single training batch in SemanticKITTI contains 10 sub-sampled point clouds. On SemanticKITTI, we measure Multiply-Accumulates (MACs) over a scene with an average of ∼12.3K points, which gives similar MAC count for KPConv architecture as is shown in (Tang et al., 2020). We use a downsample rate of dl0 = 0.06m following the original KPConv paper (Thomas et al., 2019). Specifically, we train our best model for 250 epochs with an initial learning rate of 0.04 and cosine learning rate schedule (Loshchilov & Hutter, 2016). To combat overfitting, We also adopt a L2 weight decay of 3e-4 and the default data augmentation in KPConv paper.
  A.6 OPTIMIZATION CURVE ON DIFFERENT PREDICTORS
  In this section, we demonstrate that the improvements of the proposed predictors has no interference over the improvements in search algorithms. More specifically, our contribution towards leveraging a better neural architecture representation is orthogonal to the contribution in introducing a better search algorithm. We apply 3 popular search algorithms: Random Search, Regularized Evolution (Real et al., 2019) and Reinforcement-Learning via Proximal Policy Optimization (PPO) (Zoph & Le, 2016; Schulman et al., 2017) on both Wide predictor and Wide & Deep predictor, and plot the normalized performance for visualization, see Figure 4. Despite the greater complexity of Wide & Deep predictor that may potentially formulate a curvy performance surface, we observe that the superiority of EA on Wide predictor is successfully preserved on Wide & Deep predictor. This indicates that the improvement of Wide & Deep predictor is consistent with the evaluation of different search algorithm, and sets up a new direction of improving the design of neural predictors to enhance predictor-based NAS.
  A.7 LATENCY ANALYSIS OF MAKPCONV KERNEL
  In Table 4, we notice that our NAS-crafted MAKPConv model is 24% faster than original KPConv model despite of its higher mIOU. However, we also observe that the latency reduction is less significant compared to MAC reduction of MAKPConv models, especially compared to voxel-based methods. This is because voxel-based methods (e.g., SPVNAS (Tang et al., 2020)) benefit from 1) reduced overhead in point-based neighboring mechanisms 2) mature software-hardware co-design that enables inference with high throughput. To dive deep into this issue, we profile the latency breakdown of the NAS-crafted MAKPConv model and demonstrate the profiling result in Table 10.
  Here, the preprocessing operations (including radius sampling, grid sub-sampling etc.) consumes 33.7% of the inference cost, neighbor gathering contributes to the memory-bounded operation (i.e., aten :: gather), which takes 10% of the total inference latency. Computation-bounded parts in KPConv such as depthwise convolution (i.e., aten :: mul and aten :: sum), MLPs (i.e., aten :: bmm)
  and point-wise local neighborhood movement (i.e., aten :: sub) contribute to ∼50% of the total inference cost. Although radius sampling still consumes moderate cost in MAKPConv, there is still a need to leverage design automation algorithm (e.g., our proposed MAKPConv algorithm) to perform optimization on the overall architecture on MAKPConv and seek better inference efficiency by improving computation-bounded operations.
  Besides the optimization of architecture, we also discover some potential improvements that harnesses the hardware-software co-design to improve efficiency of MAKPConv blocks, such as yet not limited to:
  • Efficient radius sampling implementation. The current implementation lies in CPU and is not optimal for parallelization. A possible CUDA version of the radius sampling can speedup the inference speed of MAKPConv networks.
  • The current MAKPConv does not utilize a fused version of convolution that aggregates the feature outputs on each kernel point efficiently. Instead, the output features on each kernel point is firstly computed via aten :: bmm and then aggregated via aten :: sum.
  • The current process of moving points towards their local neighborhoods controlled by aten :: sub is far from efficient and consumes ∼ 20% latency. Thus, a specialized hardware for efficient batch points movement is potentially beneficial for efficient inference on point-based networks.
  • As lower radius leads to more efficient sampling, we consider incorporating the choice of radius for each MAKPConv layer into the design space and harness such option to discover more efficient point-based 3D networks.
  As such, with heavy optimization on MAKPConv design, we can reduce the uncertainty and/or complexity in profiling MAKPConv blocks and thus push forward latency-aware search for better hardware specialization. We leave this as our future work to improve the design efficiency of 3D architectures on processing point cloud data.