RandomForestMLP: An Ensemble-Based Multi-Layer Perceptron Against Curse of Dimensionality

We present a novel and practical deep learning pipeline termed RandomForestMLP. This core trainable classification engine consists of a convolutional neural network backbone followed by an ensemble-based multi-layer perceptrons core for the classification task. It is designed in the context of self and semi-supervised learning tasks to avoid overfitting while training on very small datasets. The paper details the architecture of the RandomForestMLP and present different strategies for neural network decision aggregation. Then, it assesses its robustness to overfitting when trained on realistic image datasets and compares its classification performance with existing regular classifiers.

Introduction
Modern computer vision systems have achieved outstanding performance on a variety of complex tasks such as image recognition, object detection, and semantic segmentation. Their success depends essentially on the availability of large annotated datasets. However, acquiring such a dataset is time-consuming and requires expensive storage capabilities. Real-world computer vision applications are often concerned with visual categories that are not present in standard benchmark datasets or with applications of a dynamic nature where visual categories or their appearance may change over time. Various semi-and self-supervised learning [15] techniques have been used to automatically label the training set based on a small but reliable annotated dataset. However, as the dimensionality of the data points increases, especially when it comes to 2D images, automatic labeling techniques become inefficient due to the overfitting issue also known as the "curse of dimensionality." In this paper, we propose a new classification method inspired by the random forest [3] algorithm and based on the fusion of several multi-layers perceptrons (MLPs) using features and data bagging techniques. This method will be assessed against benchmark datasets and then applied to examples of image classification.
ensemble-based classification technique. [5] has designed an ensemble-based neural network architecture using training data bagging. Given a data set of N patterns, [5] builds L bootstrap samples by randomly sampling N patterns with replacement. Each bootstrap is fed forward to a single multilayered perceptron. The final model output is computed by averaging L outputs for a regression problem or by majority voting for a classification problem. Although bootstrapping aggregation is used to limit the impact of outliers on decision space construction and hence helps avoiding overfitting, feature-based bootstrapping introduced by [3] and commonly used in the regular random forest [3] model reduces the effect of the curse of dimensionality, especially when it comes to the small training datasets . [14] introduced the feature count measure that helps construct appropriate feature subspaces. His strategy helps increase the accuracy of ensemble-based decision trees when trained on UCI small datasets [2]. To the best of our knowledge, ensemble-based neural network architecture using the feature bagging technique does not exist. It extends the concept of feature bagging to neural networks.

Context and Methods
Several supervised learning approaches have been introduced and tested on various datasets. However, when it comes to automatically learning specific patterns from unlabeled datasets, several issues may occur, and the curse of dimensionality is one of them. Indeed, when training a relatively small dataset compared to its feature dimensionality, the classifier, especially a highly complex one, struggles to build a generalized decision space without overfitting. In image processing, one way to overcome this issue is to train a convolutional backbone to recognize the most important patterns inside the image, generate a feature map, and feed it forward to a simple multilayered perceptron to achieve classification. Our RandomForestMLP is based on the classical model of random forests [3] with decision trees. In contrast with the random forest [3], which uses a specific number of classifiers and each one of them is trained using a random number of features (i.e., features bagging), the RandomForestMLP model uses all the subsets of N-1 features, where N refers to the total number of features.
The figure 1 shows the four principal components of the RandomForestMLP architecture: convolutional neural network (CNN) backbone, feature selector, MLP Forest, and decision function.

CNN Backbone and Feature Selector algorithm
A modified version of VGG16 [12] including batch normalization layers and using the parametric rectified linear [11] activation function instead of non-learnable rectified linear function was trained and its feature map was extracted and fed forward to the selection algorithm. It returned all the possible combinations of feature subsets. Each subset consists of N-1 features, where N refers to the total number of features in the feature map.

MLP Forest and Decision Function
It consists of N-1 multi-layer perceptron classifiers, each one is composed of one hidden layer followed by a fully connected layer. One MLP predictor is trained on the transformed feature map and returns a list of C probabilities where C is the number of classes. Those lists are fed forward to the decision function block to predict the final class member of each data point according to two different strategies.
Voting strategy The classification probabilities are filtered and binarized according to the following policy: if (max pcj (i) (p cj (1), ..., p cj (C)) > 1 2 then the classifier cj decision is equal to argmax pcj (i) (p cj (1), ..., p cj (C)) otherwise it is rejected. All the filtered class candidates are aggregated and we only chose the class with the major vote. This voting strategy combined with the filtering process might provide more confident classification results and hence would be more robust to overfitting.
Probabilistic Strategy This process is more subtle and relies on two different probabilistic approaches. The probability P (y = i|x) to predict one class given its data point parameters is : where C refers to the set of N-1 classifiers and x j to datapoint projected on their corresponding features.P (y = i|c j , x j ) corresponds to the i − th perceptron value of the "Softmax" layer of the j − th MLP classifier. Since the classifiers are blind to the information value of each feature, we will consider that all the features are relevant and hence all the classifier's outcomes are equivalent. Therefore P (c j ) = 1 N C where N C refers to the dimensionality of the classifiers space. However, the assumption of the classifier's blindness to the relevance of the features could be overcome by sorting the features according to their statistical information value, in other words by computing their variance values or formally by processing a principal component analysis decomposition of the original features. The covariance matrix of the training set X is C = X T X. Its singular value decomposition is given by The transformed data is then X = P Λ −1/2 X. All the classifiers should take into consideration the relevance of the transformed features since they are sorted according to the eigenvalue (λ i ) of their covariance matrix C. One way to achieve this operation is to penalize the classifiers that reject features with high eigenvalues according to the following classifiers probability distribution: P (c j ) =

Training datasets and materials
The RandomForestMLP model without the Feature map extractor part was trained on several UCI datasets [2] such as (Iris plants dataset [2], Wine recognition dataset [2], Optical recognition of handwritten digits dataset [2], etc). Each dataset was split according to their class members into K folds, and a cross-validation based assessment is performed: one fold is used for the training process, and K-1 are allocated to the validation process. K has to be large enough to induce a curse of dimensionality issue(i.e, the following condition must be satisfied C N < M K where C, K, M, and N refers respectively to the number of different classes, the number of folds, the size of the training set and the dimensionality of the dataset). Other realistic 2D image dataset (e.g, Cassava [10] dataset, beans [9] and Cars [8] dataset) were used to test the whole RandomForestMLP Pipeline classification performance. All experiments were carried out on an IBM Power Systems AC922 with 256 GB of RAM and 4 NVIDIA V100 16 GB GPUs (using only a single GPU). Since on this platform, the GPU and the processors share a coherent memory space. We cannot guarantee that the previous settings are reproducible on Intel/AMD based machines as they may require smaller batches.

Architectures experiments
Two different steps of experiments were achieved to assess the efficiency and the potential of our algorithm. they consist of respectively training the RandomForestMLP on small standardized datasets without the CNN backbone feature extractor then testing it on realistic image datasets using an enhanced version VGG16 [12] architecture to select and compress the relevant patterns of an image. several multi-layer perceptrons algorithms were used to classify the standardized dataset. The exact number of classifiers depends only on the number of features subsets used. However, reducing the features space dimensionality would result in too many possible combinations of subsets and hence a huge number of classifiers. The figure2 shows the number of trainable parameters inside the MLPForest part and dimensionality of the feature sub-spaces of four different standardized datasets.  [2], (Upper Right) digits dataset [2],(Lower Left) Wine dataset [2], (Lower Right)breast cancer wisconsin dataset [2] The number of parameters of the MLP Forest is proportional to the number of all possible feature subspaces with a given dimensionality(i.e, ( n k )) which become very large when n >> 1 and 1 << k << n. This behavior is confirmed by figure 2. Hence, we choose to keep only the feature space with very high dimensionality(i.e, N f = N-1 where N f and N refer respectively to the number of feature in the dataset and dimensionality of the feature subspace) to limit the computational expenses of the MLP Forest algorithm and to preserve feature information. For computational purposes, each MLP Forest classifier consists of only one hidden layer, and each hidden layer is composed of 100 perceptrons. The hyperparameter of the MLP predictors are tuned as follows: we chose an Adam [7] optimizer with an initial learning rate of 10 −3 , a learning rate scheduler is also used to avoid undesirable divergent behavior while increasing the number of epochs(the learning rate drops to 10 −4 after 50 epochs). Each MLP is trained on 200 batches of the same size for 100 epochs and an early stopping strategy was applied to avoid overfitting issues. VGG16 [12] was used as a feature extractor backbone. It was first trained for 30 epochs on each dataset. Then, the weights of the CNN blocks were frozen and the final fully connected layer was removed.

Results
In this section, we will assess and compare the performance of our model with state of the art algorithms. We will demonstrate the capacity of the RandomForestMLP in reducing the overfitting effects when it comes to training with small datasets. The table below summarizes the weighted F1-score [4] of different classifiers trained on a small portion of the dataset (less than 20%) and tested on the rest. One major conclusion for the results given by the table 1 is that the Random-Forest-MLP model outperforms all the state of the art classifiers including random forest [3] with decision trees. However, the Weighted Probability strategy used in the decision function was effective and enhanced the F1-Score [4] of all the classification task by a minimum of 4% compared to Major voting strategy and equiprobable feature space method except for Digits dataset where Majority vote acheives better classification performance. Unlike Standardized datasets which are relatively easy to learn, 2D realistic images classification task is more complicated and will be a tougher assessment of our model. The table below compares the performance of a modified Version of VGG16 [12] architecture with our model using different decision function methods.
As for the standard dataset classification assessment, one major conclusion could be derived from the table 2. Indeed, our model clearly outperforms the modified version of the VGG16 [12] on all the classification tasks. However, the choice of the decision function doesn't affect significantly the classification performance of the Random-ForestMLP algorithm. The slight variance in F1-Score [4] between the three strategies is due to some features with high variance but less effective that misleads the classification balance strategy. Although the F1-scores [4] of the RandomForestMLP model shows its outstanding performance compared to regular CNN models when it comes to train on small portion of the dataset, it doesn't prove its robustness to overfitting. The four images in the figure 3 shows the variation of the validation and the training accuracy over 30 iterations using different models and image classification datasets. From the figures we observe that the variation in validation accuracy of the regular VGG16 [12] does not follow the increase in training accuracy, which illustrates the overfitting issue. This behavior tends to reduce when the RandomForestMLP model is trained with its different decision function strategies

Conclusion
In this paper, we have proved that an ensemble based neural network architecture with any decision function could outperform all the classification model build so far especially when training on small datasets. Indeed, while all regular classification models are affected by overfitting issues due to the curse of dimensionality in case of shortage of training data points, our model is more robust to this issue. RandomForestMLP usage could be extended to other realistic and more sophisticated tasks such as fine grained image segmentation and self-supervised 3D images classification.


NwQM: A neural quality assessment framework for Wikipedia

Millions of people irrespective of socioeconomic and demographic backgrounds, depend on Wikipedia articles everyday for keeping themselves informed regarding popular as well as obscure topics. Articles have been categorized by editors into several quality classes, which indicate their reliability as encyclopedic content. This manual designation is an onerous task because it necessitates profound knowledge about encyclopedic language, as well navigating circuitous set of wiki guidelines. In this paper we propose Neural wikipedia QualityMonitor (NwQM), a novel deep learning model which accumulates signals from several key information sources such as article text, meta data and images to obtain improved Wikipedia article representation. We present comparison of our approach against a plethora of available solutions and show 8% improvement over state-of-the-art approaches with detailed ablation studies.

Introduction
Wikipedia is one of the most prominent sources of free information in the world today. Since reliable and advertisement free material is mostly behind pay-walled sources, a huge volume of global population is directed toward this extraordinary crowd sourced platform for information ranging from history, politics, pop-culture to even scientific topics (Horta Ribeiro et al., 2020).
Although Wikipedia has grown significantly in terms of volume and veracity over the last decade, the quality of articles is not uniform (Warncke-Wang et al., 2015). The quality of Wikipedia articles is monitored through a rating system where each article is assigned one of several class indicators. Some of the major article categories are FA, GA, B, C, Start and Stub. Most complete * *Authors contributed equally and dependable content is annotated by an FA (aka featured article) tag while lowest quality content is annotated with a Stub tag. The intention behind this elaborate scheme is to notify editors regarding current state of the article and extent of effort needed for escalating to encyclopedic standards 1 .
There exist several guidelines which direct editors in annotating articles into respective classes. Some of the traits of a FA article are engaging and comprehensive prose with neutral point of view and verifiable claims. It must also rigorously follow the style manual, i.e. the page structure. Further the content should be stable, i.e. devoid of edit warring. Understanding compliance with these guidelines often require detailed knowledge about language usage as well as domain knowledge about Wikipedia page layout and style principles. Often it is nontrivial to discern qualifying differences between articles which merits their ratings without inculcating personal biases.
Consider the wikipage of two prominent US presidents Abraham Lincoln (GA) and John F. Kennedy (B). Both pages are indistinguishable in-terms of coverage and engagement, however on closer assessment it is apparent that President Kennedy's page contains unattributed opinion such as the statement This crisis brought the world closer to nuclear war than at any point before or after ... in the Cuban Missile Crisis section. It also has vague quantifiers such as some questioned, some crtics, somewhat successful etc. Similarly, if we look at wikipages of historical figures Akhenaten (B) and Cleopatra (FA) it is difficult to discern their quality just from the content. A deep dive into the individual talk pages reveal that the former page has unresolved content issues and disputes which justifies the given rating. Hence it is a difficult task to manually judge language specific nuances present in the main page text, topic level disputes manifesting in the talk pages as well as section layout and image positioning before deciding a correct rating. This is also apparent from the quality statistics 2 which shows that only 0.09%, 0.5% of the ∼ 6M English Wikipedia articles have a FA and a GA tag respectively.
Current approaches for automatic quality assessment by Wikimedia foundation 3 use handcrafted features from main article text for classifying quality class (Halfaker and Geiger, 2019). Analogous approaches exist (Dang and Ignat, 2016) which attempt to automatically generate features from main text using deep learning models such as doc2vec (Le and Mikolov, 2014). Other approaches use deep sequence models such as BIL-STM (Shen et al., 2017) as well as combining representations obtained with additional modalities like image (Shen et al., 2019b). The principal focus of existing works have been concentrated on main article text. However one of the key sources of metadata about an article, i.e., the corresponding talk page has been ignored. Talk pages contain crucial information concerning stability of a page. They also hold evidence whether discussion threads encompassing topics are decisive. Besides, representation of main page text using sequence models cannot capture high level semantic signals such as whether the introduction section is a summary, whether the coverage of topics is polarized or the information is redundant or the wording is convoluted.
In this paper we propose Neural wikipedia Quality Monitor (NwQM) which integrates information from multiple sources, i.e., main page text, metadata and html rendering resulting in improved quality assessment. We use bidirectional contextual representation (Devlin et al., 2018) for encoding article text. However, contrary to document representation using BERT (Adhikari et al., 2019), which is not adequate for large text documents, we first segment articles organically based on sections. We fine tune on each section text individually followed by a summarization layer which preserves the sequential nature of sections. Similar representation of atomic units and summarisation is applied on talk pages. We also obtain images from the raw markup using Imagekit 4 . These images are further embedded in a vector space using Inception V3 (Szegedy 2 wiki/Wikipedia:Good article statistics 3 wikimediafoundation.org 4 pypi.org/project/imgkit et al., 2016). Inception V3 is pre-trained on Imagenet 5 and we fine tune on our dataset to cater to our task. Our experiments show that combining information sources from these sources leads to improved result eclipsing current state of the art (Shen et al., 2019b) by 8%.
Our main contributions are enumerated below. 1. We propose a multimodal framework from quality assessment of Wikipedia articles which leverages contextual representation obtained from bidirectional transformers and supports conditional summarization. 2. To the best of our knowledge this is the first work which utilizes meta pages, i.e., talk pages as an additional signal for this task. All code, sample data and image embeddings related to the paper are made available 6,7 to promote reproducible research.

Related work
Automatic article assessment is one of the key research agendas of the Wikimedia foundation 8 . One of the preliminary approaches (Halfaker and Taraborelli, 2015) seeking to solve this problem extracted structural features such as presence of infobox, references, level 2 headings etc. as indicators of the article quality. Other approaches explored distributional representation as well as sequence models (Dang and Ignat, 2016;Shen et al., 2017Shen et al., , 2019b. (Zhang et al., 2018) attempted to solve this problem by formulating features capturing dynamic nature of the articles. A complementary direction of exploration has been put forward by (Li et al., 2015;de La Robertie et al., 2015) where correlation between article quality and structural properties of co-editor network and editorarticle network has been exploited. This task can also be solved by exploring the rich literature of document classification. One of the characteristics of Wikipedia articles are that these are long documents hence sequence models can suffer from (Atkinson, 2018) catastrophic forgetting. (Yang et al., 2016) proposed to solve this by leveraging a hierarchical organization of documents. Further improvements have been demonstrated by employing bidirectional transformers (Adhikari et al., 2019;Ostendorff et al., 2019). However to the best of our knowledge no 5 www.image-net.org/ 6 https://github.com/sasibhushan3/NwQM_EMNLP

Dataset
Wikimedia foundation stores all data for its multilingual wikiprojects in the form of Wikidumps 9 . We downloaded first 100 English Wikipedia dumps which are 7z archived xml files. The combined uncompressed size of these files is ∼ 8T B and it contains a sample of ∼ 6M English Wikipedia text from the first version to last update as of June 2019. Because of limitation of space we did not go though the entire English Wikipedia archive. Each uncompressed file is approximately of 80GB size and have random samples of approximately ∼ 5k Wikipedia pages. We parsed each file using mediawiki xml parser 10 and in one linear scan we tried to locate if the main page text and talk page text is in the same dump xml file. More specifically if we encounter the main Wikipedia article Cleopatra, we remember it in a dictionary and seek to locate Talk:Cleopatra in future scans or vice versa. However it is entirely possible that Talk:Cleopatra is not present at all in the currently encountered xml file and may be present somewhere else; in such case we ignore that article. If we can locate both main and talk pages of the same article we save it for reference. We include in our corpus maximum number of articles for GA, B, C, Start and Stub while maintaining equality. For FA articles, we included entire extracted corpus, because such articles are scarce. Although this process is naïve, yet we manage to extract moderately balanced number of datapoints for each class. We would further like to note that few other public datasets exits for this task. The first version is made available by the Wikimedia foundation 11 which has 30K datapoints with approximately 5k pages in each of the 6 classes. (Shen et al., 2017) has pointed out that this dataset contains many noisy datapoints, e.g., empty pages labeled as FA class. Other datasets are made available by (Shen et al., 2019a,c;Warncke-Wang et al., 2015). Our investigation shows that none of the former datasets contain meta pages which prompts us to extract this novel data ourselves. Also some of the datasets have uniform article length across classes, which is often not the case in reality. For example Dodo and Grey-necked wood rail are both FA articles with very different article length. Considering uniform article length may lead to overfitting. We used stratified random sample of 80% data for training and 10% each for validation and testing. The overall distribution of the articles with respective classes in our dataset is enumerated in Table 1.

Proposed solution
In this section we present a detailed description of our multimodal approach. We start this section by explaining the various notations that we use in the subsequent sections of the paper. We then proceed to first describe representation mechanisms of explicit signals about article quality, obtained from main text, i.e., presence of bias, claim verifiability, coherent wording, citations etc. We further elaborate on the mechanisms employed, to capture implicit quality indicators such as article stability, collaborative nature of editors obtained from article talk text as well as visual renderings of documents to capture how well the article follows the style manual. Further, we describe the different ways in which we combine the information to predict the quality of the page. We present the overall architecture of our model in Figure 1.

Preprocessing
We first pre-process the Wikipedia articles and talk pages to convert the text from Wiki Markup Language 12 to plain English text format using a python text crawler 13 . We then replace the meta content in the page such as infobox, level 1 section headings, level 2 section heading, internal wikilink, external link, inline reference, footnote template, image template, quotation template and categories into special tokens which act as additional features using mediawiki parser 14 . We use this pre-processed text in the subsequent models.

NwQM overview
Our approach is inspired by the hierarchical document representation approach proposed in (Yang et al., 2016), designed to capture signals from multiple levels of document organization. Wikipedia pages have an organic structure, i.e., words form sentences, sentences form paragraphs, paragraphs form sections and sections form a page. We build page representation by first generating section embeddings, followed by a suitable summarization. We use BERT (Devlin et al., 2018) for section representation and bidirectional GRU with self attention for summarization. Explicitly our encoder has two levels in the hierarchy, however implicitly by employing bidirectional transformers (Vaswani et al., 2017) with special tokens, we can aggregate contextual features across multiple levels.

Fine tuning BERT
We fine tune the BERT (Devlin et al., 2018) model on the extracted Wikipedia pages for classifying them into the 6 quality categories. We pass the preprocessed textual content of the Wikipedia page to the BERT model and pass the [CLS] token's representation to a dense followed by softmax layers to classify the page. We follow the fine-tuning strategy proposed by (Sun et al., 2019) for long input sequences to overcome the limitations of BERT in handling inputs more than 512 tokens. We concatenate the first 128 and the last 384 tokens of the page. For the statistical observations on advantage of this approach please refer to (Sun et al., 2019). We use the BERT's tokenizer with additional meta content tokens explained in Section 4.1. We fine tune the model end-to-end (110M parameters). Once the model is finetuned we freeze the weights, thus in subsequent steps of the training the parameters are not updated.

BERT section encoder
We use previously obtained fine tuned BERT for generating section representations. It has been 14 mwparserfromhell.readthedocs.io shown that fine tuned BERT performs extremely well in subject verb agreement task (Goldberg, 2019;Clark et al., 2019), hence it is able to capture long range semantic dependency. Besides, since it is pre-trained with next sentence prediction objective, it remembers context across multiple sentences. Also since 95% of the sections have less than 512 words, we use the pre-trained BERT model with maximum number of input tokens, to directly encode the sections, hence collapsing the hierarchy starting from words upto sections in the encoding process. We collapsed the hierarchy up to sections but do not extend further due to several drawbacks in the BERT model discovered through probing ( Figure 1).

Conditional summarizer
We next generate the page representation (D p i , see Figure 1) from the sequence of sections using attention based bidirectional GRU encoder as discussed in the previous section. The inputs i x in equations (1) to (6) correspond to the section representations obtained from the BERT models.
The α x correspond to the attention weights, the final output o n is the page representation D p i (see Figure 1). To aggregate the sequences at section level Sc x , x ∈ [1, n], we use the bidirectional GRU (Chung et al., 2014) which provides representations of the document text by summarizing information from both directions. We concatenate the forward and backward hidden states h x and feed the hidden states to the self attention module (Lin et al., 2017;Bahdanau et al., 2014).

Talk page encoder
We split the talk page content into sentences and pass each sentence through the Google Universe Sentence Encoder model (Cer et al., 2018) to obtain a 512 dimensional representation of the sentence. These sentence embeddings are then passed through a global average pooling layer (Lin et al., 2013) followed by a dense layer to get a 200 dimensional final representation of the talk page (T p i , see Figure 1). The motivation behind this approach is to obtain an aggregate representation of the discourse in talk pages.

Fine tuned Inception V3
Shen et al. (2019b) shows the effectiveness of using the visual rendering (screenshot image) of a page to predict the quality of a document in a multimodal setup. To embed the visual rendering we use the Inception V3 (Szegedy et al., 2016) model. Similar to the BERT model, to learn better representations, we fine tune the Inception V3 model. We follow the setting proposed in (Shen et al., 2019b) to finetune the Inception V3 model. We resize the images from varied high dimensions to a standard low dimension. We then fine tune the Inception v3 model to classify the screenshots of the pages into the 6 quality classes. We do an end-to-end training for fine tuning the Inception V3 model. We flatten the final representation from the last convolution layer, stack the dropout and global average 2D pooling layers, and classify using softmax layer. The hidden flat layer output (2048 dimension) of the fine tuned Inception V3 model is the final representation of the visual rendering of the page. The input to the fine tuned Inception model is the screenshot of the wikipage, and the output is a visual embedding of the page (I p i , see Figure 1). We integrate this information with the text and talk content of the page and report the improved performance of the final model.

Concatenation module
In this subsection we illustrate the concatenation of the information from various modules representing the features from text and image modalities to assess the quality of a page. We experiment with different modes of concatenation presented in (Reimers and Gurevych, 2019) (|u−v|, u * v), (|u−v|) ,(u * v), and choose the best performing strategy (u, v, |u − v|) to concatenate the vectors u and v. We experiment with various combinations of modules, i.e., conditional docu-ment summarizer (fine tuned BERT encoder with GRU summarizer), talk page encoder and visual rendering of the page (fine tuned Inception V3) and tabulate the best results in Table 2.

Model configurations
We set the hidden states ( − → h x , ← − h x ) of the GRU units of the section encoder to 100. Therefore the final page representation (D p i ) is a 200 dimensional vector. Since 90% of the data has number of sections in a page less than 16 respectively, we limit their maximum size of a page to 16 sections. For sequences less than the specified length, we leftpad using − → 0 . We load the pre-trained weights of BERT-base model from TensorflowHub 15 .
We use the implementation of Google Universal Sentence Encoder available at TensorFlow Hub 16 . We use the nltk library 17 to tokenize the preprocessed talk page content into sentences. We train the hierarchical content encoder for 10 epochs with a learning rate of 0.001 and batches of 16 using Adam optimizer (Kingma and Ba, 2014). For fine tuning the BERT models, we use the Adam optimizer with a learning rate of 2e-5. We empirically set the number of training epochs to 4. To fine tune the Inception V3 model, we again use Adam optimizer with a learning rate 1e-4 and train for 20 epochs. We employ the categorical crossentropy as loss function for all the models and train using batches of size 16. For all the joint models, we set the learning rate to 0.001, batch size to 32, number of epochs for training to 40. For classification, we use dense layers followed by softmax layer. We further utilize dropout probability of 0.5 in the dense layers. Prior fine-tuning of individual units reduces explosive training time, common in end-to-end models

Experiment
In this section we evaluate NwQM against several existing approaches.
Baselines. We provide a brief outline of the competing methods in the following.
• ORES (Halfaker and Geiger, 2019) is a machine learning service made available by Wikimedia foundation through a RESTful HTTP interface serving prediction about target arti-cles. It uses handcrafted features along with gradient boosted machine as classifier. • DOC2VEC (Dang and Ignat, 2016) proposed the first application of deep neural networks into quality assessment task where they employed distributional representation of documents (Le and Mikolov, 2014) without using manual features. • BILSTM+ (Shen et al., 2017) is a hybrid model, where textual content of the Wikipedia articles are encoded using a BILSTM model. The hidden representation captured by the sequence model is further augmented with handcrafted features and the concatenated feature vector is used for final classification • H-LSTM (Zhang et al., 2018) is an edit history based approach where every version of an article is represented by 17 dimensional handcrafted features. Hence an acticle with k versions will be represented by k × 17 matrix. This k length sequence is passed through a stacked LSTM for final representation used in classification. • M-BILSTM (Shen et al., 2019b) proposed a multimodal information fusion approach where embeddings obtained from both article text as well as html rendering of the article webpage is used for final classification. • DOCBERT (Adhikari et al., 2019) proposed this method to generate document representations using bidirectional transformers (Devlin et al., 2018). The primary idea is filtering the representation obtained from the CLS token using a fully connected layer which translates 768 dimensional encoding to class distribution using a softmax layer. This architecture is further fine tuned end-to-end for the respective document classification task. • HAN (Yang et al., 2016) proposed a hierarchical approach which iteratively constructs a document vector by coalescing important words into sentence vectors and subsequently combining important sentences vectors to obtain the document vectors. A convenient consequence of this approach is that it is suitable for large documents like Wikipedia articles.
Result We evaluate NwQM against existing solutions for automatic quality assessment and tabulate the obtained results in Table 2. Since our classes are roughly balanced, we opt to report accuracy as the metric for evaluation. Some of the approaches that we compare our model with are ORES, DOC2VEC, BILSTM+, M-LSTM, H-LSTM. We achieve an improvement of 8% compared to the best performing baseline. This we believe is a considerable leap for a 6 class classification task.
We also compare our model against novel document classification approaches, i.e., DOCBERT and HAN because inherently Wikiepedia quality assessment problem is closely related to document categorization. To this purpose, we compare the document classification approaches with textual content of the Wikipedia main article as well as concatenated version of the main article and talk pages denoted as DOCBERT-wT, HAN-wT respectively. We obtain at most 5% improvement against the existing approaches. Note that apart from our model, including the talk page meta data in the document classification models also considerably enhances their respective performances.
NwQM is constituted of concatenated representation from article text, talk and image and therefore it is important to look at how individual components perform independently. We evaluate NwQM without signals from image (NwQM-w/oI), without talk (NwQM-w/oT) and with solely the main article text, i.e., without any secondary and tertiary signals from image and metadata (NwQM-w/oTI). Our experiment show that the combined approach (NwQM) obtains the best result. Without talk and image we land in a drop of accuracy of 1.3% and 5% respectively validating our hypothesis that extracting signals from external sources can serve fruitful in this task. We also compare against representation from talk pages and fine tuned image embeddings individually for the sake of completeness and our results show significant drop in accuracy compared to the combined approach. ORES Geiger, 2019) 43.21 DOC2VEC wRF (Dang andIgnat, 2016) 44.01 DOC2VEC wLR (Dang and Ignat, 2016) 49.33 BILSTM+ (Shen et al., 2017) 54.5 H-LSTM (Zhang et al., 2018) 53.05 M-BILSTM (Shen et al., 2019b) 58.47 DOCBERT (Adhikari et al., 2019) 57.66 DOCBERT-wT (Adhikari et al., 2019) 59.87 HAN (Yang et al., 2016) 56.35 HAN-wT (Yang et al., 2016) 57  Table 2: Results obtained from different models. The best result is highlighted in green, the best result among document classification models is highlighted in red and the best result among the state-of-the-art quality assessment models is highlighted in blue.

Discussion and qualitative analysis
In this section we perform a deep dive into the predictions obtained by our model and we contrast it with those from closest known competitor M-BILSTM (Shen et al., 2019b) for quality assessment. Confusion matrix: We first tabulate the confusion matrix obtained by NwQM on test data in Figure 2.
Results show that most of the misclassifactions made by NwQM are on average between very similar classes. This is due to the inherent ordinal nature of the classes in this dataset. More specifically, there is an increasing degree of quality in pages from Stub to FA, though the gradient may not be smooth. Thus FA, GA articles have overlapping guidelines which is very different from other pages. Likewise B, C and Start, Stub have mutually overlapping guidelines. NwQM can successfully recover this structure, hence misclassifications have occurred between closer classes. However, com- Distance of wrong predictions from ground truth: In cases of wrong predictions, we calculate how far individual models are from ground truth labels. We transform each class into integers with Stub transformed to 0 and FA transformed to 5. We then find the mean absolute distance of the incorrect predictions from the true labels for each individual class. The results are tabulated in Figure 3 where we observe that for NwQM the absolute distance is consistently lower. The results are statistically significant with a p-value of 1e-5 using Stuart-Maxwell test for multiclass classification (Sun and Yang, 2008).
Effect of article length: We further ascertain how the main text and talk page length of an plays a role in prediction, for respective models. Articles with detailed coverage of topics and multiple discussion threads may not be nominated among high quality articles. This is because even if the coverage is diverse, the language of the article may be biased, convoluted, there could be unverified claims or disaccord among editors. Deep learning models such as BERT have often been shown to rely on surface forms (Ettinger, 2020) as shotcuts for classification instead of semantic understanding. We investigate whether NwQM is relying spurious signals like article length in final classification. For example, article Kauri Gum (GA) and Guar Gum (Start) have very similar coverage, but are distant in terms of the quality criteria. Similarly, article Frog cake (GA) and Sugar (GA) have very different coverage but the same quality tag. For further investigation we rank predictions with respect to article and talk page length and divide our predictions into 4 quan-tiles. We investigate the accuracy of NwQM in the individual quantiles starting from the smallest quantile, i.e., Q 1 to the largest, i.e., Q 4 . Accuracy scores for test set ordered by main article text are illustrated in Figure 4 and those ordered based on talk pages are shown in Figure 5. Results indicate that NwQM is not biased toward length and irrespective of the quantiles we obtain improved prediction accuracy almost always compared to the closest baseline. t-SNE plots: Finally, we visualize the representations obtained by NwQM and M-BILSTM using t-SNE (Maaten and Hinton, 2008) scatter plots (see Figure 6). The degree of separation obtained by NwQM is much better which translates to the improved accuracy. Interpreting the effect of the different modalities: One of the critical issues in deep learning models is interpretability. In order to ascertain whether different sources of signals are indeed contributing toward the final prediction task, we leverage the model agnostic evaluation tool LIME (Ribeiro et al., 2016). We generate representations for text, talk and images, i.e., D p i , T p i , I p i respectively (see Figure 1) from our learned model. These embeddings are passed through classification layers comprising dense layer and softmax layer for prediction. This network takes concatenated input of D p i , T p i , I p i and outputs the classification probabilities for each test instance. We evaluate this black box neural network using LIME. For data points on the test set we identify top 500 features contributing to the outcome of the highest class probability. We further calculate the average contribution from each modality toward the respective classes. More specifically for every test page instance, we identify the top 500 contributing features as per LIME. Each feature can be contributed by any one of the three modalities. We compute the mean of features scores per modality in the top 500 for a page and then aggregate that over all pages. The results are tabulate in Figure 7. Our results show that signals from D p i play most important overall role in prediction of quality. I p i play almost equal role in prediction of all the classes. Interestingly, for high quality pages, T p i , i.e., the talk page information contributes in classification higher than text and image information. Talk also contributes for low quality pages such as Start, Stub. We speculate that since high quality pages have larger discussion archives and low quality pages have very low discussion threads T p i plays significant role in distinguishing these classes. We stress that these interpretability results are the prime insights that these paper neatly establishes.

Conclusion
In this paper we proposed a novel multimodal deep learning based model NwQM for quality assessment of English Wikipedia articles. Our model combines signals from article text, meta pages and image rendering to construct an improved document representation. We evaluate it against several existing approaches and obtain at most 8% improvement compared to the state-of-the-art method. For a 6 class classification this leap in accuracy is notable. We also perform extensive investigation of the different components of our model to understand their individual utility. We perform in-depth qualitative analysis of the obtained predictions and contrast them with the closest baseline.
To the best of our knowledge this is the first work which combines several aspects of information available for Wikipedia articles and, in particular, the talk page dynamics toward quality assessment. We also showcase the utility of fine tuned bidirectional transformers toward document classification especially when combined with niche platform specific signals. We believe our work opens up the necessity of further investigation pertaining to careful information fusion techniques for downstream tasks.


Unsupervised Partial Point Set Registration via Joint Shape Completion and Registration

We propose a self-supervised method for partial point set registration. While recent proposed learning-based methods have achieved impressive registration performance on the full shape observations, these methods mostly suffer from performance degradation when dealing with partial shapes. To bridge the performance gaps between partial point set registration with full point set registration, we proposed to incorporate a shape completion network to benefit the registration process. To achieve this, we design a latent code for each pair of shapes, which can be regarded as a geometric encoding of the target shape. By doing so, our model does need an explicit feature embedding network to learn the feature encodings. More importantly, both our shape completion network and the point set registration network take the shared latent codes as input, which are optimized along with the parameters of two decoder networks in the training process. Therefore, the point set registration process can thus benefit from the joint optimization process of latent codes, which are enforced to represent the information of full shape instead of partial ones. In the inference stage, we fix the network parameter and optimize the latent codes to get the optimal shape completion and registration results. Our proposed method is pure unsupervised and does not need any ground truth supervision. Experiments on the ModelNet40 dataset demonstrate the effectiveness of our model for partial point set registration.

INTRODUCTION
P OINT set registration is generally defined as estimating the spatial transformationeither rigid or non-rigid transformation to align one point set to another. It is a key component in various applications such as robotics, shape correspondence, and large-scale 3D reconstruction [1]. During the past decades, the point set registration problem has been an active research area in the computer vision community, with a large number of both traditional and recent approaches [1], [2], [3], [4], [5].
Given a pair of point sets, traditional methods mostly solve the registration problem by iteratively optimizing a pre-defined alignment loss between the transformed source point sets and their corresponding target ones. These methods, however, need to start over a new optimization process for each new pair of inputs and require an individual tuning process for the hyperparameters for each case. Therefore, the trade-off between efficiency and effectiveness becomes one major concern for traditional methods.
In recent years, with the prevalence of deep learning in various vision tasks, researchers have been shifting their focus from "case-driven" iterative-based methods towards "data-driven" learning-based methods for point set registration. Recently proposed methods, such as PointNetLK [6] and Deep Closest Point (DCP) [1], show that deep neural networks with strong non-linear modeling capacity have superior performance in comparison to classical methods. After training, deep learning-based methods can produce high-quality registration outputs in one network forward pass. Most importantly, these methods show good generalization abilities when transferred to unseen pairs, even when trained on different datasets. Learning-based methods greatly enhance the registration efficiency and enable the possibility of registration of large datasets. Despite the great successes achieved, these methods suffer from performance degradation when the input point sets have missing parts.
More recently, in [7], the author proposes probably the first method for partial point set registration. The authors firstly detect keypoints from partial observations and then learns a mapping from keypoints of source shape to keypoints of the target shape as the desired transformation. This method needs to refine the alignment in an iterative process, which makes the method timeconsuming. Moreover, their registration performance highly depends on the performance of keypoint detection.
In this paper, we approach the partially point set registration task from a different perspective. Our method is motivated by two general findings: 1) the problem of point set registration on full point sets is a well-studied problem with satisfying and significantly better performance than the problem of partial point set registration. For example, the recently proposed DCP [1] method reports a mean square error of 45.01 and 1.31 for rotation angle prediction on the ModelNet40 partial and full point sets, respectively; 2) recent progress in deep learning fields has seen a lot of learning-based methods for shape completion from partial observations. These findings naturally motivate us to handle the partial point set registration problem to recover full shapes from partial inputs while performing point set registration. Therefore, in this paper, we proposed to achieve partial point set registration by jointly optimizing two tasks: 1) to recover full 3D shapes from the input partial point sets, and 2) to learn a rigid transformation from source point set to the target point set. See Figure 1 for illustration.
In order to integrate these two tasks for joint optimization, we proposed to incorporate a latent code for each pair of shapes, which captures the geometric essence of the input shapes. Both our shape completion and point set registration network take a shared latent code as input and optimizes it during network training. Our shape completion network enforces the optimized latent codes to represent the information of full shapes instead of partial inputs. Therefore, our point set registration network can benefit from the shape completion network during the joint optimization of the shared latent code. Our contributions are listed as below: • We propose a novel unsupervised point set registration framework that enables partially point cloud registration using deep neural networks.
• To eliminate the explicit design of the feature encoding network, we formulate a learnable latent vector for each shape as the global feature representation.
• Our model achieves simultaneously shape completion and point set registration by optimizing a shared latent vector. The point set registration network can thus benefit from the shape completion network that enforces the learned latent code to represent the information of full shape instead of partial ones.
• Experiments on the ModelNet40 benchmark dataset demonstrate the effectiveness of our model for partial point set registration.

Point Set Registration
Point sets registration is generally defined as finding the geometric transformations to align the source point set with the target one. Classical iterative optimization methods usually search a set of optimal parameters of a transformation during the process of minimization of a pre-defined alignment loss. The Iterative Closest Point (ICP) algorithm [2] is one of the classical solutions which achieves great success for rigid point set registration. ICP leverages a set of corresponding points to define the transformation and iteratively refine the transformation by minimizing an error metric. In ICP, the initial guess of the desired rigid transformation can have a large impact on the final performance. Yang et al. [3] propose Go-ICP to release the local initialization problem of ICP by using a BnB searching scheme over the entire 3D motion space. Other methods such as TPS-RSM [8] and CPD [9] propose solutions to solve non-rigid registration problem. One common disadvantage of these transitional methods is about registration efficiency. All these methods register every single pair of source and target point sets as an independent optimization process, which cannot transfer knowledge from registering one pair to another.
Recently, deep learning methods have achieved great success in many computer vision applications, including image classification [10], [11], semantic segmentation [12], [13], object detection [14], [15], [16], image registration [17], [18], point signature learning [19], [20], [21], [22], etc. Many researchers shift their attention from classical methods to learning-based methods to approach the point set registration problem [7], [17], [19], [20], [23], [24]. Aoki et al. [6] propose PointNetLK to combine the Lucas & Kanade algorithm with the PointNet feature exaction module into a single trainable recurrent deep neural network. Liu et al. propose FlowNet3D [25], which predicts the flow field to move the source point set towards the target one. Deep Closest Point [1] leverages a DGCNN structure for point feature learning, followed by an attention-based feature matching module to generate the correspondence. Wang et al. [5] proposed PR-Net for learning the partially rigid registration. PR-Net firstly leverages a keypoint detector to find the corresponding points of the source and target point sets. The keypoint is determined based on the per-point feature embeddings of input shapes. Then based on the matching keypoints, the desired mapping can be further predicted. Unlike DCP and PR-Net that both use an explicit point feature encoding network to learn per-point features, in our model, we represent the geometric features of 3D shapes using an optimizable latent vector and thus enable joint shape completion and point set registration.

Shape Completion
3D shape completion is generally defined as recovering the unseen parts of the original partial observations. It's a long-standing problem in computer vision and graphics fields with extensive researches, including both traditional non-learning based methods and recent learning-based methods. Traditional shape completion methods fill the unseen parts by assuming local surface or volumetric smoothness. For example, one can fill holes with surface primitives, such as planes or quadrics, or casts the problem as an energy minimization, e.g., Laplacian smoothing [26], [27]. Other researches leverage observed structures and regularities in 3D shapes, such as symmetries, to complete shapes [28], [29]. Another commonly used strategy is to search a most similar template shape from a database and align it with partial observations to achieve shape completion [30], [31]. This kind of methods works well for objects with a consistent topological structure, such as human faces [32], [33] and bodies [34], [35]. Nevertheless, all these methods deal with every single shape independently, thus cannot transfer the local structural pattern or prior knowledge learned from one shape to another.
In contrast, learning-based methods leverage deep neural networks to learn shape completion patterns under full supervision on synthetic datasets. These methods mostly adopt an encoderdecoder architecture to extract global feature representations from depth maps [36], RGB images [37], [38] discrete SDF voxels [39], or point clouds [40] and subsequently produce a full volumetric shape using the learned priors. Early efforts [41], [42], [42], [43] mostly use deep neural networks to predict discrete 3D shapes. For example, some early methods [41], [42], [43] explore generative models to generate 3D shapes. In [42], the author build a mesh variational auto-encoders (mesh VAE) to generate 3D mesh from the probabilistic latent space. [44] further enhances [44] by replacing fully connected layers with graph convolutional layers. More recent works [43], [45] explore deep neural networks to learn the implicit functions of 3D shapes in the continuous space. In [43], the author proposes to use the signed distance function to represent a 3D shape. A generative model is trained to produce a continuous field for 3D shape representation, which is characterized by the zero iso-surface decision boundaries of discretized SDF samples. A similar idea comes from Occupancy Networks [45] that uses the binary voxel occupancy to implicitly represent the 3D surface as the continuous decision boundary of discretized voxel occupancy.

Latent Space Optimization
Instead of using an explicit encoder network for feature learning, one can search the optimal latent representations by training a decoder only network. Tan et al. [46] propose probably the first work to use this idea that simultaneously optimizes the latent codes and the decoder network parameters through backpropagation. During inference, the decoder parameters are fixed as a prior, and a latent code is randomly initialized from a Gaussian distribution and optimized to reconstruct the new observation. Similar approaches have been explored in a large amount of works [47], [48], [49], [50], [51], for applications such as noise reduction, missing measurement completions, fault detection, meta-learning, and shape correspondence. For example, in [51], the author optimizes the latent code extracted from the encoder network to minimize the Chamfer distance between deformed shapes and target shapes, which provides a strong performance boost for shape correspondence. In [43], the authors propose to search the optimal latent representations for shape completion. In the paper, we call this type of encoder-free network as auto-decoders. In this paper, both our point set registration network and shape completion network are build upon auto-decoders.

METHOD
In this section, we introduce the proposed method for partial point set registration via Joint shape Completion and Registration using deep neural Network, called JCRNet. Our method contains two parts: a shape completion network to recover full shapes from corresponding partial observations, and a point set registration network that conducts point set registration from source shapes to target shapes. First, we define the partially registration problem in section 3.1. Then, it gives an overview of the proposed method in section 3.2. We introduce our shape registration network and shape completion network in section 3.4 and section 3.5 respectively. Finally, we introduce our optimization strategy in section 3.6.

Problem Statement
.., M }, our goal here is for each pair of point sets to estimate the rigid transformation from source point set X i to target point set Y i . Without loss of generality, the rigid transformation here is represented by a homogeneous transformation matrix, p ∈ SE(3), which is composed by a rotation matrix R ∈ SO(3) and a translation vector T ∈ R 3 , i.e., p = [R|T ].
To properly deal with partial observations, we incorporate a shape completion task along with the point set registration task. For this task, our model takes the partial point set S as input and outputs a recovered point set S to better approximate the full shape. In this paper, we design two separate deep neural networks to learn shape completion function g : S → S and the point set registration function f : X → Y.
To integrate these two tasks for joint optimization, we further proposed to incorporate a learnable latent code for each target shape, which can be regarded as a global encoding of the shape. Our method benefits from this design by two points: 1) first, we do not need to explicitly define a feature encoding network to learn the shape representation. Previous methods mostly rely on hand-crafted features or learning deep feature embeddings with deep neural networks. However, neither hand-crafted features nor a learning-based feature encoding network can guarantee the robustness towards partial observations. 2) both our shape completion network and point set registration network takes as input the shared latent code representation, it, therefore, enables the communication of these two tasks and enhances the registration performance. In this way, both our point set registration network and shape completion network benefit from the design of autodecoder architecture. During training, we optimize all latent codes along with the network parameters. While in the inference stage, we fix the network parameter and optimize these latent codes to get the optimal shape completion and registration results. Our method contains two parts: a shape completion network to recover full shapes from corresponding partial observations (upper part), and a point set registration network conduct point set registration from the recovered shapes (lower part). Both parts use an autodecoder network that takes as input a randomly initialized latent code of target shape and optimizes the latent code during network training. More specifically, given a pair of input point sets, source point sets X ∈ R N ×3 and target point set Y ∈ R N ×3 , our model incorporates a learnable latent code z ∈ R 256 for each target point set, which can be regarded the global representation of target point set. The shape completion network takes the latent code z of target shape Y as inputs and produces the recovered shape Y under the supervision of Y. In this paper, we use an auto-decoder network to reconstruct the partial target shape Y following [43] from a learnable latent code. The point set registration network takes the source point sets X and the latent code z as input and predicts the rigid transformation from source point set X to target point set Y. More specifically, we stack the coordinates of each point on shape X with the latent code z to formulate the inputs for our point set registration network. We use a fully connected network to regress the rigid transformation parameters. Our model achieves simultaneous shape completion and registration from partial observations. Fig. 2 gives an overview of the proposed method for partial point set registration. Our method contains two parts, and both parts use an auto-decoder architecture. In the first part (upper part in Fig.  2), our method performs shape completion for the target shape Y. It takes the latent code z of the target shape as input. Then we design a decoder network g β to recover the full point set with the partial observation as supervision. In the second part (lower part in Fig. 2), our model learns the geometric transformation from source point set X to the target point set Y. Specifically, we stack the coordinates of each point in source shape with latent vector z as the input, i.e., [z, x], for each x ∈ X , and feed them to another decoder network f θ to predict the rigid transformation. By applying the predicted geometric transformation to source shape, we then formulate an unsupervised alignment loss between the transformed source point set X T with the source point set Y. The shared latent codes are initialized from a Gaussian distribution. During training, the latent codes are optimized along with two decoder networks. While in the inference stage, we fix the network parameters and optimize the latent code for each pair of input shapes to produce the optimal geometric transformation.

Global Feature Representation
To characterize the global feature of a given point set, previous methods need to explicitly define a feature encoder network (i.e., PointNet [19]) for the deep spatial feature extraction. However, the design of an appropriate feature encoder for unstructured point clouds is challenging as standard discrete convolutions assume the availability of a structured input (e.g., 2D image). To avoid the hand-crafted network design for the extraction of spatial features, in this paper, we define a learnable latent code for each partial target point set with an intention to characterize the essence of its global feature. As shown in Fig. 2, the latent code is randomly initialized from a Gaussian distribution and is fed into two different decoder networks for the dual tasks of shape completion and registration, respectively.
By the design of this latent code representation, both our shape completion network and shape registration network takes as input the same global feature representation and is optimized along with the network parameters during training. In the inference stage, we fix the network parameters and only optimize the latent code for each pair of input shapes. In this way, the latent codes are enforced to characterize the global feature of a 'full shape' instead of a partial one.
In this paper, we use auto-decoder architecture for two reasons: 1) previous research [43] has shown that using auto-decoder architecture can lead to better performance compared with encoderdecoder architecture during SDF learning; 2) use auto-decoder architecture enables a joint optimization of our shape completion and point set registration tasks.

Point Set Registration
Our point set registration network f θ takes as input the source point set X i and the latent vector z i of the target point set Y i , and it outputs the rigid transformation from source point set X i to target point set Y i . The architecture of our point set registration network is illustrated in Fig. 3. For each point on shape X i , we stack its coordinates with the associating latent vector to formulate the input [z i , X i ] and feed it into transformation decoder network f θ to predict the desired transformation.
In this paper, our transformation decoder network firstly uses a multi-layer perceptron network to map the inputs to highdimensional feature embeddings, followed by a max-pooling layer to get the global transformation representation. Several fully connected layers are further used to map the global transformation representation to desired geometric transformation, i.e., three rotation angles r i = (rx, ry, rz) and a translation vector t i = (ta, tb, tc). [ The rotation matrix R i can then be formulated from the ration angles as, R i = Rz Ry Rx where denotes matrix multiplication.
The transformed source point set can be generated by, Generally, Chamfer distance can be used to measure the alignment between two point sets. In this paper, considering the target point set is a partial observation, we use the clipped Chamfer distance to measure the alignment loss between transformed source shape X T i and partial target shape Y i , i.e., we penalize the distance from each point on partial target shape Y i to the nearest point on transformed source shape X T i and also the distance from each point on transformed source shape X T i to its nearest point on target shape Y i . Our registration loss is calculated as follows, where σ t denotes the threshold for clipping the distance.
In the training stage, we simultaneous optimize registration decoder network parameters θ and the latent vector z i using all shape pairs in the training set. While in the inference stage, we fix the decoder network parameter and optimize the latent vector for each input pair of point set to produce the desired rigid transformation.

Shape Completion
In this paper, we design the shape completion network to recover the full shapes from partial observations. It enforces the latent code to represent the global representation for the full shapes instead of partial ones and thus also makes it easier for the dual task of point set registration.
Our shape completion network is designed using auto-decoder architecture following the method presented in DeepSDF [43]. To represent a shape S, we aim to learn a signed distance function (SDF) that, for any given spatial point p ij , it outputs the distance from this point to the closest point on the surface of the shape. S, SDF (S, p ij ) = s ij . Note that p ij does not need to be on shape S. The sign of SDF value indicates whether the point is inside (negative) or outside (positive) the input surface. We refer interested readers to [43] for more details.
More specifically, given each target shape Y i , a latent code z i is paired with the shape to represent its shape information. The SDF value at spatial location y ij can be generated via a deep feedforward network g β (·), see Eq. (8). Fig. 4 illustrates the pipeline of our shape completion network.
In the training phase, our model tries to minimize a pre-defined loss function over all training shapes with respect to all latent codes {z i } M i=1 and the network parameters θ: where M denotes the number of training shapes, K denotes the number of point samples for each shape. L(·, ·) denotes the loss function penalizing the predicted SDF value s ij and ground truth SDF value s ij . The second term is used to penalize the latent codes. σ is the hyperparameter to balance these two loss terms. We use the clamped L1 loss for L with a threshold of 0.03. The point sampling strategy will be explained in Section 4.2. At inference time, we fix the network parameters β, thus a shape code z i for each shape Y i can be estimated via Maximum-a-Posterior (MAP) estimation.

Optimization Strategy
We train our shape completion and point set registration network with the full shapes in training set. In the training stage, we simultaneously optimize the parameters of our shape completion network and point set registration network, as well as the latent codes for all shapes. The optimal network parameters can be generated follows where the first term denotes point set registration loss and the second term denotes shape completion loss, and λ denotes the hyper-parameters to balance these two tasks. Without specific mention, we set λ to 0.1 in the following sections. After training on full shapes from the training set, the learned decoder network parameters θ and β here provides a prior knowledge for the optimization of latent code representation.
During model evaluation, given a pair of partial observation with source point set X i and target point set Y i , we fix the two decoder networks and only optimize the latent code of partial target shape Y i . To achieve this, we randomly initialize a latent code z i for the target point set and optimize it via Maximum-a-Posterior (MAP) estimation as: After the optimization process, the optimal latent code can be used to perform shape completion for the target shape as well as predict the desired geometric transformation from source point set to target point set simultaneously using equation (1) and equation (4). One may note that it would be possible to split our model into a two-step process: 1) optimize latent code z i to get optimal target shape reconstruction, 2) optimize latent code z i using the Chamfer distance between the reconstructed target shape Y i and transformed source shape X T . However, in our experiments (in section 4.7), we find that this two-step process does not lead to better performance compared to our model using end-to-end training.

Experimental Dataset
We evaluate the proposed JCRNet for partially registration on ModelNet40 [52] benchmark dataset. ModelNet40 dataset contains 12,311 shapes of 40 object categories. We split this dataset into 9,843 for training and 2,468 for testing. We train our model using full shapes from the training set and evaluate the registration performance with the partially shapes from the test set. To generate the data for training and model evaluation, we randomly sample 1024 points on each shape. Then, for each source shape X we generate the transformed shapes Y by apply a rigid transformation: the rotation matrix is characterized by three rotation angles along the xyz axes, where each value is uniformly sampled from [0, 45] unit degree, and the translation is uniformly sampled from [−0.5, 0.5]. At last, we simulate partial point sets of Y by randomly select a point in unit space and keep its 768 nearest neighbors.

Implementation details
For the point set registration network, we use the sequential structure of C(256)-C(128)-M-FC(128)-FC(64)-FC(3), where C denotes 1D convolution layer with a kernel size of 1, FC denotes fully-connected layer, and M denotes 1D max-pooling. To formulate a rigid transformation, our point set registration network predicts a 3-dimensional output for rotation angles and translation vector.
For the shape completion network, we leverage the same architecture as [43] and use a 7-layer MLP to decode the latent code into SDF values, each layer with a size of 512. To train the completion network, we need to construct numerous of SDF samples for each target shape. Each SDF sample include a 3D point and its SDF value. Specifically, we perturb each point on the target shape with zero-mean Gaussian noise with standard deviation of 0.2 to generate three spatial samples per point. For each point sample, we find the nearest point from the original target shape and calculate the distance as its SDF values.
Our model is optimized using Adam optimizer with an initial learning rate of 1e-3. We set the batch size to 50, momentum to 0.9, and weight decay to 1e-5. We use batch normalization and dropout for all fully connected layers, except for the output layer, both in our shape completion network and in point set registration network. The latent vectors {z i } are randomly initialized from a Gaussian distribution N (0, 0.06) with a dimension of 256. Our method is implemented with the PyTorch library on a TESLA K80 GPU.
We evaluate the registration performance by the metrics, including mean squared error (MSE), root mean squared error (RMSE), and mean absolute error (MAE). Angular measurements are in units of degrees. To demonstrate the effectiveness of our proposed JCRNet, we compare our model to Closest Point (DCP) [1] and PR-Net [5]. We omit other comparing methods because DCP and PR-Net have demonstrated state-of-the-art point set registration performance for full observations and partial observations respectively.

Transformation Process
Before going to the experimental results, we examine the alignment process of three test pairs by the optimization steps. As shown in Fig. 5, our proposed model is able to align the main parts of the input shapes after 100 optimization steps. The rotation predictions are close to the ground truth ones, while it still takes thousands of optimization steps for our model to get the accurate translation predictions. After 5000 steps, our model produces satisfying alignment for all input shape pairs.

Results on ModelNet40
We first evaluate the performance of our JCRNet on the Mod-elNet40 dataset. Table 1 listed the performance of our method and the comparing methods, including DCP and PR-Net on the airplane, bench, chair categories of the ModelNet40 dataset. As shown in Table 1, our model gets an order of magnitude better performance than the comparing methods for the rotation prediction and also achieves significantly better performance for translation prediction. Specifically, for the rotation prediction, our JCRNet achieves an RMSE(R) less than 1 degree for all four classes. Note that both DCP and PR-Net solve the geometric transformation based on dense point correspondences between the transformed source shape and the target shapes. Nevertheless,  it's still a non-trivial to find the dense point correspondences using point-wise features, especially for partial observations. In contrast, our JCRNet model does not use point correspondence information but directly regress the geometric transformation using fully connected layers. This makes our model can be trained in a purely unsupervised way and also makes it more flexible to be equipped in different network architectures. Some randomly selected examples are shown in Fig 6. As shown in Fig 6, our model is able to align partial inputs with different orientations on all four categories. We do not include the results of other categories here because our model is trained independently for each category. It would take much time to conduct experiments on all 40 categories of the ModelNet40 dataset.

Resilience to Gaussian Noise
We further test the robustness of our model under Gaussian noise. To do this, we add noise to each of the points on both source and target point sets, which was randomly sampled from N (0, 0.01) and clipped to [−0.05, 0.05]. Other settings are same as section 4.2. We trained our JCRNet model and comparing methods, including DCP and PR-Net, on noise-free data and evaluate the performance of all methods on the test set with Gaussian noise. Evaluation results are listed in Table 2. As shown in this table, our model achieves significantly better performance than the comparing methods for both rotation and translation prediction. By comparing Table 1 and Table 2, one can find that on the sofa category, the performance of the DCP model degrades a lot, with RMSE(R) increase from 3.266247 to 3.864215. The performance of our model and PR-Net [5] decreases much smaller, with RMSE(R) increases from 2.419480 and 0.408753 to 2.497723 and 0.484324, respectively. This demonstrates that comparing to the DCP model, our model and PR-Net are more robust to data noise for the task of partial point set registration. Moreover, compared with PR-Net, our JCRNet model gets a smaller performance drop. Fig. 7 shows some selected examples of our model on airplane and chair categories. As shown in Figure 7, even when the input point sets are with Gaussian noise, our model can still successfully align the source and target point sets.

Number of Missing Points
In this section, we explore the performance of our model with different numbers of missing points. We conduct experiments on the airplane and chair categories with 256, 384, and 512 missing points, i.e., each partial shape has 768, 640, and 512 remaining points. Table 3 shows the quantitative results of our model. As can be seen in Table 3, with the increase in the number of missing points, the performance of our model decreases as expected. For the partial shapes with 640 remaining points, our model gets an RMSE(R) around 1 degree and RMSE(t) less than 0.06. Even with 512 missing points, our model can still maintain an RMSE(R) less than 5 degrees for both airplane and chair categories. We also notice that the MSE(R) becomes large when the input shapes have 512 missing points. In our experiments, we notice that our model fails to align several cases, and the rotation predictions are with large errors. This is because when input shapes are with a large number of missing points, our model may converge to local minimal, considering the latent codes {z i } are randomly initialized from Gaussian distribution. These failure predictions dominate the prediction errors, especially for MSE(R). Figure 8 shows some selected examples of the registration results of our model on airplane and chair categories with 640 and 512 remaining points. As shown in Fig. 8, our model can produce satisfying alignment for most of the cases. The cases in the blue boxes show failure cases. For these failure cases, our model predicts the rotation matrix with large errors. In our experiments, we find our model fails for only a few cases for each object category.

Ablation Analysis
In this section, we conduct further experiments to demonstrate the effectiveness of our shape completion module. For this, we remove the shape completion part from our method, i.e., set λ to 0. In this way, our method directly performs point set registration for the partial point sets. Table 4 lists the performance of our model with and without shape completion module, as well as the comparing methods, i.e., DCP [1] and PR-Net [5]. As can be seen in the table, both our model and our baseline model achieve significantly better performance than the comparing methods. More importantly, our model enjoys a significant performance boost by the joint training of the shape completion network, especially for rotation prediction.   TABLE 4 Test performance on airplane category of ModelNet40 dataset with and without shape completion for partial observation. The baseline is our model without the shape completion network.

CONCLUSION
In this paper, we introduce an unsupervised method for partial point set registration. Given the fact that recent learning-based methods have achieved significantly better registration performance on the full shapes than partial observations, we propose to bridge the performance gaps by incorporating a shape completion network to recover full shapes from partial observations. To achieve this, we design a latent code for each of the target shape, which can be regarded as the global feature encoding the target shape. This latent code is initialized from a Gaussian distribution and is taken as the inputs for both shape completion and registration networks. In this way, our model eliminates the explicit design of point feature encoding network and, more importantly, enables the joint training of both shape completion and registration networks and therefore boosts the performance of our registration network. During training, the latent code is optimized along with the parameters of shape completion and registration networks.
While in the inference stage, the network parameters are fixed and used as prior knowledge to guide the optimization of the latent codes to get the optimal shape completion and registration results. Experiments on the ModelNet40 dataset demonstrate the effectiveness of our model for rigid transformation prediction on partial observations. Results also show that our model is robust to inputs with Gaussian noise.

ACKNOWLEDGEMENT
This work was supported by the NYU Abu Dhabi Institute (AD131).