## Learning Local Feature Descriptor with Motion Attribute For Vision-based Localization

Yafei Song 12* , Di Zhu 1 , Jia Li 3 , Yonghong Tian 2 and Mingyang Li 1

Abstract -In recent years, camera-based localization has been widely used for robotic applications, and most proposed algorithms rely on local features extracted from recorded images. For better performance, the features used for openloop localization are required to be short-term globally static, and the ones used for re-localization or loop closure detection need to be long-term static. Therefore, the motion attribute of a local feature point could be exploited to improve localization performance, e.g. , the feature points extracted from moving persons or vehicles can be excluded from these systems due to their unsteadiness. In this paper, we design a fully convolutional network (FCN), named MD-Net, to perform motion attribute estimation and feature description simultaneously. MD-Net has a shared backbone network to extract features from the input image and two network branches to complete each sub-task. With MD-Net, we can obtain the motion attribute while avoiding increasing much more computation. Experimental results demonstrate that the proposed method can learn distinct local feature descriptor along with motion attribute only using an FCN, by outperforming competing methods by a wide margin. We also show that the proposed algorithm can be integrated into a vision-based localization algorithm to improve estimation accuracy significantly.

## I. INTRODUCTION

Image local features are important for many tasks and applications, e.g. , image-based localization [1], [2], [3], [4], structure from motion [5], [6] and simultaneous localization and mapping [7], [8], [9]. Previous researchers have designed various and powerful local feature detectors and descriptors, e.g. , SIFT [10], ORB [11] and FREAK [12]. Due to the recent success of deep neural networks on various tasks [13], [14], several previous papers have attempted to learn robust image local features automatically [15], [16], [17], [18], [19].

For a vision-based localization system, the motion attribute of each feature point could be exploited to improve the performance, e.g. , as demonstrated in Fig. 1(a), the points on moving people should be excluded. Existing systems usually perform this step by eliminating the point matches via epipolar geometry constraints [20]. These methods can successfully remove points corresponding to fast-moving objects but have difficulties to block points corresponding

1 Yafei Song, Di Zhu and Mingyang Li are with the AI Labs, Alibaba Group, Hangzhou 311121, China. { huaizhang.syf, david.zd, mingyangli @alibaba-inc.com }

2 Yafei Song and Yonghong Tian are with the National Engineering Laboratory for Video Technology, School of Electronics Engineering and Computer Science, Peking University, Beijing 100871, China. { songyf, yhtian @pku.edu.cn }

3 Jia Li is with the the National Key Laboratory of Virtual Reality Technology and System, School of Computer Science and Engineering, Beihang University, Beijing 100191, China. { jiali @buaa.edu.cn } * Corresponding author.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01180v2-figure-aeea5625-8841-4032-9f79-7edaa9413ee2.jpg)

points will move in a short time

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01180v2-figure-7bb51faa-12b9-4a22-86ee-ced8d611e2d1.jpg)

(b) points will move in one or several days

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01180v2-figure-4c6ff308-ff4f-4f63-a281-8c91c031e952.jpg)

Fig. 1. The moving points are ubiquitous in the images, including short term moving points such as in (a) and long term moving points such as in (b). All these points will decrease the performance of a localization system.

to slow-moving objects. In addition, this strategy will fail on long-term moving points, e.g. , the points on the parked vehicles, as shown in Fig. 1(b). Such points will be static in a short time but will move to another place in a few minutes or even days. This will inevitably lead to reduced performance for a couple of localization operations (e.g., loop closure detection or re-localization[8], [21], [22]). Beyond that, there are also some points in unstable areas, such as the sky. In this paper, we focus on selecting the long term static points for localization systems by estimating the motion attribute of each point.

Some previous methods have also exploited the motion attribute to perform localization or mapping. Kaneko et. al. [23] first segmented the input image and then only used the points with long term static motion attribute derived from the semantic label. However, semantic segmentation is a difficult task, which will dramatically increase the computational cost. In this paper, we transform the task to estimate the motion attribute for each feature point, which is easier than semantic segmentation but is sufficient for vision based localization. Naseer et. al. [24] exploited geometrically robust regions to determine the position of the input image, which is a different task from localization.

Motion attribute estimation is generally considered as a highly computational costly task for a robotic platform, and thus efficiency is of significant importance for algorithm design. Inspired by the recent progress on local feature learning [16], [17], [18], [19], we try to use a single network to perform the descriptor calculation and the motion attribute estimation tasks simultaneously, since these tasks can be formulated into similar computation processes. A lot of existing methods, e.g. , [16], [17], [18], focused on extracting distinct features from an image patch. However, it is difficult to estimate the motion attribute only from a local patch since the context is essential. Moreover, patch-based algorithms usually have poorer computational efficiency than whole image-based algorithms due to the repetitive computation between different patches, e.g. , [25] and [13].

With the above analysis in mind, we design a fully convolutional network, named MD-Net, which takes the whole image as input and performs motion attribute estimation and feature description simultaneously. MD-Net has a shared backbone network, named N B, and two network branches, named N M and N D. The heavy backbone network N B takes the whole image as input and outputs shared features to each following network branch. The motion attribute estimation branch N M then assigns each point as unstable, moving or static. Finally, the feature description branch N D extracts distinctive descriptor for each point.

The most similar methods with this paper are from DeTone et. al. , i.e. , [19] and [26]. In [19], DeTone et. al. have designed an FCN [14] based single network to perform local feature localization and description simultaneously. However, the localized points are not robust enough due to the poor localization ability of the standard convolutional neural network (CNNs) [27]. Therefore, we resort to handcrafted detectors in this paper. Moreover, the strategies to learn descriptor are also different. In [26], the model also estimates whether a point is steady. However, this work can only handle short term moving points as shown in Fig. 1(a), but not long term moving points as shown in Fig. 1(b). We also experimentally show that, when integrated into visionbased localization, the proposed algorithm is able to achieve significantly better accuracy, compared to [26].

The contributions of this paper mainly lie in four aspects:

- 1) We design an FCN to estimate the motion attribute of each local feature point to distinguish static points from moving or unstable points.
- 2) We further enhance the FCN to calculate the descriptor of each point simultaneously, which is more efficient than patch-based methods.
- 3) We integrate the proposed local feature processing pipeline into a vision-based localization method to seek accuracy gain.
- 4) Experimental results demonstrate that both the proposed local feature method and integrated localization algorithm outperform competing state-of-the-art algorithms by a wide margin.

## II. RELATED WORK

A local feature algorithm usually can be divided into two steps: feature localization and feature descriptor calculation. In this section, we briefly introduce some well-known local features, including hand-crafted and learning-based, from these two aspects.

Hand-crafted local features . Over the last few decades, researchers have designed various algorithms to localize each robust point as well as calculate its distinct descriptor. At the very beginning, many methods devoted to detecting and localizing corners in the image, e.g. , Harris corner [28], FAST [29]. Besides corner detection methods, researchers also introduced the scale-space theory and detected the extrema as feature points, e.g. , Laplacian of Gaussian (LoG) [30], difference of Gaussians (DoG) [10] and MSER [31]. Researchers have also designed various local feature descriptors, e.g. , SIFT [10], SURF [32], and HOG [33]. As robotics tasks usually need real-time algorithms, some fast binary descriptors also have been proposed, e.g. , BRIEF [34], FREAK [12], ORB [11]. These local feature detectors and descriptors have been successfully applied in many tasks. However, with the successful applications of deep learning based methods on various tasks, researchers have attempted to learn more robust local features from the data automatically.

Learning-based local features . Learning based methods can be divided into three classes. The first class methods only learn robust detectors [15], [35], [36]. One challenge is how to generate ground truth. To this end, Verdie et. al. [15] applied DoG algorithm on all images of a unique scene and selected the points which can be detected in most cases. This strategy can enable the learned detector to outperform the baseline detector. Savinov et. al. [35] formulated and solved the problem in an unsupervised manner. Zhang et. al. [36] focused on learning a discriminative and transformation covariant detector. These detectors cannot remove the points on moving objects.

The second class methods aim at learning robust descriptors. To this end, several large-scale datasets [37], [38], [39] are collected for training and evaluation. Tian et. al. [17] designed a convolutional neural network named L2-Net to extract descriptors from a local patch. Mishchuk et. al. [18] further improved the performance with a large margin via minimizing the hard negative samples. Luo et. al. [40] exploited geometric constraints to learn more robust descriptors. However, these methods calculate descriptor for each image patch, which may be not efficient than FCN based algorithms.

Yi et. al. [16] and Ono et. al. [41] divided the whole process into several steps as previous algorithms and performed each step using a neural network. These methods can take full advantage of previous knowledge on the problem but not efficient than FCN based methods. DeTone et. al. [19] designed an FCN based single network to perform local feature localization and description simultaneously. However, these methods are not designed for localization and cannot remove moving points. To this end, DeTone et. al. [26] further estimated whether a point is steady and only selected the static points. However, this work can only handle short term moving points but not long term moving points.

Fig. 2. The training process for our MD-Net. We feed a mini-batch of training samples into the backbone network. Subsequently, the extracted features are feed into two branches simultaneously. For motion attribute, we use cross-entropy loss (3), and for descriptor, we use mean square error loss (4). Note that, Conv denotes a convolutional layer, ConvB denotes that it is followed by a batch normalization layer and a ReLU layer, K denotes kernel size, P denotes padding size, S denotes stride size, and F denotes the number of output feature maps.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01180v2-figure-37a96530-874e-4c2e-9417-9d14467f6771.jpg)

## III. LEARNING DESCRIPTOR WITH MOTION ATTRIBUTE

In this section, we first briefly overview the whole method and detailedly introduce each module subsequently. As demonstrated in Fig. 2, we design an FCN named MD-Net to estimate motion attribute and calculate descriptor for each point in the input grayscale image. The MD-Net consists of a heavy backbone network named N B to extract shared features from the input image, a light branch N M to estimate motion attribute, and another light branch N D to calculate descriptors. For a fair comparison, the backbone network N B has the same structure with the backbone network in [19], which consists of 8 convolutional layers and 3 maxpooling layers. Each convolutional layer is followed up with a batch normalization layer [42] and a ReLU activation layer [43]. The hyper-parameters of each layer also can be found in Fig. 2(b). The backbone network N B outputs a group of feature maps to the following branches. This structure has high computational efficiency benefiting from the fully convolutional structure and the down-sampling pooling layer.

## A. Motion Attribute Estimation

A local feature detector usually localizes a large number of points in an image. However, not all of these points are suitable for localization or mapping. In this paper, we divide the motion attribute into three classes, i.e. , A = {A u , A m , A } s , where A u denotes unstable, A m denotes moving, A s denotes static. The unstable points are on the image regions, which will vary fast, such as the cloud in the sky. The moving points are on the moving objects, including short term and long term. The static points are steady for a long time. A localization system usually can automatically discard short-term fast-moving points as these points could not meet the epipolar constraint. However, when it comes to very-slowly-moving points, it becomes more challenging. Moreover, the long-term moving points can not be removed, which will result in decreased performance, even with a selfimproving method [26]. When localizing an input image, all of the unstable and moving points will decrease the performance. To alleviate this phenomenon, we propose to assign a point's motion attribute according to its semantic label. As previous works have constructed many semantic segmentation datasets, e.g. , Cityscapes [44], we only need to perform a label transforming process.

To estimate the motion attribute, we add a network branch N M following the backbone network, as shown in Fig. 2, which consists of two convolutional layers, one batch normalization layer, one ReLU layer and one softmax layer. This branch can transform the input features of each point to the probability of each motion attribute. The hyperparameters of this branch can be found in Fig. 2(c). To train this branch, we adopt the cross-entropy loss

$$\mathcal { L } _ { \text{M} } = - \sum _ { i, j } y _ { i j } \log ( p _ { i j } ),$$

where y ij = 1 only if the motion attribute of point i is A j , otherwise y ij = 0 , p ij is the predicted probability.

The cross-entropy loss can effectively supervise a network when the distribution of sample sizes of all classes is uniform. Otherwise, the model will incline to the class with large size [45]. This unbalance bias is ubiquitous in the training data. To this end, we re-weight the loss of each class according to its size as

$$w _ { j } = \frac { \frac { 1 } { N _ { j } } } { \sum _ { j } \frac { 1 } { N _ { j } } },$$

where N j is the sample size of class A j . Then the loss function (1) can be transformed to its re-weighted version as

$$\mathcal { L } _ { \text{M} } = - \sum _ { i, j } w _ { j } y _ { i j } \log ( p _ { i j } ).$$

With the re-weighting strategy, each class would contribute equally to the loss function to avoid the influence of unbalance classes distribution.

## B. Descriptor Calculation

To learn distinct descriptors from the data automatically, previous methods have constructed several elaborate datasets, e.g. , [37], [38], [39]. As these datasets only have image patches, most methods also based on patches [17], [18], [40]. For deep learning algorithms, however, image-based models usually are more efficient than patch-based models. If the network structure is unique, the computational cost has a linear relationship with the size of the input image. Previous methods typically take a 32 × 32 patch as input. Its computation cost will be equal to a whole image-based model if the input image is 320 × 320 and the detector only detects 100 points. In practical, there are often more than 300 points in an image. In other words, a patch-based model usually need several times of computational cost compared with the corresponding whole image-based model.

In this paper, we design an FCN network to calculate feature descriptor from the whole input image as in Fig. 2(a)(b)(d). However, it is a challenge to design an effective training strategy. DeTone et. al. [19] directly used all points in the image. However, this strategy will introduce plenty of noisy samples. To take full advantage of existing datasets, we resort to a teacher-student framework, which is usually exploited for feature embedding [46]. Without loss of generality, we transform the HardNet model [18] to an image-based model and take it as the teacher model, which can be used to supervise the descriptor learning. As shown in Fig. 2(e), we can see the detailed structure of the modified HardNet. Compared with the initial version, we only remove the last reshape layer and add padding on the last convolutional layer. This network surgery has little influence on the effectiveness of the model.

To train the descriptor branch N D, which is the student model, we minimize the mean square error between the outputs of N D and HardNet. The loss function can be defined as

$$\mathcal { L } _ { \text{D} } = \sum _ { i, k } \frac { 1 } { K } ( d _ { i k } - \hat { d } _ { i k } ) ^ { 2 }, \quad \ \ ( 4 ) \quad \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \$$

where K = 128 is the dimension of one output descriptor, d ik is the k -th dimension feature at point i outputted by N D, ˆ d ik is the counterpart outputted by HardNet.

## C. Multi-task Learning

As our model complete two tasks simultaneously, it is a typical multi-task learning problem to train the model. Under deep learning framework, it is easy to perform this process. We can simply combine the two loss functions as

$$\mathcal { L } = \lambda _ { \text{M} } \mathcal { L } _ { \text{M} } + \lambda _ { \text{D} } \mathcal { L } _ { \text{D} }, \quad \ \ ( 5 ) \quad \text{cam}.$$

where λ M and λ D are the parameters to adjust the weight of each loss. In our experiment, we set λ M = 1 , λ D = 1 empirically. With the multi-task learning strategy, we can simultaneously optimize the backbone network and two taskspecific branches. One problem for multi-task learning is that the tasks may conflict with each other, which lead to an unsatisfied performance on some tasks. In this paper, this

Fig. 3. The process to integrate our model with a local feature detector.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01180v2-figure-ef13878d-ee1b-417b-a446-d7a7abc3acb3.jpg)

phenomenon has not appeared, which indicates that it is reasonable to integrate these two tasks into one model to save computational cost.

The complete training process can be summarized as follows. As demonstrated in Fig. 2, we first generate a minibatch training sample from the training set and feed them into our model. Then our model outputs the motion attribute probabilities and descriptors. For the motion attribute, the ground truth is generated from semantic segmentation annotations. For descriptor, we take the output of HardNet as the supervision signal. The model can be optimized via minimizing the loss function (5) using Adam optimization algorithm. We set the batch size as 16 , set the initial learning rate as l 0 = 0 01 . and gradually reduce it after the e -th epoch as

$$l _ { e } = l _ { 0 } \times b ^ { \frac { e } { E } },$$

where E = 100 is the total number of epochs, b = 0 01 . is the factor to control the rate of decay. To avoid over-fitting, we also set the weight decay as 1 e -6 in all experiments.

## IV. ASSOCIATION WITH LOCALIZATION SYSTEM

We integrate our model with a localization system to verify its effectiveness. As illustrated in Fig. 3, we combine our model with a local feature detector. The detected feature points are filtered according to its motion attribute. Only static points are reserved for the following steps and the other points are discarded. Subsequently, the descriptor of each reserved point can be obtained from the outputs of the branch N D. Note that, as our model down-samples the resolution, we up-sample the results using bilinear interpolation.

The localization algorithm used to test the proposed local feature pipeline is a sliding-window based visual-inertial SLAM method. Visual-inertial SLAM is widely used recently by utilizing the complementary properties of both cameras and inertial measurement units (IMU) to greatly enhance the localization performance [3], [9], [47], [22], [48]. The exact implementation follows that of [22], with the following changes. Firstly and most importantly, the proposed local feature pipeline is used instead of FREAK in [22]. Secondly, we do not use odometry measurements during our tests. As a result, the cost function does not contain the odometry terms. Due to the same reason, the poses are expressed with respect to IMU frame, and IMU to

TABLE I THE CORRESPONDENCE BETWEEN MOTION ATTRIBUTE AND SEMANTIC

CLASS IN CITYSCAPES DATASET.

| Motion attribute   | Semantic class                                                                                  |
|--------------------|-------------------------------------------------------------------------------------------------|
| unstable           | sky, vegetation, terrain                                                                        |
| moving             | human, vehicle, static, dynamic, traffic light                                                  |
| static             | ground, flat, building, wall, fence, guard rail, bridge, tunnel, pole, pole group, traffic sign |

TABLE II IOU OF MOTION ATTRIBUTE ON THE CITYSCAPES VALIDATION DATASET.

|                      | unstable   | moving   | static   | mean   |
|----------------------|------------|----------|----------|--------|
| Our w/o re-weighting | 62.9       | 74.6     | 85.3     | 74.3   |
| Our                  | 68.3       | 74.5     | 85.7     | 76.2   |
| Proportion           | 2.7%       | 35.8%    | 61.5%    | -      |

odometry extrinsic parameters are ignored. Lastly, the pose integration and keyframe selection policy are purely based on IMU integration.

## V. EXPERIMENTS AND COMPARISONS

In this section, we first introduce experimental details to train our MD-Net and evaluate our motion attribute estimation results, our descriptors, and localization with our model in turn.

## A. Motion Attribute Estimation Results

To train our MD-Net, we use the Cityscapes dataset [44], which is with finely annotated semantic segmentation ground truth. This dataset is collected from various urban street scenes and consists of a training set with 2993 images and a validation set with 503 images. Before using the annotations, we transform the initial semantic classes into 3 motion attribute as in Tab. I. The final model is obtained after trained on the training set. We implement our training process using PyTorch [49] on a PC with an NVIDIA 1080 Ti GPU.

As shown in the third row of Tab. II, we can see that our model can accurately predict the motion attribute with a mean IoU as 76 2 . . To verify the effectiveness of the classreweighting strategy, we perform an ablation experiment by removing this strategy whose results are shown in the second row. The proportion of each class is also presented in the fourth row. We can see that this strategy can markedly improve the performance of the class with a small size, i.e. 5 4% . on unstable class, and increases the mean IoU about 2% . As shown in the top two rows of Fig. 4, our model also achieves good visual results. Moreover, to verify the generalization of our model, we also applied our model on some data from our work park. The results are shown in the bottom 2 rows of Fig. 4, which show that our model can well generalize to other data.

TABLE III

INLIER RATIO ON THE DATA FROM 3 DIFFERENT SCENES.

| Detector   | Descriptor   |   Scene1 |   Scene2 |   Scene3 |   Mean |
|------------|--------------|----------|----------|----------|--------|
| FAST       | FREAK        |     87.6 |     87.1 |     84.1 |   86.2 |
| FAST       | SuperPoint   |     88.8 |     88.4 |     90.2 |   89.1 |
| FAST       | Our          |     91.2 |     93.3 |     92.2 |   92.2 |

## B. Performance of the Descriptor

To evaluate our descriptors, we use the HPatches to perform patch verification, matching and retrieval tasks on the FULL split. The details about the tasks and evaluation protocols can be found in [38]. The quantitative results are demonstrated in Fig. 5. For a fair comparison, we direct use the pre-computed results of SIFT [10], RootSIFT, BRIEF [34], ORB [11], and DDESC [50] released by HPatches. The results of SuperPoint [19] and HardNet [18] are calculated using the released models by the authors. We can see that our model can successfully mimic HardNet and achieve comparable results on verification and matching tasks compared with previous FCN based method SuperPoint, and outperforms it on the retrieval task with a large margin. The results also indicate that HardNet, a patch-based method, can obtain better performance than FCN based methods, which would be a future work of FCN based methods.

## C. Performance in Vision based Localization

In this section, we present the results of vision-based localization using the proposed features from MD-Net. In our experiments, the sensor suite consists of a MYNT camera and a BOSCH BMI160 IMU. The camera captured images at 10Hz, and the IMU measurements were provided at 200Hz. The temporal, intrinsic, and extrinsic parameters of those sensors were calibrated offline using the method in [2]. During the data collection process, the sensor suite was mounted on ground vehicles (robots or cars).

The first experiment is to evaluate the inlier feature ratio via two-view based RANSAC algorithm at three different testing environments, i.e. , normal scene, with pedestrians, and with a couple of slowly moving vehicles. Tab. III shows the RANSAC inlier ratio for different local feature algorithms, which is computed by averaging all image pairs used in localization. For the compared local feature algorithms, the same detector was used for focusing on the descriptor comparison. Results in Tab. III demonstrate that the proposed algorithm is able to reach the best RANSAC inlier ratio, meaning that features corresponding to moving and unstable objects can be pre-removed. The proposed algorithm also outperforms SuperPoint in all cases.

The second and the third experiments are to show the performance of the proposed visual localization algorithm using our local feature method, in two different environments. In the second test, we collected a dataset around a local restaurant by a ground robot, whose trajectory started and stopped at the exactly the same location. This allows us to compute the 'final drift' as an error metric. Tab. IV

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01180v2-figure-1bd37339-0475-46b5-90ec-17b0e750dd3f.jpg)

Fig. 4. Motion attribute estimation results. Top two rows are results on Cityscapes dataset. Bottom two rows are results on the data collected in Alibaba campus, which are not in the training set.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01180v2-figure-0e8619e1-e958-4bf3-91f0-076d2149f565.jpg)

Fig. 5. The performance of 8 different descriptors on HPatches.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01180v2-figure-fcd182f8-3509-4d7a-919d-110a715e6a24.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01180v2-figure-cf8a378a-4878-4639-86a6-d387be70d679.jpg)

Fig. 6. Localization trajectory and drift.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01180v2-figure-2112f1fb-d308-4a90-84df-5ad011fa5a12.jpg)

and Fig. 6 show the final drifts by using different feature detectors and descriptors but under the same localization method. A couple of observations can be made from the result. Firstly, by explicitly modeling motion attribute, the localization error is reduced compared to the alternative method without the attribute. This is consistent with our claim that motion attribute is able to filter out more 'bad' features. Additionally, we note that our proposed method

TABLE IV LOCALIZATION DRIFT.

| Detector   | Descriptor        | Drift (x, y) [m]   |   Drift [m] |
|------------|-------------------|--------------------|-------------|
| FAST       | FREAK             | (-1.9, -0.4)       |        1.94 |
| FAST       | SuperPoint        | (-5.4, -1.8)       |        5.69 |
| SuperPoint | SuperPoint        | (4.4, 1.8)         |        4.75 |
| FAST       | Our w/o mot. att. | (0.8, 1.1)         |        1.36 |
| FAST       | Our               | (-1.2, 0.3)        |        1.23 |

achieves the best precision, compared to all other methods, including SuperPoint families and classic FAST/FREAK combination. These results show that the proposed method is the best one for performing vision-based localization, at least among the experiments we conducted.

In the last experiment, we collected two series of urban street view datasets by mounting our sensors along with RTK-GPS on top of a car. The representative images captured during the data collection process are shown in Fig. 7. The readings from RTK-GPS are taken as the ground-truth

TABLE V

THE ROOT-MEAN-SQUARED-ERROR[M] BETWEEN THE RESULT AND THE GROUND-TRUTH TRAJECTORIES.

| Detector      | Descriptor        |   On series1 | On series2   |
|---------------|-------------------|--------------|--------------|
| FAST          | FREAK             |         2.87 | 27.06        |
| FAST          | SuperPoint        |        10.02 | Failed       |
| SuperPoint    | SuperPoint        |         7.96 | Failed       |
| FAST          | Our w/o mot. att. |         7.82 | 20.86        |
| FAST          | Our               |         1.1  | 7.25         |
| Series length | Series length     |       279.1  | 880.8        |

Fig. 7. Representative images captured during the data collection of the urban street view datasets.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.01180v2-figure-766be9a8-e41e-4102-9534-70c7e73052c0.jpg)

pose. For quantitative evaluation, the computed trajectory is aligned to the ground-truth using Umeyama's method [51]. The root-mean-square errors of each method can be found in Tab. V. We can see that our method obtains the lowest error among several comparison methods. We also note that the dataset is extremely challenging, where some images are full of or dominated by moving vehicles (see Fig. 7). However, by filtering out improper features by motion attribute, the proposed method can still achieve highprecision localization.

## VI. DISCUSSION AND CONCLUSION

In this paper, we design a fully convolutional network MD-Net to perform motion attribute estimation and feature description simultaneously. MD-Net has three modules: the backbone network N B to extract shared features from the input image, the motion attribute branch N M to estimate motion attribute, the description branch N D to extract distinctive descriptor. We further integrate the proposed method into a visual-inertial localization system to perform highprecision pose estimation. Experimental results demonstrate that the proposed method can improve the performance and outperforms previous similar algorithms, especially in complicated dynamic environments when multiple moving objects exist. The limitation of this paper is that the proposed method still relies on a third-party feature detector, which will be integrated into our FCN model in the future to enhance the performance further.

## ACKNOWLEDGMENT

We would like to thank Dongsheng Hong for his invaluable help. This work was supported by grants from the National Key R&amp;D Program of China (2017YFB1002400), the National Natural Science Foundation of China (61672072, U1611461 and 61825101), Beijing Nova Program (Z181100006218063), and China Postdoctoral Science Foundation (2018M641110).

## REFERENCES

- [1] Y. Song, X. Chen, X. Wang, Y. Zhang, and J. Li, '6-DOF Image Localization from Massive Geo-tagged Reference Images,' IEEE Transactions on Multimedia , vol. 18, no. 8, pp. 1542-1554, 2016.
- [2] M. Li, H. Yu, X. Zheng, and A. I. Mourikis, 'High-fidelity sensor modeling and self-calibration in vision-aided inertial navigation,' in ICRA , May 2014, pp. 409-416.
- [3] M. Li and A. I. Mourikis, 'High-precision, consistent ekf-based visualinertial odometry,' IJRR , vol. 32, no. 6, pp. 690-711, 2013.
- [4] A. Concha and J. Civera, 'Dpptam: Dense piecewise planar tracking and mapping from a monocular sequence,' in IEEE/RSJ International Conference on Intelligent Robots and Systems , 2015, pp. 5686-5693.
- [5] R. Huang, D. Zou, R. Vaughan, and P. Tan, 'Active Image-Based Modeling with a Toy Drone,' in ICRA , 2018.
- [6] S. Zhu, R. Zhang, L. Zhou, T. Shen, T. Fang, P. Tan, and L. Quan, 'Very Large-Scale Global SfM by Distributed Motion Averaging,' in CVPR , 2018.
- [7] R. Mur-Artal and J. D. Tard´ os, 'ORB-SLAM2: An Open-Source SLAM System for Monocular, Stereo, and RGB-D Cameras,' IEEE Transactions on Robotics , vol. 33, no. 5, pp. 1255-1262, 2017.
- [8] T. Schneider, M. Dymczyk, M. Fehr, K. Egger, S. Lynen, I. Gilitschenski, and R. Siegwart, 'maplab: An open framework for research in visual-inertial mapping and localization,' IEEE Robotics and Automation Letters , 2018.
- [9] M. Li and A. I. Mourikis, 'Optimization-based estimator design for vision-aided inertial navigation,' in Robotics: Science and Systems , 2013, pp. 241-248.
- [10] D. G. Lowe, 'Distinctive Image Features from Scale-Invariant Keypoints,' IJCV , vol. 60, no. 2, pp. 91-110, 2004.
- [11] E. Rublee, V. Rabaud, K. Konolige, and G. R. Bradski, 'ORB: An efficient alternative to SIFT or SURF,' in ICCV , 2011.
- [12] A. Alahi, R. Ortiz, and P. Vandergheynst, 'FREAK: Fast Retina Keypoint,' in CVPR , 2012.
- [13] S. Ren, K. He, R. B. Girshick, and J. Sun, 'Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks,' IEEE Trans. Pattern Anal. Mach. Intell. , vol. 39, no. 6, pp. 1137-1149, 2017.
- [14] E. Shelhamer, J. Long, and T. Darrell, 'Fully Convolutional Networks for Semantic Segmentation,' IEEE Trans. Pattern Anal. Mach. Intell. , vol. 39, no. 4, pp. 640-651, 2017.
- [15] Y. Verdie, K. M. Yi, P. Fua, and V. Lepetit, 'TILDE: A Temporally Invariant Learned DEtector,' in CVPR , 2015.
- [16] K. M. Yi, E. Trulls, V. Lepetit, and P. Fua, 'LIFT: Learned Invariant Feature Transform,' in ECCV , 2016.
- [17] Y. Tian, B. Fan, and F. Wu, 'L2-Net: Deep Learning of Discriminative Patch Descriptor in Euclidean Space,' in CVPR , 2017.
- [18] A. Mishchuk, D. Mishkin, F. Radenovic, and J. Matas, 'Working Hard to Know Your Neighbor's Margins: Local Descriptor Learning Loss,' in Advances in Neural Information Processing Systems (NeurIPS) , 2017.
- [19] D. DeTone, T. Malisiewicz, and A. Rabinovich, 'SuperPoint: SelfSupervised Interest Point Detection and Description,' in CVPR Workshops , 2018.
- [20] R. I. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision , 2nd ed. Cambridge University Press, ISBN: 0521540518, 2004.
- [21] S. Lynen, T. Sattler, M. Bosse, J. A. Hesch, M. Pollefeys, and R. Siegwart, 'Get out of my lab: Large-scale, real-time visual-inertial localization.' in Robotics: Science and Systems , 2015.
- [22] M. Zhang, Y. Chen, and M. Li, 'Vision-aided localization for ground robots,' in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2019.
- [23] M. Kaneko, K. Iwami, T. Ogawa, T. Yamasaki, and K. Aizawa, 'MaskSLAM: Robust Feature-Based Monocular SLAM by Masking Using Semantic Segmentation,' in CVPR - Workshops , 2018.
- [24] T. Naseer, G. L. Oliveira, T. Brox, and W. Burgard, 'Semanticsaware Visual Localization Under Challenging Perceptual Conditions,' in ICRA , 2017.
- [25] R. B. Girshick, J. Donahue, T. Darrell, and J. Malik, 'Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation,' in CVPR , 2014.
- [26] D. DeTone, T. Malisiewicz, and A. Rabinovich, 'Self-Improving Visual Odometry,' CoRR , vol. abs/1812.03245, 2018.
- [27] C. Szegedy, A. Toshev, and D. Erhan, 'Deep Neural Networks for Object Detection,' in Advances in Neural Information Processing Systems (NeurIPS) , 2013.
- [28] C. G. Harris and M. Stephens, 'A Combined Corner and Edge Detector,' in Alvey Vision Conference (AVC) , 1988.
- [29] E. Rosten and T. Drummond, 'Machine Learning for High-Speed Corner Detection,' in ECCV , 2006.
- [30] T. Lindeberg, 'Edge Detection and Ridge Detection with Automatic Scale Selection,' IJCV , vol. 30, no. 2, pp. 117-156, 1998.
- [31] J. Matas, O. Chum, M. Urban, and T. Pajdla, 'Robust Wide Baseline Stereo from Maximally Stable Extremal Regions,' in BMVC , 2002.
- [32] H. Bay, T. Tuytelaars, and L. J. V. Gool, 'SURF: Speeded Up Robust Features,' in ECCV , 2006.
- [33] Q. Zhu, M. Yeh, K. Cheng, and S. Avidan, 'Fast Human Detection Using a Cascade of Histograms of Oriented Gradients,' in CVPR , 2006.
- [34] M. Calonder, V. Lepetit, C. Strecha, and P. Fua, 'BRIEF: Binary Robust Independent Elementary Features,' in ECCV , 2010.
- [35] N. Savinov, A. Seki, L. Ladicky, T. Sattler, and M. Pollefeys, 'Quadnetworks: unsupervised learning to rank for interest point detection,' in CVPR , 2017.
- [36] X. Zhang, F. X. Yu, S. Karaman, and S.-F. Chang, 'Learning Discriminative and Transformation Covariant Local Feature Detectors,' in CVPR , 2017.
- [37] M. Brown and D. G. Lowe, 'Automatic Panoramic Image Stitching using Invariant Features,' IJCV , vol. 74, no. 1, pp. 59-73, 2007.
- [38] V. Balntas, K. Lenc, A. Vedaldi, and K. Mikolajczyk, 'HPatches: A Benchmark and Evaluation of Handcrafted and Learned Local Descriptors,' in CVPR , 2017.
- [39] R. Mitra, N. Doiphode, U. Gautam, S. Narayan, S. Ahmed, S. Chandran, and A. Jain, 'A Large Dataset for Improving Patch Matching,' CoRR , vol. abs/1801.01466, 2018.
- [40] Z. Luo, T. Shen, L. Zhou, S. Zhu, R. Zhang, Y. Yao, T. Fang, and L. Quan, 'GeoDesc: Learning Local Descriptors by Integrating Geometry Constraints,' in ECCV , 2018.
- [41] Y. Ono, E. Trulls, P. Fua, and K. M. Yi, 'LF-Net: Learning Local Features from Images,' in Advances in Neural Information Processing Systems (NeurIPS) , 2018.
- [42] S. Ioffe and C. Szegedy, 'Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift,' in ICML , 2015.
- [43] V. Nair and G. E. Hinton, 'Rectified Linear Units Improve Restricted Boltzmann Machines,' in ICML , 2010.
- [44] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, 'The Cityscapes Dataset for Semantic Urban Scene Understanding,' in CVPR , 2016.
- [45] J. Li, Y. Song, J. Zhu, L. Cheng, Y. Su, L. Ye, P. Yuan, and S. Han, 'Learning from Large-scale Noisy Web Data with Ubiquitous Reweighting for Image Classification,' CoRR , vol. abs/1811.00700, 2018.
- [46] Y. Song, X. Chen, J. Li, and Q. Zhao, 'Embedding 3D Geometric Features for Rigid Object Part Segmentation,' in ICCV , 2017.
- [47] T. Qin, P. Li, and S. Shen, 'Vins-mono: A robust and versatile monocular visual-inertial state estimator,' IEEE Transactions on Robotics , vol. 34, no. 4, pp. 1004-1020, 2018.
- [48] S. Leutenegger, S. Lynen, M. Bosse, R. Siegwart, and P. Furgale, 'Keyframe-based visual-inertial odometry using nonlinear optimization,' IJRR , vol. 34, no. 3, pp. 314-334, 2015.
- [49] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, 'Automatic differentiation in pytorch,' in Advances in Neural Information Processing Systems (NeurIPS) - Workshop , 2017.
- [50] E. Simo-Serra, E. Trulls, L. Ferraz, I. Kokkinos, P. Fua, and F. MorenoNoguer, 'Discriminative Learning of Deep Convolutional Feature Point Descriptors,' in ICCV , 2015.
- [51] S. Umeyama, 'Least-squares estimation of transformation parameters between two point patterns,' IEEE Trans. Pattern Anal. Mach. Intell. , no. 4, pp. 376-380, 1991.
Yafei Song Di Zhu Jia Li Yonghong Tian Mingyang Li
Learning Local Feature Descriptor with Motion Attribute for Vision-based Localization
In recent years, camera-based localization has been widely used for robotic
applications, and most proposed algorithms rely on local features extracted
from recorded images. For better performance, the features used for open-loop
localization are required to be short-term globally static, and the ones used
for re-localization or loop closure detection need to be long-term static.
Therefore, the motion attribute of a local feature point could be exploited to
improve localization performance, e.g., the feature points extracted from
moving persons or vehicles can be excluded from these systems due to their
unsteadiness. In this paper, we design a fully convolutional network (FCN),
named MD-Net, to perform motion attribute estimation and feature description
simultaneously. MD-Net has a shared backbone network to extract features from
the input image and two network branches to complete each sub-task. With
MD-Net, we can obtain the motion attribute while avoiding increasing much more
computation. Experimental results demonstrate that the proposed method can
learn distinct local feature descriptor along with motion attribute only using
an FCN, by outperforming competing methods by a wide margin. We also show that
the proposed algorithm can be integrated into a vision-based localization
algorithm to improve estimation accuracy significantly.