Moving Car Recognition and Removal for 3D Urban Modelling Using Oblique Images

: With the progress of photogrammetry and computer vision technology, three-dimensional (3D) reconstruction using aerial oblique images has been widely applied in urban modelling and smart city applications. However, state-of-the-art image-based automatic 3D reconstruction methods cannot effectively handle the unavoidable geometric deformation and incorrect texture mapping problems caused by moving cars in a city. This paper proposes a method to address this situation and prevent the inﬂuence of moving cars on 3D modelling by recognizing moving cars and combining the recognition results with a photogrammetric 3D modelling procedure. Through car detection using a deep learning method and multiview geometry constraints, we can analyse the state of a car’s movement and apply a proper preprocessing method to the geometrically model generation and texture mapping steps of 3D reconstruction pipelines. First, we apply the traditional Mask R-CNN object detection method to detect cars from oblique images. Then, a detected car and its corresponding image patch calculated by the geometry constraints in the other view images are used to identify the moving state of the car. Finally, the geometry and texture information corresponding to the moving car will be processed according to its moving state. Experiments on three different urban datasets demonstrate that the proposed method is effective in recognizing and removing moving cars and can repair the geometric deformation and error texture mapping problems caused by moving cars. In addition, the methods proposed in this paper can be applied to eliminate other moving objects in 3D modelling applications.

Introduction
Three-dimensional (3D) urban modelling is an important and basic task for smart city applications such as city planning, autonomous driving, and emergency decision making [1,2]. Recently, oblique image photogrammetry has been rapidly applied in urban modelling because it can provide both geometric and texture information after automatic model reconstruction processing. The efficiency of oblique image acquisition has dramatically improved with the development of unmanned aerial vehicle (UAV) technology [3]. In addition, many exceptional algorithms for image-based 3D reconstruction have emerged [4][5][6][7][8] and have greatly increased the process efficiency and reduced the cost of 3D city digitization.
The basic assumption of multiview stereo (MVS) vision is that objects maintain a static state during image collection. However, this assumption cannot be easily satisfied in urban modelling applications because there are many moving cars during image collection. Thus, moving cars will inevitably cause errors in the modelling results, especially in the areas containing intersections. The problems caused by moving cars are summarized as follows: 1.
Geometric Deformation. If a car is stationary and then leaves during image acquisition, parts of the images will contain car information, and the other parts will not contain car information. The texture inconsistency will lead to mismatching and mesh deformations in the reconstructed 3D model. As shown in Figure 1a, the car in the red selection box was stationary and then left the region during data collection, which resulted in geometric deformations in the generated triangle mesh.

2.
Error Texture Mapping. In the texture generation stage of 3D model reconstruction, moving cars will influence texture selection. If there is no pre-processing to identify moving cars, the incorrect moving-car texture information may be selected for mesh patches, resulting in texture mapping errors. As shown in Figure 1b, there are many moving cars at the intersection, and there is a large traffic flow. Due to incorrect texture selection problems caused by the moving cars in road regions, the texture of the reconstructed 3D model looks very disorganized and is quite different from the 3D model of a common road area (e.g., most of the white road marks in front of the zebra crossing are contaminated by car textures). Currently, the existing 3D reconstruction methods using multiview images cannot effectively solve the problems caused by moving cars, which are inevitable in metro cities. However, the need for realistic and accurate 3D models is still very high, especially in the applications of smart transportation, unmanned driving, etc. Therefore, manual mesh editing is a practical solution to improve model quality [9]; however, it requires considerable time and manpower. Therefore, we hope to resolve the issues by integrating moving car recognition into the 3D reconstruction pipelines to prevent the need for postprocess editing of the produced 3D models.
In this paper, we propose a moving car recognition and removal method that combines deep learning and multiview constraints. First, we use an object detection method based on depth learning to detect cars from oblique images. Subsequently, the car detection results and multiview image information are combined to recognize the moving cars. Then, the textures of the moving cars are erased from the original images, and the corresponding local deformation meshes are flattened. Finally, the car textures recognized during error detection are rejected by the random sample consensus (RANSAC) [10] method in the texture mapping step, and the texture holes are filled using available deep learning approaches. Figure 2 shows the comparison results before and after processing the moving cars using the method proposed in this paper. The rest of this paper is organized as follows: Section 2 is a brief introduction to object detection and multiview 3D reconstruction. Section 3 elaborates on the method presented in this paper. Section 4 provides an evaluation and analysis of the experiments. The discussion is presented in the last section.

Object Detection
Object detection methods mainly include three steps: region selection, feature extraction [11][12][13][14][15][16] and classification [17][18][19][20][21][22]. For example, paper [23] adopted the deformable part model (DPM [21]) classifier with a histogram of oriented gradient (HOG [14]) to detect street-view cars, and it achieved robust car detection. However, traditional object detection using rule or texture patterns has two main defects: (1) artificially designed features are subjective, usually only designed for specific detection tasks, and lack robustness to the diversity of targets; (2) when making region proposals, using a sliding window generates redundant windows, resulting in low efficiency. Thus, traditional object detection methods have low performance and high computational costs [24].
With the development of deep neural networks and the continuous improvement in GPU performance, convolutional neural networks (CNNs) have been widely used in image recognition and object detection [25]. An object detection method based on a region proposal, called R-CNN [24], was proposed to solve the problem of window redundancy in traditional methods and improve window quality while guaranteeing the recall rate. To improve detection efficiency, Fast R-CNN [26] and Faster R-CNN [27] were subsequently proposed. However, these approaches consist of two separate stages: bounding box detection and classification, and the demand for real-time detection still cannot be achieved. Therefore, a method named "you only look once" (YOLO) [28] was proposed. This method can generate a region proposal box during classification. The entire detection process has only one stage, and the speed is greatly improved. To solve the issue of inaccurate locations determined by YOLO, the single-shot detector (SSD) [29] network applies the anchor mechanism of Faster R-CNN, but SSD is not effective in detecting small objects. Similar approaches have been proposed to further improve the effectiveness [30][31][32][33] and to reach a better balance between detection accuracy and efficiency. Recently, a novel fully convolutional method named ImVoxelNet [34] was proposed, and it can obtain state-of-theart results in car detection on KITTI [35] and nuScenes [36] benchmarks among all methods that accept RGB images.
In summary, object detection methods based on deep learning are far superior to traditional algorithms in terms of accuracy and efficiency. On the other hand, the texture patterns of cars are relatively simple and easy to detect. The reference geographic information of an image can be used to solve the scale issue of car detection because the images were collected by a drone with GPS information. Both pieces of information can improve car detection recall. In this paper, we apply the deep learning-based algorithm in reference [37] to detect car information in oblique images.

Moving Object Detection
Moving objects are unavoidable interferences in real-world scenes. In order to build clean scene models, some researchers have proposed image-based methods to detect and remove moving objects in video sequences. The related algorithms can be summarized into two types: optical flow algorithm [38] and background subtraction [39]. The general idea of the optical flow method to detect moving objects is using the concept of flow vectors in two subsequent images. The background subtraction method detects moving objects by evaluating the difference of pixel features of the current scene image against the reference background images. However, such algorithms require the camera to be fixed. Some extensions are able to handle moving cameras by registering the images against the background model [40,41]. However, this class of algorithms needs information about the appearance of the background scene, and in most cases, this assumption does not hold [42]. Moreover, other deep learning approaches to detect moving objects have also been proposed [43].
Recently, some researchers have proposed methods to detect and remove moving objects in laser data [42,44,45]. In these works, the authors exploit Dempster-Shafer Theory [46] to evaluate the occupancy of a lidar scan and discriminate points belonging to the static scene from moving ones. The differences of these works are the methods to calculate the occupancy probability, which will seriously influence the detection results. In this paper, combined with the overlap redundancy of oblique images in multiview 3D reconstruction, we introduce a method to detect moving cars using multiview constraints.

Multiview 3D Reconstruction
As shown in the green wireframe in Figure 3, multiview image-based 3D reconstruction pipelines can be divided into four stages: structure from motion (SfM) [4,8], MVS [5,47,48], surface reconstruction (mesh) [6,49] and texture mapping [7,50]. Currently, these 3D reconstruction methods can generate high-quality 3D models, but they cannot effectively solve the geometric deformation and error texture mapping problems caused by moving cars. Researchers have proposed methods to solve these problems. Reference [7] proposed a method to remove the image patch of moving objects by texture RANSAC during the texture mapping procedure and achieved relatively good results in some situations. However, because the textures of moving objects are eliminated gradually using RANSAC iteration processing, this method will fail when there are many incorrect texture samples in the candidate images. Directly applying postprocessing to the final mesh model is another way to address geometric deformation and error texture mapping problems. Reference [9] proposed a postprocess editing method for a 3D model that is able to remove vehicles in the model by object detection, mesh flattening and image completion in the selected areas. However, this method requires manual intervention and has an image completion instability problem. We want to provide an automatic solution for the texture and geometric modelling problems caused by moving cars by integrating moving car recognition into a 3D modelling procedure.

Method
The method proposed in this paper combines multiview 3D reconstruction with moving car detection to achieve automatic recognition and removal of moving cars in the process of 3D model generation. The pipelines of this method are shown in Figure 3, and the green wireframe is the basic process of multiview 3D reconstruction. To solve the mesh deformation and texture error problems caused by moving cars, we integrate car detection, moving car recognition and cleaning (as shown in the red wireframe in Figure 3) into the abovementioned 3D reconstruction procedure. Each step in this method will be explained below.

Cars Detection Using Mask R-CNN
Aerial oblique image datasets captured by UAVs are the most widely used datasets for city-scale 3D modelling [51]. Compared to traditional aerial images, oblique image data have the characteristics of high spatial resolutions, a large range of scales, and large numbers. The accuracy and efficiency of an algorithm applied on oblique images should be considered when selecting the object detection method. To obtain accurate masks of cars in oblique images, we select the Mask R-CNN [37] algorithm as the network framework for car detection. Mask R-CNN adds a full convolutional network (FCN) layer to Faster R-CNN for each region of interest (ROI) and is able to detect the target object bounding box and predict the segmentation mask simultaneously. Thus, there is no need for further postprocessing of the image segmentation mask based on the bounding box. The Mask R-CNN model is shown in the green dotted box in Figure 4. Considering that most of the vehicles in cities are cars, we divided vehicles into two types, cars and non-cars, without taking into account other vehicle types. In addition, parked-car switching (PCS), which is when a parked car leaves and another car parks in the same parking space, occurs very often in cities. We introduce a lightweight network, called MobileNet v3 [52], to classify the colour information of the detected cars and provide auxiliary reference information for PCS detection (as shown in the red dotted box of Figure 4). The colours of cars in this paper are mainly divided into white, red, yellow, blue, black, silver grey and other colours (indicated by brown in the figure). Figure 5 shows the results of the car information detected by our network model, and the masks in the figure represent the car locations and colour information.

Definition of the Car Moving State (CMS)
Using the method described in Section 3.1, the location and colour information of cars can be identified in each image. However, in actual applications, the moving state of a car is more complicated than expected, and the recognition result of a car state has a serious influence on the processing strategy selection in 3D reconstruction. Therefore, to effectively solve the problems of mesh deformations and texture error mapping, we need to classify the states of the detected cars to perform a proper processing strategy according to the CMS.
As shown in Figure 6, we divide the CMSs into the following three categories: (1) Moving: Cars that were in a driving state during image collection, as indicated by the red line box in Figure 6a; (2) Short stay: Cars that were temporarily stationary and then left during image collection, as shown in the yellow boxes in Figure 6b,c, and in Figure 6c, different cars that are stationary in the same parking region; and (3) Stationary: Cars whose parking positions never changed throughout the whole image collection process, such as the car identified by the green wireframe in Figure 6d.   According to the above definitions, for stationary cars, we do not need to perform any special processing in the 3D modelling. For a moving car, the geometric information of the car cannot be generated in the dense matching stage because it is not able to find correct correspondence pixels due to its constantly changing position in the images. Therefore, we only need to erase the texture of the moving car in the source image to prevent the problem of error texture mapping caused by the selected texture of the moving car. In the same way, for a temporarily stationary car, we not only need to erase the texture of the car in the source image but also flatten the triangle mesh of the region where the car was temporarily stationary.

Moving Car Recognition
• Overview Notably, moving cars cannot be detected by only a single image because we need to compare content between images to find "moving" information. Therefore, combined with the overlap redundancy of oblique images in multiview 3D reconstruction, we introduce a method to detect moving cars using multiview constraints. The basic principle of the method is shown in Figure 7. During image collection, if a car does not move, the corresponding texture patches of the parked car among the multiview images should be consistent. Conversely, if the texture patches of a detected car among multiview images are different, the car has moved. The green scene area in Figure 7 shows three images with different car texture patches, indicating that these cars moved. Therefore, a moving car can be recognized according to the texture differences of the detected car among multiview images.

• Detailed Procedure for Moving Car Recognition
A triangular mesh is a data structure that is commonly used to represent 3D models created by the MVS method; however, there are multiple types of mesh structures, so in this paper, we also apply triangles as the basic computation elements. The geometric structure of an urban scene can be represented by a triangular mesh S = { f i }(i = 1, 2, ..., n). We denote the captured images as I = {I i }(i = 1, 2, ..., m), and the car masks detected in Section 3.1 for image I i are denoted by Mask(I i ) = mask j . The perspective projection determined by SfM of image I i is denoted by ∏ i : R 3 → R 2 , and the projection from mesh S onto the image I i is denoted by ∏ i,S : ∏ i (S) → S i . The specific steps for CMS recognition are shown in Algorithm 1:

1.
For each image I i , sequentially remove the car mask (mask j ) that has not yet been processed in Mask(I i ). According to the set of triangles (S i ) that is visible on I i , we can obtain patch j by collecting the triangles whose projected positions are within the area of mask j (the green marked region in Figure 7).

2.
Count the number of other images (denoted by n k ) that are also visible to patch j but were not processed during the previous step. Count the number of detected car masks (denoted by n 1 ) that overlap with the projection region of patch j in these n k images. If ratio 1 = n 1 /n k < δ 0 (0 < δ 0 < 0.08), only a very small number of cars passed by patch j during image collection, and mask j and the other n 1 car masks should be marked as moving. Then, apply this process to the next car mask mask j+1 . Otherwise, go to the next step. 3.
If ratio 1 < δ 1 (0.8 < δ 1 < 0.95), there were cars parked in patch j for a period of time during the image collection. Then, the states of these n 1 + 1 car masks should be marked as short stay, and the next car mask mask j+1 should be in the processing sequence. Otherwise, the local region patch j is always occupied by the car; hence, go to the next step.

4.
Calculate the colour histogram of these n 1 car masks and find the largest peak value n 2 . If ratio 2 = n 2 /n 1 > δ 2 (0.75 < δ 2 < 0.9), the detected corresponding texture patches of the area of patch j in the multiview images are consistent and should be considered the same car. Then, the states of these n 1 + 1 masks are marked as stationary. Otherwise, the car in patch j has changed, so the states of these n 1 + 1 masks are recorded as short stay. Continually apply these steps to all the unprocessed masks of cars sequentially until there are no unprocessed masks. Count the number of other visible images (n k ) to patch j and count the number of detected car masks (n 1 ) that overlap with the projection region of patch j 7.
The CMS is moving 9.
The CMS is short-stay 11. else 12.
Calculate the colour histogram of these n 1 car masks and find the largest peak value n 2 13.
The CMS is short-stay 15. else 16. The

Mesh Clean
From Section 3.2.1, we know that some regions of the 3D urban scenes will contain temporarily stationary cars during the period of image collection. If these regions containing short-stay cars were captured from multiple positions and directions by cameras during flight, some geometric errors will be introduced into the reconstructed meshes, as shown in the red marquee in Figure 1a. The deformation of the car model is very obvious, and some textures are incorrectly selected. Therefore, it is necessary to clean the triangles of the reconstructed mesh model corresponding to the patch of short-stay cars before texture mapping. Using the recognition method in Section 3.2.2, we can obtain the CMS from the given oblique images. Because the geometric structures of most parked-car regions are planar, we can directly flatten the distorted geometric meshes.

Texture Preprocessing
To prevent texture mapping errors caused by selecting moving cars in the texture mapping stage, it is necessary to clean the moving car textures before texture mapping. According to the recognition results in Section 3.2.2, we can erase the textures of the cars from the source oblique images that are in moving or short-stay states. However, on the one hand, some cars may not have been detected by the process in Section 3.1, and on the other hand, buses, trucks and other vehicles may not have been trained in our object detection network. Thus, the textures of these missed detection vehicles may still appear in the final 3D urban scenes. To eliminate the influence of these few unrecognized vehicle textures, we apply the RANSAC method in article [7] during texture mapping to select correct texture patches. Different from article [7], we removed most of the incorrect texture samples in the previous step, so the probability of successful missed vehicle texture cleaning is higher in our paper.
In addition, there may be different cars parked at the same place during the process of image collection, such as in the parking region in the yellow box in Figure 6c. After removing the textures of the short-stay cars, it is possible that no other textures will be available for texture mapping in these places, which will lead to a blank texture in the obtained 3D models. To prevent this situation, we use the image inpainting method in reference [53] to fill the texture holes generated by erasing the moving car textures in the candidate texture images.

Data Description
In order to cover all parts of the objects with the minimum flight time and suitable camera inclination, we collect aerial oblique images via the penta-view oblique photography platform on a DJI M600 UAV with a size of 7952 * 5304 pixels, and the ground sample distance (GSD) is 2-6 cm. We collect all experimental data in the Wuhan area, China. We select three typical regions in the urban scene to validate the proposed methods, including viaduct expressways, parking lots, and intersections with high traffic flows. In addition, aerial oblique images are captured under different weather and light conditions. The experimental data preprocessing part uses the open-source algorithms OpenMVG [54] and OpenMVS [55] on the network.

Car Detection
The ground truth of the training datasets for car detection was obtained manually. There are 5000 training samples with different scales, colours and shooting angles. We train the detector for 70 epochs in total and apply 1000 iterations for each epoch. With the experimental environment shown in Table 1, the training time for car detection is approximately 12 h. As depicted in Figure 8, the loss values gradually stabilize as the number of training epochs increases, indicating that the model is close to convergence. By applying the well-trained detector to the validation datasets, we obtain a mean average precision (mAP) of 92.9% for car detection and a precision of 89.7% for car colour recognition.  We also collected testing datasets with different GSDs, different shooting angles, and different lighting conditions to verify the robustness of the well-trained detector. Figure 9 shows several results of testing. Specifically, Figure 9a,b contain aerial top-view photos with GSDs of 2.5 cm, and Figure 9c contains oblique aerial images with GSDs of 3.5-6 cm. In addition, Figure 9a,c depict pictures taken under sufficient light conditions on sunny days, while Figure 9b shows the images collected on cloudy days. As shown in Table 2, the detectors perform well in the three different test scenes. However, a small number of cars are missed or falsely detected (see the red bounding boxes in Figure 9b). Furthermore, the colours of a small number of cars are incorrectly recognized. The main reasons for missed car detection are as follows:

1.
Difficult samples, such as cars that are difficult to distinguish from the background, are not easy to notice. For example, a black car in a shadow is not detected (see the green box in Figure 9c).

2.
It is challenging to detect cars occluded by a large region, as shown in Figure 9c in the red wireframe. Compared to the other two experimental scenes (Figure 9a,b), the third scene (Figure 9c) with a lower recall rate conforms to reality. On the one hand, there are more occlusion cases in oblique images. On the other hand, there are more shadow areas in urban scenes when light is sufficient.
In addition, we obtain different precision results for car colour recognition in the three types of scenes. (1) The test results reveal that car colour recognition precision on a cloudy day is higher than that on a sunny day. On the one hand, when the light is sufficient, some cars have reflective and overexposure problems, leading to colour recognition errors. In Figure 9a, the silver-grey car in the green wireframe is mistakenly recognized as white. On the other hand, there are more shadow areas on sunny days, which will decrease the performance of car colour recognition. As shown in the green wireframe in Figure 9c, because the quality and colour discrimination of the image in the shaded area are reduced, the white car is mistakenly identified as silver-grey. (2) We obtain a higher colour recognition precision under ortho shooting than under oblique shooting. Due to a significant variation in object scale in oblique shooting scenes, the qualities of different areas in the image are uneven, resulting in a relatively low precision for car colour recognition.

Moving Car Recognition
According to the movement state of cars defined in Section 3.2.1, we mainly select three urban scenes, including rapid urban roads, public parking areas and traffic light intersection areas, to conduct car state recognition experiments. On rapid urban roads, such as viaducts, cars are mainly moving. In public parking areas, PCS happens very often. We use the same parameters for all the recognition experiments. Specifically, we set δ 0 as 0.05, δ 1 as 0.85 and δ 2 as 0.8. Among them, δ 1 is positively related to the mAP of car detection, and δ 2 is related to the precision of car colour recognition. In our experimental results, we depict the different states of cars by mask with different colours. Red, yellow and green represent the moving state, short-stay state, and stationary state, respectively. The geometric structures of the obtained 3D models are stored using the open-source algorithm OpenMesh [56].
As shown in Figure 10, the method in Section 3.2.2 is applied to recognize car states using urban viaduct images. Figure 10e shows that all the detected cars are accurately recognized as moving because cars on rapid roads, such as urban viaducts, do not stop under normal circumstances and are always in moving states. On the other hand, Figure 10c also shows that among the textures in the multiview images of the red frame area in Figure 10a, most do not have corresponding car information. Therefore, according to the recognition rules in Section 3.2.2, the car corresponding to the local scene area is a moving car, and its texture in Figure 10a should be erased. We select a small parking lot in front of an office building for the public parking lot urban scenes, as pictured in Figure 11. It can be seen from the recognition results of the car state in Figure 11c that most of the cars in the parking lot area never moved, and only a small number of cars were temporarily stationary. Table 3 shows the statistical results using the method in Section 3.2.2 for some of the car areas selected in Figure 11a. Table 3 reveals that the value of ratio 1 or ratio 2 is relatively low for short-stay cars. Moreover, the value ratio 1 of the yellow box is relatively high, while the value ratio 2 is relatively low, indicating that different cars were located in the same parking region and that this is a case of PCS. Furthermore, the geometric mesh of the scene in Figure 11b helps to further verify the recognition results in Figure 11c. For the regions where cars have stayed (see the green and yellow masked parts in Figure 11c), the geometric structures of the cars at the corresponding positions in Figure 11b have been reconstructed. Conversely, if the geometric structure is not reconstructed, as shown in the red box in Figure 11a,b, the car is in a moving state, and its value n 1 is relatively small. In the same way, we can also learn from the 3D car models in the yellow boxes in Figure 11b that there are cars parked on these unoccupied regions in Figure 11a during other periods of image acquisition, so they are in the short-stay state. In addition, it can be seen from the green-box areas in Figure 11d that the triangle meshes of the short-stay regions are flat.
To further verify the effectiveness of the proposed method, we select an intersection area (see Figure 12a) that has a heavy traffic flow and contains various cars in the three different states defined for the experiments. Figure 12b shows the results of car detection in Figure 12a, from which it can be seen that the cars in the scene are well detected. Figure 12d shows the state recognition results of the cars detected in Figure 12b. Figure 12d reveals that the cars on the road are mostly in moving states, while the cars waiting for the traffic lights are likely in the short-stay state. The geometric deformation results of the car models in the yellow boxes in Figure 12c also explain this result. PCS frequently occurs in parking lot spaces (see the top left of Figure 12a). However, unlike Figure 11, more PCS cases occur in this region, where the yellow masks are more than the green masks. Parked cars change more frequently since the parking lot in Figure 12a is in front of a mall.

Moving Car Cleaning
Once the states of the cars are recognized, the textures of the moving cars can be erased effectively from the oblique images. Additionally, the geometric deformation caused by the temporarily stationary cars can be easily repaired. In this section, we carry out comparative tests based on the experimental results in Section 4.3 to validate the effectiveness of our method in removing moving cars during 3D reconstruction. Figure 13 depicts the comparison results of moving car removal from the urban viaduct scene in Figure 10. Specifically, we obtain the 3D scene in Figure 13a using traditional 3D reconstruction pipelines. In addition, we reconstruct the 3D model in Figure 13b via the proposed method in this paper. From the comparison results, the method proposed in this article can effectively remove the textures of all moving cars from the viaduct pavement, and the 3D scene of the whole viaduct road looks clean. Moreover, the information about stationary cars is preserved completely. Figure 14 shows the experimental results of moving car removal from the parking lot scene in Figure 11. Two cases (the red and green box regions) were selected from Figure 14a,b, respectively, and demonstrate that our proposed method can better handle the geometry and texture of areas where cars were temporarily stationary. In addition, the parking spaces highlighted by the blue and yellow boxes in Figure 14 are occupied by different cars during the whole process of image acquisition. Therefore, for this kind of region, we need to fill in the texture voids in the images according to the rules in Section 3.3.2. Taking advantage of the algorithm in [53], we obtain a satisfactory region filling result. The comparison results are shown in the blue and yellow boxes in Figure 14.    Figure 15 shows the results of moving car removal from the intersection area in Figure 12. Similarly, we reconstruct the 3D scenes in Figure 15a,b using the traditional pipeline and the proposed method, respectively. The entire 3D scene in Figure 15a undoubtedly looks disorganised due to the moving cars. Compared to the results in Figure 15a, we obtain a more accurate and realistic 3D intersection scene after mesh cleaning and texture preprocessing, especially for typical road areas. All the moving vehicles on the road, including undetected buses and vans, are effectively removed, indicating that the proposed method is robust.

Conclusions
This paper proposes a method for moving car recognition and removal that combines deep learning and multiview constraints. Our method mainly includes three steps: car detection, moving car recognition and cleaning. The main contribution of this paper is to integrate object detection into 3D reconstruction pipelines to eliminate the geometric and texture problems caused by moving cars. Compared with the traditional 3D reconstruction method based on multiview images, the experimental results indicate that the proposed method can effectively moderate geometric deformations and texture distortions caused by moving cars in 3D urban scenes. In addition, the method in this paper has a high degree of automation, which can distinctly reduce the workload of manual postprocessing to repair 3D scene models, thereby improving the production efficiency of urban 3D scene modelling.
However, the method proposed in this paper is limited by the following issues, which provide insights for developing future works. (1) We have not yet considered other types of vehicles due to a lack of training samples in our object recognition experiments. Unfortunately, our training samples only contain cars, so other vehicles, such as motorcycles, buses, and trucks, are easily ignored during detection. (2) We only take advantage of colour information to distinguish the PCS cases. This method for similarity detection is too limited to be robust. Therefore, we will further explore this area in future work.

Data Availability Statement:
The data presented in this study are available on request from the corresponding author.


Machine learning model for clinical named entity recognition

Received Feb 22, 2020 Revised Aug 1, 2020 Accepted Sep 25, 2020 To extract important concepts (named entities) from clinical notes, most widely used NLP task is named entity recognition (NER). It is found from the literature that several researchers have extensively used machine learning models for clinical NER. The most fundamental tasks among the medical data mining tasks are medical named entity recognition and normalization. Medical named entity recognition is different from general NER in various ways. Huge number of alternate spellings and synonyms create explosion of word vocabulary sizes. This reduces the medicine dictionary efficiency. Entities often consist of long sequences of tokens, making harder to detect boundaries exactly. The notes written by clinicians written notes are less structured and are in minimal grammatical form with cryptic short hand. Because of this, it poses challenges in named entity recognition. Generally, NER systems are either rule based or pattern based. The rules and patterns are not generalizable because of the diverse writing style of clinicians. The systems that use machine learning based approach to resolve these issues focus on choosing effective features for classifier building. In this work, machine learning based approach has been used to extract the clinical data in a required manner.

INTRODUCTION
The patient's data ranging from diagnoses, treatments, problems, medications to imaging and clinical notes like discharge summaries are available in electronic health records (EHR). For quality, billing and outcome structured data are important. On the other hand, narrative text is more engaging, more expressive and captures patient's data more accurately. Clinical notes also contain data indicating the level of concern and uncertainty to others who are reviewing the note. Hence, in order to obtain clear perspective on the condition of the patient, an analysis of narrative text needs to be done. But, the manual analysis of huge number of narrative text is time consuming and prone to errors.
To resolve this issue, machine learning based systems can be used. It can be observed from the literature that various machine learners have been used. support vector machines (SVMs) [1] and hidden markov model (HMM) [2] are examples of such learners. To understand the natural language [3], natural language processing that focuses on development of models is being used. The framework of NLP includes modules for syntactic processing like tokenization, parts of speech tagging and sentence detection. Modules for named entity recognition tagging, extraction of relation and concept identification are included in the NLP systems. An NLP system that has semantic processing models for extraction of pre-defined information 1690 is information extraction system. In the medical field, researchers are using NLP systems for identification of biomedical concepts and clinical syndromes from radiology reports [4] and discharge summaries [5].
Clinical researchers and other medical operations make use of important information extracted by analysis of clinical notes in detailed manner. These clinical notes provide rich and detailed medical information. In the present work, we have built a machine learning model for extraction of medical NERs namely disease, test and treatment. An analysis has been done from the text of doctor's notes and records generated during interaction with patient.

RELATED WORK
Decision tree based NER model was built by Sekine et al. [6] that used features such as part-ofspeech tags extracted by morphological analyzer, specialized dictionary and character based information. This was developed for Japanese. Bikel et al. [7] used hidden markov model (HMM) for identification of named entity. Features like bi-gram and orthographic features like word case, word shape etc. were used. In his Ph.D thesis, Borthwick [8] used maximum entropy (MaxEnt) algorithm. McCallum et al. [9] extracted NER using algorithm based on conditional random fields. A semi Markov conditional random field algorithm was proposed by Sarawagi et al. [10] for extraction of named entity. The researches extended the semi Markov model with use of dictionary and notion of similarity function. An overall survey of NER research was provided by Naidu and Sekine [11].
Luu [12] proposed a framework that is based on different text mining and machine learning algorithms for addressing the challenges of clinical named entity recognition. The framework proposed has multiple levels and builds complex NER tasks. Different data sets-the CLEF 2016 challenge and BIONLP/NLPBPA 2004 were used for evaluation of the proposed method and the results validated the framework.
Mao et al. [13] opine that important clinical information related to diagnosis is available in Electronic medical record. By data mining of electronic medical record, recognition of medical named entity is done. In this research work, authors have taken ophthalmic electronic medical record as research object. In the beginning, under the guidance of specialist, training corpus is annotated. Later, trained HMM model is used in test set for recognition of entity. Finally, experiment is conducted for making comparison between the proposed algorithm and the algorithm based on word segmentation model. The results of the experimentation indicate that the algorithm achieves good results in the named entity recognition of electronic medical record. Li et al. [14] proposed a deep neural model BiLSTM-Att-CRF that is a combination of bidirectional long-short time memory network and attention mechanism. This improved the performance of NER in Chinese electronic medical records (EMRs). The proposed model achieved better results than other widely used models.
Qiu et al. [15] write that the goal of the clinical named entity recognition (CNER) is identification and classification of clinical terms like symptoms, exams, treatments, diseases. This is a crucial and fundamental task for clinical and translation research. In recent years, deep learning models have been successful in CNER tasks. These models depend on recurrent neural networks which maintain a vector of hidden activations that propagate through time. This causes too much time for model training. In the present work, the researchers have proposed a residual dilated convolutional neural network with conditional random field (RD-CNN-CRF) to solve it. In this method, dictionary features and Chinese characters are projected first into dense vector representations. Later, they are fed into the residual dilated convolutional neural network to capture contextual features.
Li et al. [16] proposed a model combining language model conditional random field algorithm (CRF) and bi-directional long short-term memory networks (BiLSTM) to realize automatic recognition and entity extraction in unstructured medical texts. The researchers crawled 804 specifications of drug for asthma treatment from the Internet. Later quantization is done for the normalized field of drug specification word by a vector as the input to the neural network. Experimentation indicated that recall, system accuracy and F1 value are improved by 5.2%, 6.18% and 4.87% compared to traditional machine learning model. The proposed model can be applied to extract named entity information from drug specification.
Summarising the concepts, the electronic medical record is a description of patients physical condition [17]. Named entity recognition is the method used for clinical data extraction. The NER was a combination of dictionary and rules [18]. In clinical decision, NLP has become recent trend [19]. Researchers have evaluated various machine learning algorithms with various features [20]. UMLS, Ctakes and Medline were introduced as characteristics and using semi-Markov model, an accuracy of 85.23% was achieved [21]. Wang et al. [22] constructed tagged symptom corpus including 11,613 chief complaints. Wang et al. [23] completed manual annotation for 12 data of liver cancer in 115 medical records. Yan et al. [24] put forward a united model of word segmentation and named entity recognition based on dual decomposition. Jianbo, et al., [25] selected 800 medical records and established named entity tagged corpus among which word segmentation and partof-speech tagging utilize tools developed by Stanford University.

THE PROPOSED MODEL
The proposed model classifies clinical data and provides the data to concerned expert using machine learning framework and NLP technique. In the manual system, physicians and nurses have to go through the medical data and directs this data to concerned experts. It is time consuming, expensive and challenging task. The records of the patients include medical history, family history etc. The significant difference between classification of medical records and general text classification is word distribution. The proposed model uses machine learning framework for recognizing and extraction of concepts from clinical data. The framework includes an approach known as bidirectional long short tem memory-conditional random field (LSTM-CRF) initialized with general-purpose, off-the-shelf word embeddings. Figure 1 depicts the data flow used in the proposed model.
This is a mapping of entire input sequence paired with an entire state sequence to some dimensional feature vector. The probability as a log-linear model with the parameter vector has been modeled as where o ranges over all possible output sequences. The expression . ᶲ( , ) = ( , ) indicates a scoring how well the state sequence fits the given input sequence. Hence score can be defined as, where − 1, oi are weight vector is the bias corresponding to the transition from − 1 to oj1espectively. The algorithm used for the overall process is given in Figure 2. Medical records that consist of test conducted, patient's health status, response to the treatments and diseases are given as input. In the next stage, concepts like medical tests, diagnosis and treatments mentioned in the clinical records are classified into categories. Later, the records are divided into training data and testing data. 70% of data is used as training data and it is fed to the model. Testing data (30% of data) that consists of patient's information are fed to the model. Once the model is tuned for accuracy, the model will be ready to receive the real data. Then, the real data which is actually clinical records are fed to the pre developed model. The output includes list of words that indicate test conducted, problem diagnosed or treatment given. From the list of diseases and test conducted, the specializations are classified and displayed. The benefit of this is that the experts in specific area need not read all clinical record, they can directly read summary which saves lot of time. Using LSTM method which is based on machine learning, extraction of diagnosis and test names is extracted. NLP has been used for this. The screenshot is shown in Figure 3. Model building using training data The records are divided into training data and testing data. 70% of data is used as training data and it is fed to the model.

Step 4
Testing the model accuracy Testing data (30% of data) that consists of patient's information are fed to the model.

Step 5
Input Medical records The real data (clinical records) are fed to the pre developed model.

Step 6
Obtain output The output includes list of words that indicate test conducted, problem diagnosed or treatment given.
Step 7 Classify From the list of diseases and test conducted, the specializations are classified and displayed.
Step 7 End Once NER with NLP is applied for extraction of entities and their relationships, further processing is done. The disease names, test, diagnosis test are fed as input to machine learning framework. An output of the model will be classified data labeled with specialization as shown in Figure 4. Figure 5 and Figure 6 shows the execution screenshot during classification.

RESULTS AND DISCUSSIONS
In the proposed model machine learning algorithms used are support vector machine (SVM), naïve bayes, logistic regression, decision tree, random forest and light GBM. The screenshot related to accuracy of these algorithms is shown in Figure 7. The accuracy of the algorithms used is presented in graphical form in Figure 8. The model proposed can be used for extraction of medical data using NER and NLP technique. The machine learning model built into medical automation systems can be a good resource for medical experts as it saves lot of time spent for referring clinical records in detail. Also, administrative tasks can be easier as the model separates the diseases and treatment in to specializations.
The existing NLP systems for NER using clinical data consist of syntactic processing modules like sentence detection, tokenization, part-of-speech tagging etc. The semantic modules include concept identification, entity recognition, relation extraction and anaphoric resolution etc. So far, in the literature, it is observed that systems exists for extraction of named entities like disease, treatment etc which was useful for doctors to read summary information without reading complete clinical records. But, the proposed model goes one step further by classifying the named entities as per specialization. This can be embedded in health automation system for efficient delivery of services saving lot of time. Hence the proposed system can be a good candidate for the research in the area of NER in medical field.

CONCLUSION
Because of diverse writing style of clinicians, the rules and patterns are not generalizable. These issues can be addressed by making use of technologies like machine learning. Named entity recognition is grouped into three approaches. Machine learning based approaches, rule-based approaches and dictionary based approaches. The systems that use machine learning based approach focus on choosing effective features for classifier building. Several researchers have extensively used machine learning models for clinical NER. Databases such as PubMed which include medical publications have generated lot of interest among researchers for applying information extraction techniques to medical literature. In an attempt to contribute to the research in this area, this work proposed a machine learning model for clinical NER. The model proposed perfomed better compared to some of the existing methods.


A Hybrid Tracking Algorithm for Multistatic Passive Radar

In multistatic passive radars (MPR), multiple measurements from different bistatic pairs are usually used to locate the target due to the inaccurate direction of arrival estimation. In the real-world scenarios, the available bistatic pairs may be not enough and the probability of detection for each bistatic pair may be relatively low. It cannot be guaranteed that multiple simultaneous measurements will be available. In this case, the existing tracking algorithms face problems in the fast track confirmation, track continuity, and accuracy. In this article, we propose a novel hybrid tracking algorithm for the MPR. To handle the above-mentioned problems, we construct two classes of tracks, namely the high-precision track and low-precision track, for both the track initialization and track maintenance stages. The low-precision track can be converted into the high-precision track under certain conditions. This design facilitates the fast confirmation of high-precision tracks. Combined with the independent management of the high-precision tracks, the track continuity and accuracy metrics will be enhanced as well. The superiority of the proposed algorithm is validated via theoretical analyses and Monte Carlo simulations. Moreover, the experimental results using real data also demonstrate the practical effectiveness.

In the past four decades, a variety of architectures are available for the design of a multisensor tracking system [9], [10]. The main ideas of these tracking system can be grouped into two categories: state vector fusion and measurement fusion. The state vector fusion method [11]- [14] combines the filtered state vectors from different sensors to form a new estimate while the measurement fusion method [15]- [18] directly fuses measurements produced by different sensors. A comparison of these two methods can be found in [19].
The state vector fusion method requires each sensor to process its own measurements and keep its tracks separately. However, for the MPR, the direction of arrival (DOA) estimation is usually inaccurate, which results in a significant deviation in the tracks. Besides, complex clutter environments cause lots of false tracks. These inaccurate tracks and false tracks pose a great challenge to the track-to-track association and the track fusion.
To overcome the above-mentioned problems, the measurement fusion method is considered for the MPR. In the MPR, one bistatic range defines an ellipse/ellipsoid with the foci at the corresponding transmitter and receiver positions. Intercepting multiple ellipses/ellipsoids from multiple bistatic pairs make the target localization possible [20]- [24]. Further considering that the bistatic ranges are usually of high accuracy, it is attractive and popular to fuse multiple bistatic range measurements in the MPR.
Several multitarget tracking methods based on the measurement fusion have been proposed in [25]- [28] for the MPR. In [25], each track is initialized using at least three bistatic pairs. A nonlinear Kalman filter is then used to update tracks. This nonlinear Kalman filter makes it possible to update 3-D tracks with only one or two simultaneous detections. Such an idea is also used in [26]. Combined with the 2-D assignment method, the problem in the track maintenance stage under a single frequency network (SFN) can be solved. Both [25] and [26] do not provide a detailed description of the track initialization. Yi et al. [27] proposes an approximately optimal assignment method to solve the association problem among targets, measurements, and illuminators. The target's Cartesian position and velocity, namely secondary measurement, can be estimated after the association operation. The secondary measurements are then used to update tracks in the Cartesian coordinate. Choi et al. [28] present the auxiliary particle filter and the bootstrap particle filter under the probabilistic multihypothesis tracker (PMHT) measurement model for a DAB/DVB network.
The above-mentioned measurement fusion methods could work well if there are sufficient bistatic pairs and high detection This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ probability. But in the real-world scenarios, the multiple simultaneous measurements for the same target may not be guaranteed due to the insufficient bistatic pairs and low detection probability. In this scenario, the probability of obtaining a secondary measurement is reduced. Thus, the above-mentioned tracking algorithms encounter difficulties in the fast track confirmation, track continuity, and accuracy.
The objective of this article is to improve the tracking performance in the case of insufficient bistatic pairs and low detection probability. We propose a novel hybrid tracking algorithm for the MPR. The key to the proposed algorithm is that two classes of tracks, namely the high-precision track and low-precision track, are constructed. Unlike the above-mentioned tracking algorithm, the target information detected by only one bistatic pair is saved in the low-precision tracks instead of being directly discarded. After the association between low-precision tracks and the secondary measurement, the associated low-precision tracks are converted into a high-precision track. This design directly increases the probability of the high-precision track confirmation. Meanwhile, to make it possible to update tracks with one detection, a nonlinear Kalman filter is used in the track maintenance stage. In addition, combined with the independent management of the high-precision tracks, the track continuity and accuracy metrics will be enhanced as well. Monte Carlo (MC) simulations and theoretical derivation are used to evaluate the performance of the proposed algorithm. Its practical effectiveness is further verified using real data.
The rest of this article is organized as follows. Section II describes the difficulties in the target localization and track initialization in the case of insufficient bistatic pairs and low detection probability. Then, the proposed algorithm is described in Section III. Section IV evaluates the performance of the proposed algorithm through theoretical derivation and simulations. Section V demonstrates the performance using field experimental data. Finally, concluding remarks are drawn in Section VI.

II. PROBLEM FORMULATION
Consider a MPR system with N t transmitters and N r receivers. It is assumed that all the transmitters' and receivers' positions are known. A transmitter and a receiver form a bistatic pair. In the considered case, there are N = N t × N r bistatic pairs. Let x(t) and Z n (t) = {z (1) n (t), z (2) n (t), · · · , z (M n ) n (t)} be the target state and the measurements of bistatic pairn(n = 1, 2, · · · , N), respectively, where M n denotes the number of measurements in the nth bistatic pair. The timestamp t may be omitted under unambiguous situation. The relationship between the target and the m n th(m n = 1, 2, · · · , M n ) measurement can be expressed as (1) where h n (·)is the measurement function associated with bistatic pair n, and w n is the measurement noise. The measurement consists of the bistatic range r b , bistatic velocity v b , and azimuth θ. Without loss of generality, it is assumed that all the measurement noises of different bistatic pairs are independently and identically distributed (i.i.d.), with covariance matrix R.
In current passive radar systems, multiple measurements from different bistatic pairs are usually used to locate the target due to the inaccurate DOA estimation. At least two different bistatic ellipses are needed to locate the target in the case of 2-D space. When faced with a situation where the bistatic pair is insufficient and the detection probability is low, the probability that the target can be located, denoted by P L , is reduced. P L can be expressed as the function of the detection probability and the number of bistatic pairs. For description brevity, we assume that all the bistatic pairs have the same detection probability P d . The probability P L can then be calculated as where C i N is the number of combinations of i selected from N. Fig. 1 shows the P L calculated under various N and P d . As it is depicted, P L is less than 0.8 when the number of bistatic pairs is small (three or less), or the detection probability is low. If we only use the multistatic localization results to initialize tracks, it will result in a low probability of new track confirmation. The reason is intuitive. In the multistatic localization, the target can only be located when it is simultaneously detected by two or more bistatic pairs. The target measurement detected by only one bistatic pair is directly discarded. In order to make full use of all bistatic measurements, we introduce the concept of highprecision track and low-precision track and propose a hybrid tracking algorithm based on these two kinds of tracks in this article.

III. PROPOSED HYBRID TRACKING ALGORITHM
As shown in Fig. 2, the original measurements of each bistatic pair serve as the input of the multistatic hybrid tracking. In the multistatic hybrid tracking we introduce the concept of high-precision and low-precision track. The low-precision track refers to the track formed by the measurements of each single bistatic pair. The high-precision track is a track that is directly initialized by a secondary measurement sequence or that is obtained by the fusion of low-precision tracks associated with the same secondary measurement. It should be emphasized that the low-precision tracks are confirmed tracks, not tentative tracks.
If there is no more information to convert these low-precision tracks into high-precision tracks, they will be considered as target tracks instead of being deleted.
There are two modules in the tracking process, namely the track maintenance and track initialization. The track maintenance module mainly consists of the measurement-tomeasurement association, the conversion from low-precision tracks to the high-precision track, and the track update processes for both high-precision tracks and low-precision tracks. These key steps will be detailed below. The track initialization module is mainly used to initialize and confirm new tracks in both classes of tracks.
The main difference between the proposed algorithm and existing MPR tracking algorithms is that we have added a branch related to low-precision tracks after the high-precision track filtering. The role of this additional branch is to preserve the target measurements obtained by all the bistatic pairs, including those discarded in the measurement-to-measurement association step. After the measurement-to-measurement association, the low-precision tracks and the secondary measurement that represent the same target are associated with each other under certain conditions. Then, we convert the low-precision tracks and the associated secondary measurement to a high-precision track. Such design can increase the probability of high-precision track confirmation. Moreover, combined with the independent management of the high-precision tracks, the track continuity and accuracy metrics will be enhanced as well. In the proposed algorithm, both the measurement-to-measurement association and the low-precision track update impose a one-to-one constraint. There is no case where the same low-precision track is associated with multiple secondary measurements.

A. High-Precision Track Update
In Fig. 2, the high-precision track update is marked as step A. This step is mainly about the measurement-to-track association and the high-precision track update. Let X H (t) = {x (1) H (t)} be the state of all the high-precision tracks, where λ is the number of high-precision track at time t. If λ = 0, it means there is no high-precision track at this time, then skip to the next step directly. Since bistatic pairs are independent of each other, the measurement-to-track association can be performed in each bistatic pair individually. For example, in the nth bistatic pairs, the association problem between the high-precision tracks X H (t) and the measurements Z n (t) can be transformed into a 2-D assignment problem. It can be solved by the Hungarian algorithm [26]. This operation is repeated for all bistatic pairs. Let λ}. An extended Kalman filter is then used to update the high-precision track due to the highly nonlinear between Z (j) (t) and x (j) H (t). The detailed process of using EKF to update the target status can be found in [31]. With such an efficient nonlinear filter, it is possible to update tracks with only one measurement.
In order to avoid track duplication, the measurements associated with high-precision tracks will be deleted after highprecision track filtering. The remaining measurements are taken as the inputs of both step B and step C. The inputs for step B and C are exactly the same.

B. Measurement-to-Measurement Association
In Fig. 2, this part is marked as step B. The purpose of this step is to find the measurements from different bistatic pairs that represent the same target. At the same time, we obtain the secondary measurement mentioned in Section I by fusing these measurements and record the indexes of these measurements in a 2-D matrix. We use the method introduced in [27] to deal with this problem. The method in [27] is originally proposed for the MPR with an SFN. It is also applicable to the general MPR system. The method can be briefly described as follows.
1) Take two measurements from two bistatic pairs to construct a low-dimensional association hypothesis. 2) Given association probability P Ass , make a quick decision on all the association hypotheses constructed in step 1. 3) Fuse the association hypotheses accepted in step 2 and further collate them into candidate target groups.

4)
Reallocate the association between the measurements and the candidate targets by the global association model. At the same time, the target's Cartesian position and velocity, namely the secondary measurement are obtained. 5) Record the composition of the secondary measurement. It should be noted that the association probability P Ass is a user-defined parameter, which is discussed in [32].
Let z β snd (t) and P β snd denote the target state and covariance matrix of the βth secondary measurement at timet, respectively. The composition of the secondary measurement is recorded as which is a 2 × p matrix and p is the number of bistatic pairs that compose the secondary measurement. In the matrix, the first row of each column stores the index of the bistatic pair n, and the second row indicates the index of the measurement in the corresponding bistatic pair m n . For example, the composition of one secondary measurement is as follows: the first measurement in the first bistatic pair, the fourth measurement in the second bistatic pair, and the third measurement in the fourth bistatic pair. The composition of this secondary measurement can then be recorded as

C. Low-Precision Track Update
In Fig. 2, this part is marked as step C. The association between the low-precision tracks and measurements is carried out in the bistatic coordinates. We also formulate the association problem between tracks and measurements into a 2-D assignment problem and solve it using the Hungarian algorithm. Meanwhile, the index of the measurement associated with the track is recorded. A nonlinear filter is then used to update low-precision tracks [33].
We use l q (t) to indicate the number of low-precision tracks in the qth(q = 1, 2, · · · , N T −R ) bistatic pair at time t. There are a total of l(t) = N T −R q=1 l q (t) low-precision tracks in all bistatic pairs. We define a matrix L(t) with two rows and l(t) column: Each column in the first row of L(t) indicates the index of the bistatic pair where the low-precision track is located. The corresponding element in the second row denotes the index of the measurement associated with the track.

D. Low-Precision Track Conversion
In Fig. 2, this part is marked as step D1 and step D2. In step D1, we associate low-precision tracks with secondary measurements. It is conducted as follows. For each secondary measurement, e.g.,z β snd (t), we compare all the columns in the matrix L(t) with each column of T β (t). Record the number of identical columns as n β col . There are three cases for n β col .
1) If n β col > 1, it indicates that there are multiple lowprecision tracks associated with z β snd (t). We claim that these low-precision tracks and the secondary measurement correspond to the same target. 2) If n β col = 1, it indicates that only one low-precision track is associated with z β snd (t). To guarantee the reliability of the association, an extra judgment is added to further confirm whether the secondary measurement and the low-precision track correspond to the same target. Let x l and P low denote the state and covariance matrix of the low-precision track. We calculate the square of the following Mahalanobis distance and do the decision where μ is the decision threshold obtained by a given probability. The square of the Mahalanobis distance d 2 β approximately follows the chi-square distribution with degree-of-freedom2 × D, where D represents the spatial dimension, i.e., d 2 β ∼ χ 2 (2 × D). μ is the upper quantile of this chi-square distribution, usually calculated according to the 95% confidence interval. When d 2 β < μ, the secondary measurement z β snd (t) and the low-precision track correspond to the same target. Otherwise, we draw the contrary conclusion.
1) If n β col < 1, it indicates that there is no low-precision track associated with z β snd (t). In step D2, if there are low-precision tracks associated to secondary measurement z β snd (t), we further convert these lowprecision tracks into a high-precision track. In practical applications, the merged state obtained by track fusion method is usually bad due to the inaccuracy of low-precision tracks. In addition, there may be only a single low-precision track associated with the secondary measurement. On the other hand, note that the secondary measurement is quite accurate in the position and velocity. Thus, we directly use the secondary measurement z β snd (t) as the state of the first point of the high-precision track. The corresponding covariance matrix is P β snd . At the same time, terminate those n β col low-precision tracks. The above operations are repeated for all the secondary measurements.

E. Track Initialization
In Fig. 2, this part is marked as step E1 and step E2. After processing by the track maintenance module, those measurements that are not related to all the existing tracks, including bistatic measurements and secondary measurements, are considered as new targets or false alarms. We can then initialize new tracks and eliminate false alarms by standard algorithms (e.g., M/N logic-based track initialization (LBTI) algorithm [29]) for both high-precision track and low-precision track in both step E1 and E2. It should be noted that the high-precision track and low-precision track that are successfully initialized at this time may correspond to the same target. But in the subsequent cycles, the high-precision track will be preferentially associated with the bistatic measurements in step A, so that the low-precision track corresponding to the same target in step C will be terminated because there is no more measurement associated with it. This operation avoids the case where the high-precision track and the low-precision track correspond to the same target are both in the output.

IV. PERFORMANCE EVALUATION
In this section, we will verify the performance of the proposed algorithm in terms of the fast track confirmation, track continuity, and accuracy. Only the 2-D space is considered in the following discussion.

A. Probability of High-Precision Track Confirmation
This part will quantitatively analyze the performance of the proposed algorithm in terms of the fast track confirmation by theoretical derivation and simulations. The LBTI algorithm will be used as a reference. It directly uses the multiframe secondly measurements to initialize tracks.
We use the probability of high-precision track confirmation as an indicator to measure the performance of the fast track confirmation. It refers to the possibility that the track is confirmed successfully in the specified number of frames. For simplicity, we consider the simple case of the 2/2 LBTI method, where the track will be confirmed if the target is detected and associated in two consecutive frames. The algorithm proposed in [27] is used to obtain the secondary measurement. The association probability mentioned in step 2 of Section IV-B is denoted as P Ass . We set the same track-to-measurement association threshold in both high-precision tracks and low-precision tracks. A given probability, denoted by P Threshold , is used to obtain this track-to-measurement association threshold.
To calculate the high-precision track confirmation probability of the 2/2 LBTI method and the proposed algorithm, respectively, we define the following five events first.
Event A i : the target that is simultaneously detected by i(i ≥ 2) bistatic pairs. Event B k : the target is simultaneously detected by i bistatic pairs, but the number of bistatic pairs that make up the secondary measurement is k(2 ≤ k ≤ i).
Event C k : in the k bistatic pairs that make up the secondary measurement, at least one bistatic pair successfully initiates a low-precision track.
Event D i : the target is simultaneously detected by i bistatic pairs, and at least one of the bistatic pairs that constitute the secondary measurement has successfully initiated the low-precision track.
Event E i : the secondary measurement is not associated with the low-precision track, and it is not converted to a high-precision track.
Using P (A i ),P (B k ),P (C k ),P (D i ), and P (E i ) to indicate the probability of occurrence of events A i , B k , C k , D i , and E i , respectively.
Then, the probability P L defined in Section II can be expressed as follows: Assuming that each association hypothesis is independent, P (A i ) can be approximately calculated as follows: The probability of track confirmation using the 2/2 LBTI method can then be expressed as follows: Next, we calculate the probability of high-precision track confirmation of the proposed algorithm. The calculation is divided into three steps.

1) Probability of Converting Low-Precision Tracks Into High-Precision Track:
The measurements from k bistatic pairs that make up the secondary measurement involve a total of C 2 k association hypotheses. In fact, not all association hypotheses are independent of each other. Some association hypotheses that contain the same measurement from the same bistatic pair are correlated, which makes it difficult to accurately calculate the probability of event B. Hence, it is necessary to take some approximation to calculate the probability of event B. Here, we assume that all association hypotheses are approximately independent and these C 2 k association hypotheses can all pass the association hypothesis decision. Then, P (B k ) can be approximated as follows: The low-precision track confirmation is still used the 2/2 logic method. Thus, P (C k ) can be expressed as follows: In fact, the event D i indicates that the event B k and event C k occur at the same time. Therefore, we can calculate P (D i )as follows: We assume that event A i and event D i are approximately independent. The probabilityP (1) Init−high of converting low-precision tracks into high-precision track can then be approximated as follows:

2) Probability of High-Precision Track Conformation by Remaining Secondary Measurements:
The probability of occurrence of events E i is related to event B k and event C k . This relationship can be expressed as follows: Therefore, the 2/2 logic method, the probability P (2) Init−high of the remaining secondary measurement for the high-precision track conformation is  (9) with P Ass = 0.95 and P Threshold = 0.95. The number of bistatic pairs N from 2 to 4 and the detection probability P d from 0.7 to 0.9.  (16) with P Ass = 0.95 and P Threshold = 0.95. The number of bistatic pairs N from 2 to 4 and the detection probability P d from 0.7 to 0.9.

3) Total Probability of High-Precision Track Conformation:
After the above calculation, the Total probability P Init−high of high-precision track conformation is calculated as follows: In order to verify the correctness of the above theoretical derivation, 100 000 MC simulation results were used for comparison. Tables I and II give the theoretical and simulation results of the 2/2 LBTI algorithm and the proposed algorithm respectively. The theoretical calculation results in both tables are in good agreement with the simulation results. The higher probability means the better performance of the track confirmation. Comparing the two tables, we can find that the performance of the two algorithms is positive correlation to the number of bistatic pairs and the detection probability. The performance in terms of the fast track conformation of the proposed algorithm is significantly better than the LBTI method when the number of bistatic pairs is small and the detection probability is low.
It should be noted that the use of the M/N logic for the initialization of the high-precision and low-precision tracks herein is to facilitate the above quantitative analysis. It does not mean that we have to initialize the tracks using the M/N logic in the proposed algorithm. In theory, similar conclusions can be obtained when other track initialization approaches are applied to the proposed algorithm. The essential reason why the proposed algorithm performs better in terms of the fast track conformation lies in that the low-precision track retains additional information and can be converted into the high-precision track under certain conditions. The typical accuracies of the bistatic range, bistatic velocity, and azimuth are 40 m, 1 m/s, and 3°, respectively. Assume that all bistatic pairs have the same detection probability. False alarms per frame are generated by a Poisson process with a mean 50 and uniformly distributed in the detection area. The LBTI method is used to initialize both low-precision track and high-precision track. The period of measurements update is 1 s and the simulation duration is 100 s.

B. Track Continuity and Accuracy
The simulations mainly analyze the performance of the proposed algorithm in the case of insufficient bistatic pairs and low detection probability. There are three targets, as shown in Fig. 3. Considered two cases with N = 2, P d = 0.9, and N = 4, P d = 0.7. Here, the optimal sub-pattern assignment (OSPA) distance proposed in [30] is used as a performance measure. The advantage of the OSPA distance is that it not only calculates the position error of the track but also penalizes the false alarms or missed tracks. Here, the parameters in the OSPA distance are set as follows. The order of the norm is p = 2, the critical distance is c = 200. Two measurement fusion algorithms are used to characterize the tracking performance of the typical centralized estimation algorithm. One of the algorithms is the hypothesis decision (AHD) algorithm proposed in [27], the other is based on the M/N LBTI algorithm and Kalman filter (M/N-KF). The M/N-KF algorithm uses the M/N LBTI algorithm and a nonlinear Kalman filter to initialize new tracks and update the maintenance tracks, respectively. 100 MC simulations are performed.
The simulation results with three targets and N = 2,P d = 0.9 are shown in Figs. 3 and 4. The two bistatic pairs are T1-R1 and T2-R1. Fig. 3 shows one MC simulation tracking results of the AHD algorithm, M/N-KF algorithm, and proposed algorithm, respectively. Compared with the truth, the proposed algorithm can continuously track all target when the number of bistatic pairs is small. At this time, the tracks obtained by AHD algorithm are not continuous. Fig. 4 shows the OSPA distance and the average number of targets versus time. In Fig. 4(b), the average number of tracks of the three algorithms does not exceed the number of real targets, indicating that all these algorithms can perform well in clutter environment. The average number of tracks obtained by the AHD algorithm is significantly lower than the other two algorithms. The penalty in OSPA metric leads to far larger OSPA as shown in Fig. 4(a). The reason is that the target can be located only when it is detected by both bistatic pairs. When only one bistatic pair detects the target, the  AHD algorithm cannot update the track, causing the track to be interrupted. As indicated in Fig. 4(a), when the OSPA distance tends to be stable, the M/N-KF algorithm yields slightly larger results than the proposed algorithm. The main reason is that the proposed algorithm can quickly initialize the target track again after the target is interrupted. It should also be noted that the OSPA distance of the proposed algorithm converges faster than the AHD and M/N-KF algorithm, which indicates that the proposed algorithm has faster high-precision track confirmation ability. This phenomenon is consistent with the discussion in Section IV-A.
Next, we consider the case with three targets and N = 4, P d = 0.7. The four bistatic pairs are T1-R1, T1-R2, T2-R1, and T2-R2. The simulation results are plotted in Fig. 5. For the proposed algorithm, the average number of tracks basically matches the number of real targets and the OSPA distance also converges faster than the other two algorithms. Fig. 6 shows the root mean square error (RMSE) of the position and velocity of the proposed algorithm in the case of various number of bistatic pairs. Since we only care about the accuracy of the track, we set the detection probability and false alarms to 1 and 0, respectively. The RMSE is calculated as where S is the number of MC simulations, Δx s and Δy s represent the error in the x and y direction in the sth (s = 1, 2, · · · , S) simulation, respectively. The number of the bistatic pairs is 2 to 5. The results show that the RMSEs of the position and velocity converge to less than 10 m and 1 m/s, respectively. The more bistatic pairs, the lower RMSEs of the position and velocity.

V. FIELD EXPERIMENTAL RESULTS
The field experimental data, which was acquired in September 2018 by Wuhan University in the northwest of China, is employed to verify the practical feasibility of the proposed algorithm. The selected duration was about 5 min. A quadrotor drone with a diameter of about 0.5 m is used as the cooperative target. The data period is 1 s. The scenario of the MPR configuration is depicted in Fig. 7. There are two transmitters and one receiver. The transmitters Tx1 and Tx2 transmit digital television terrestrial multimedia broadcasting signal in 626 and 746 MHz,   respectively. The receiver Rx receives the signals from both the transmitters Tx1 and Tx2. The position information of the drone recorded by the global positioning system (GPS) is used as the reference. Fig. 8(a) shows the track results of the first bistatic pair. Apart from the target tracks, there are many other tracks, which may be originated from birds, cars, and even other noncooperative drones. Regarding the target track, it is not continuous. Compared with the GPS, the track error is relatively large. Fig. 8(b) shows the processing results of the other bistatic pair. This result is slightly better than Fig. 8(a) in the track continuity and accuracy. Fig. 9 shows the fusion results of the two bistatic pairs. The results of all the three algorithms are given here. We only focus on the track results of the cooperative target. Fig. 9(a) shows the tracking result of the AHD algorithm. The performance of the AHD algorithm is worse than the other two algorithms in the track continuity. Fig. 9(b) and (c) show the tracking results of the M/N-KF algorithm and proposed algorithm, respectively. Compared with GPS, both algorithms seem to perform well in terms of the track continuity and accuracy. Note that the track discontinuity marked by green circle in Fig. 8(b) is caused by the hovering behavior of the drone. The proposed algorithm can initialize the target track faster when the drone starts moving again. We use the target's position recorded by the GPS as the

VI. CONCLUSION
We proposed a hybrid tracking algorithm for the MPR in this article. In the MPR, the existing tracking algorithms face problems in the fast track confirmation, track continuity, and accuracy in the case of insufficient bistatic pairs and low detection probability. We construct the high-precision track and low-precision track to handle these problems. Theoretical analyses show that even if the number of bistatic pairs is only set to 2, the proposed algorithm can still significantly increase the probability of high-precision track confirmation compared with the LBTI algorithm. It is also shows that the proposed algorithm performs well in terms of the track continuity and accuracy in clutter environment. The practicability of the proposed algorithm is also validated using real data. With the development of the MPR, asynchronous sensor fusion is imperative. However, most of the existing measurement fusion methods are based on time synchronization. Our future work is to develop an asynchronous sensor fusion strategy based on measurement fusion method. Jianxin Yi (Member, IEEE) received the B.E. degree in electrical and electronic engineering, and the Ph.D. degree in radio physics from Wuhan University, Wuhan, China, in 2011 and 2016, respectively, and the Ph.D. degree from the University of Calgary, Calgary, AB, Canada.
He is currently an Assistant Professor with the School of Electronic Information, Wuhan University. He has been supported by the Postdoctoral Innovation Talent Support Program of China. His main research interests include radar signal processing, target tracking, and information fusion.
Dr. Yi was the recipient of the 2017 Excellent Doctoral Dissertation Award from the Chinese Institute of Electronics.
Xianrong Wan received the B.E. degree from the former Wuhan Technical University of Surveying and Mapping, Wuhan, China, in 1997, and the Ph.D. degree from Wuhan University, Wuhan, China, in 2005.
He is currently a Professor and the Ph.D. Candidate Supervisor with the School of Electronic Information, Wuhan University. In recent years, he has hosted and participated in more than ten national research projects, and authored or coauthored more than 80 academic paper. His main research interests include design of new radar system such as passive radar, over-the-horizon radar, and array signal processing.
Feng Cheng received the B.E. degree in electrical and electronic engineering, and the Ph.D. degree in radio physics, from Wuhan University, Wuhan, China, in 1998 and 2006, respectively.
He is currently an Associate Professor with Wuhan University. His main research interests include radar signal processing, radio ocean remote sensing, and radar software engineering.