Automated Maxillofacial Segmentation in Panoramic Dental X-Ray Images Using an Efficient Encoder-Decoder Network

The panoramic dental X-ray images are an essential diagnostic tool used by dentists to detect the symptoms in an early stage and develop appropriate treatment plans. In recent years, deep learning methods have been applied to achieve tooth segmentation of dental X-rays, which aims to assist dentists in making clinical decisions. Because the original images contain plenty of useless information, it is necessary to extract the region-of-interest (ROI) to obtain more accurate results by focusing on the maxillofacial region. However, a fast and accurate maxillofacial segmentation without hand-crafted features is challenging due to the poor image quality. In this study, we create a large maxillofacial dataset and propose an efficient encoder-decoder network model named EED-Net to solve this problem. This dataset consists of 2602 panoramic dental X-ray images and corresponding segmentation masks annotated by the trained experts. Based on the original structure of U-Net, our model structure contains three major modules: a feature encoder, a corresponding decoder, and a multipath feature extractor that connects the encoding path and the decoding path. In order to obtain more semantic features from the depth and breadth, we replace the convolution layer with the residual block in the encoder and adopt Inception-ResNet block in the multipath feature extractor. Inspired by the skip connection in FCN-8s, the lightweight decoder has the same channel dimension as the number of segmented objects. Besides, a weighted loss function is used to enhance segmentation accuracy. The comprehensive experimental results on the new dataset demonstrate that our model achieves better accuracy and speed trade-offs for maxillofacial segmentation than the latest methods.

I. INTRODUCTION
Dental radiographs are widely used in dentistry for clinical diagnosis, treatment, and surgery. These images enable dentists to find hidden dental structures, bone loss, and cavities [1], which are hard or impossible to detect in visual inspection. Hence, dentists can detect symptoms at an early stage and develop appropriate treatment plans [2]. In the dental examination, there are three types of conventional radiographs: bitewing, periapical, and panoramic [3]. The bitewing X-rays show portions of the upper and lower teeth to The associate editor coordinating the review of this manuscript and approving it for publication was Kumaradevan Punithakumar . detect changes in bone thickness caused by gum disease. The periapical X-rays show the whole teeth in part of the upper jaw or lower jaw to evaluate the root area and surrounding bone structures. While the bitewing and periapical X-rays focus on the details of individual teeth or parts of teeth, the panoramic X-rays capture the entire mouth area, including all the teeth, gums, jaws, and bone structure [4] to provide more diagnostic evidence. Since the panoramic X-rays are filmed outside the mouth, they have better patient acceptance with faster shooting, less radiation exposure, and lower infection rate [5]. During diagnosis and treatment, such as root canal treatment, caries diagnosis and tooth orthodontics, dentists are required to analyze panoramic radiographs and record specific symptoms of diseased teeth in the electronic medical record. However, it takes a long training time for a young doctor to read dental films accurately [6]. Therefore, considerable attention has been paid to automatic panoramic X-rays analysis.
Many studies have been carried out to explore tooth segmentation since the diagnosis is based on the analysis of teeth and surrounding tissues. Traditional image segmentation methods [7]- [10] can quickly extract dental information from a single radiograph type. However, these methods rely on well-designed manual features and lack sufficient generalization capability. The emergence of deep learning methods [11], [12] dramatically improves the accuracy of tooth segmentation under different tooth distributions. The image segmentation models based on deep learning mostly adopt an end-to-end learning framework [13], where both training and inference are performed by dense feedforward computation and backpropagation to learn the entire image at once. The training process is designed to optimize the overall parameters according to the error between predicted results and true labels. As the deep learning models need to focus more on teeth and surrounding areas to enhance pixel recognition, their typical inputs are often the regionof-interest (ROI) derived from the original image [14], [15]. In panoramic X-rays, the ROI is the maxillofacial region [16], [17], which contains all the teeth and discards most of the irrelevant information. Therefore, real-time extraction of the maxillofacial region is an essential preprocessing step in the panoramic dental X-ray analysis. FIGURE 1. An example of the panoramic dental X-rays. The inner area of the red circle is the approximate outline of maxillofacial region that contains all the teeth. The pixels and illumination of the image are not uniformly distributed, and the high-noise outer area accounts for more than half of the pixels. Due to low-contrast and overlapping anatomical structures, the maxillofacial region has no strong edges.
However, an accurate and rapid maxillofacial segmentation is challenging due to the poor image quality of panoramic dental X-rays. As shown in Fig. 1, the low quality of the image comes from three aspects [18]: 1) unavoidable noise, 2) varying illumination, and 3) low pixel contrast of the tissue. A major dilemma is that panoramic radiographs show overlapping anatomical structures [19]. The overlap among teeth, jaws and surrounding bones then causes complex variations in grayscale levels of panoramic X-rays. In this case, static and dynamic methods have been developed to extract the ROI from panoramic X-rays. The static method defines a fixed rectangular window for all images. Through a statistical analysis of the maxillofacial morphology [20], [21], the anchor point (often the image center) and the rectangle size are determined, and this window is cropped from the image as the maxillofacial region. This method is theoretically effective for samples with similar tissue morphology. Nevertheless, the shape and position of the maxillofacial region are considerably variable in practice, which limits the application of the static window. The dynamic method employs manual threshold selection [22], morphological transformation [23], or wavelet variation [24] to extract the separation line of the upper and lower jaws for correcting the anchor point. Then, the modified anchor point and rectangular, trapezoidal [25] or oval window [26] are combined to capture the maxillofacial region. Although this method improves the segmentation accuracy and has higher flexibility to some extent, it requires a fine selection of pixel thresholds and fails to accurately locate the separation line when teeth are missing or overlapping. To circumvent the limitations of the above methods, an approach with greater feature extraction and generalization capability is needed. Fortunately, the deep learning approach [27]- [30] is competent for this task because of its remarkable success in medical image segmentation [31], [32]. However, the performance gain of this approach usually comes at the cost of high computation [33] and long processing time. Therefore, ensuring both the accuracy and the real-time performance of maxillofacial segmentation still remains a significant challenge.
In this study, we propose an efficient encoder and decoder network named EED-Net to address automatic maxillofacial segmentation using a newly built dataset. Since there is no open maxillofacial dataset in relevant research fields, we elaborately collect 2602 panoramic dental X-rays, excluding the hypoplasia and tooth decay. A group of dental experts are trained to implement the maxillofacial annotations. On this basis, we evaluate the performance of FCN-8s [34] and U-Net [35], which are commonly used as the baseline models in image segmentation. Despite that both models have their own advantages in aspect of speed and accuracy, they cannot balance these two performances. Against this background, we combine the structure of U-Net with the decoding method of FCN-8s to design a new maxillofacial segmentation model. The proposed model is mainly composed of three modules: a feature encoder, a corresponding decoder, and a multipath feature extractor that connects the encoder and the decoder in the last layer. Specifically, the common convolution connection in the encoder is replaced with a residual structure [36] to obtain deeper features. Besides, we construct a modified Inception-ResNet block [37] in the multipath feature extractor to increase the width of the feature search. The skip connection between the encoder-decoder pairs is reserved for passing the low-resolution texture and location information.
To reduce parameters, we employ the category number of segmented objects to identify the decoder's channel dimension, VOLUME 8, 2020 which is far less than the encoder. The extensive experiment results on the maxillofacial dataset demonstrate that our proposed model achieves better overall performance than the latest real-time semantic segmentation models, which verifies the efficiency of EED-Net.
The remainder of this paper is organized as follows. Section II introduces the related works on the maxillofacial segmentation methods and some latest deep learning semantic segmentation models. Section III describes the new dataset and the proposed model in detail. Section IV presents the experimental results. In Section V, our conclusion and discussion are provided.

II. RELATED WORKS
In the literature of maxillofacial segmentation methods for panoramic dental X-ray images, two strategies have been developed: 1) static method: observing the distribution of teeth and surrounding tissues in the image samples and artificially selecting a static window to crop the maxillofacial region; 2) dynamic method: enhancing the contrast of the original image, extracting the separation line of upper and lower jaws, and correcting the position and shape of the dynamic window.
To implement the first strategy, Oliveira and Proença [20] defined a rough maxillofacial region that took out the nasal and chin bones as the first stage of teeth segmentation. They measured the boundary distance of the maxillofacial region and fitted each distance variable to a corresponding Gaussian distribution. The shortest distance with 95% certainty was adopted as the window size to extract the ROI. According to the statistical analysis of image morphology, Fares and Feghali [21] positioned the anchor point at the maxillofacial center to reduce the distance variables.
To improve the adaptability and accuracy of segmentation algorithms, the dynamic methods were proposed. Jain and Chen [22] used transverse scanning to find the lower pixel intensity in the y-axis projection histogram. Combined with the Gaussian distribution assumption and a user-assisted initialization, this method found the gap valley between the upper and lower teeth. By locating these gaps, they divided the maxillofacial region into vertical strips for subsequent tooth segmentation. Wanat and Frejlichowski [26] utilized the areas between necks of teeth to determine the separation line, which did not depend on the gap between adjacent teeth. Harandi et al. [23] applied morphology transform and modified geodesic active contour on panoramic X-rays to achieve better separation and subtraction. Gumus [24] employed the discrete wavelet transform for better location of the ROI and adopted polynomial regression to form a smooth separation line with absent teeth.
The above-mentioned dynamic methods aim to separate the upper and lower jaws, whereas the deep learning method requires a complete maxillofacial image to achieve the training and inference of end-to-end models. Fares and Feghali [21] searched for the occlusal gap to locate the maxillofacial center and extracted the entire ROI with a fixed rectangle. However, their method cannot find the accurate maxillofacial region when the upper and lower jaws are connected. Hasan et al. [25] combined the gradient vector flow(GVF) snakes with K-means clustering to accomplish maxillofacial segmentation with missing gaps between upper and lower jaws. Although this method has been proved to work with more types of panoramic radiographs, it is highly dependent on manual threshold selection to extract the maxillofacial edges.
In recent years, significant progress has been made to improve the performance of low-contrast medical image segmentation using deep learning algorithms. Among these works, FCN [34] and U-Net [35] are the popular classical models. FCN lays the foundation for most modern segmentation architectures. As shown in Fig. 2(a), the removal of the full connected layer allows the model to predict the dense output from an arbitrary-sized image theoretically. Specifically, VGG [38] is employed as the encoder to extract highresolution semantic features. For the decoder, the transposed convolution and skip connection are integrated to restore the object spatial. In this context, three versions of the model: FCN-32s, FCN-16s, and FCN-8s are presented with different depths of the connection architecture. Since the frequent use of pooling layers in FCN results in a loss of low-resolution features, U-Net builds dense skip connections to further explore the recovery of image details. Fig. 2(b) shows that U-Net has a symmetrical encoder-decoder structure. Due to the concentration operation for multi-scale feature integration, low-resolution context and high-resolution features can be fused without loss. As a result, the abundant feature information effectively improves model performance at the cost of increased computation.
More recently, many lightweight models have been proposed for real-time semantic segmentation. RefineNet [39] constructed an encoder-decoder network in the full residual form [36] and leveraged multiple-level abstract features to perform high-resolution semantic segmentation. The multi-branch systems ICNet [40] and BiSeNet [41] learned a global context with reduced-resolution input in a deep branch, while boundaries were learned in a shallow branch at full resolution. Fast-SCNN [42] merged the two-branch pattern with the encoder-decoder framework and introduced a novel initial layer to share the computations of both branches. LEDNet [43] proposed an asymmetric encoderencoder architecture with channel split and attention pyramid to lighten the model complexity.
However, real-time maxillofacial segmentation with high accuracy remains challenging. Inspired by the above works, the proposed EED-Net incorporates the multipath feature extractor and the simplified decoder in the residual encoderdecoder framework to guarantee both speed and accuracy of maxillofacial segmentation.

III. METHODOLOGY
In this section, we describe our research methodology in detail from the following aspects: the establishment of the dataset, the structure of EED-Net, and the weighted loss function.

A. DATASET 1) THE SOURCE AND ATTRIBUTES OF THE IMAGES
In the early stage of our research, we found it difficult to find a maxillofacial segmentation dataset in the community. The existing open dental datasets contain no maxillofacial results, and their quantity is not enough to support the training of deep learning models. Therefore, we cooperate with the experts from the Hospital of Stomatology Wuhan University to build a maxillofacial dataset. To gather data quickly, we choose to screen panoramic dental X-rays from electronic medical records rather than waiting for new patients. It is worth noting that these images do not contain any personal privacy, so there is no ethical issue involved. Considering the maxillofacial growth and deformity caused by age, the experts remove the samples that are too young or too old. Moreover, every image is labeled ''non-periodontal disease'' or ''periodontal disease'' for the follow-up research. In total, our dataset is composed of 2602 panoramic X-rays, including 1146 ''nonperiodontal disease'' samples and 1456 ''periodontal disease'' samples.

2) THE ANNOTATION PRINCIPLES OF THE MAXILLOFACIAL REGION
In medicine, the maxillofacial region is located at the beginning of the digestive tract and respiratory tract to communicate with the outside world through the oral cavity and nasal cavity. This description aims to delineate the specific functional area, but our research is designed for maxillofacial segmentation. Therefore, a clear maxillofacial definition is required to accommodate the panoramic X-rays. After analyzing the correlation between teeth and maxillofacial region in different types of the images, we develop the following three principles for maxillofacial annotation: a) the segmented region should contain all the teeth; b) there should be a significant difference between the segmented region and the unsegmented region; c) The boundary of the segmented region should be as smooth as possible. According to these rules, the new maxillofacial region is precisely outlined between the nose and chin, forming a smooth surface with the left and right jaws.

3) THE ANNOTATIONS OF THE PANORAMIC DENTAL X-RAYS
The maxillofacial annotations are achieved by anchoring and connecting the points in the images. In general, 30-50 points are required in an image to outline the target region, which makes the annotations labor-intensive and time-consuming. To improve annotation efficiency, we test some common annotation methods and finally choose VGG Image Annotator (VIA) [44]. This tool is easy-to-use and compatible with different operating systems, which is more suitable for collaborative annotations than other tools. Based on the proposed segmentation principles, our experts are trained to outline the maxillofacial region in the image using VIA, where the annotation files are obtained in JSON form. By converting the files into images in python, the maxillofacial masks are generated in grayscale. The segmentation results are verified by our experts and unqualified images are relabeled. Finally, we establish a dataset of 2602 panoramic X-rays and corresponding maxillofacial masks for deep learning models. Fig. 3 shows the typical images and their masks.  For the performance assessment of the models, we divide the whole dataset into a training set and a test set with a ratio of 4:1. Table 1 describes the distribution of different . The structure of EED-Net. In addition to the initial convolution layers and the last sigmoid activation layer, the model is mainly composed of three parts: 1) the residual encoder modules, 2) the multipath feature extractors based on the simplified Inception-ResNet block, 3) the object-oriented decoder modules. The main path realizes the breadth extraction of the encoder output features, and passes high-resolution features to the decoder after channel compression. In order to recover image details accurately, the dense short paths are utilized to achieve the gradient flow in shallow layers. Since there is only one target type in maxillofacial segmentation, the corresponding decoder has a single feature channel, converted from a 1 × 1 convolution layer.
categories of images in the dateset. In order to maintain category balance, both types of samples are randomly assigned to the homologous datasets in the same ratio of 4:1.
Since the original panoramic X-rays have a resolution of 2976 × 1536 pixels, our GPU will run out of memory during the calculation of such a large image. Therefore, we resize them to a resolution of 744 × 384 (which is a quarter of the original size) by the nearest interpolation. This dataset serves the purpose of supporting the verification and optimization of maxillofacial segmentation models.

B. EED-NET
Before developing our model, we evaluated the common segmentation models, FCN-8s and U-Net, on our maxillofacial dataset. The comparison results indicate that U-Net has higher accuracy, while FCN-8s uses less computing time with more parameters. It is worthwhile to design a new network model to inherit the advantages of these two baseline models. In this case, both accuracy and efficiency can be guaranteed.
Based on the structural analysis of existing models, we propose a novel efficient encoder-decoder network named EED-Net for maxillofacial segmentation. As illustrated in Fig. 4, the main architecture of EED-Net consists of three components: an encoder module, a decoder module, and a multipath feature extractor module. Before entering the serial encoder modules, a 3 × 3 convolution layer is used for the preliminary feature extraction and channel transformation of the input image. The encoder module adopts the residual structure to obtain effective high-resolution features and speed up the model calculation. In the encoding path, its channel increases by multiples to compensate for the information loss caused by maxpooling layers. Then, two concatenated multipath feature extractor modules (which we call the main path) are utilized to connect the encoding path and the decoding path. Since high-resolution features have a decisive influence on the results, the multiple feature extractor is constructed on the simplified Inception-ResNet structure. Due to the combination of receptive fields with different widths, the main path can obtain multi-dimensional deep features to enhance the discriminative capacity of the segmentation target. Next, our decoder module realizes the addition of input feature maps after the channel transformation using a 1 × 1 convolution layer. Most importantly, the number of segmented objects, not the encoder, determines the channel number in the decoders. Since each panoramic X-ray has only one maxillofacial region, the object-oriented decoders dramatically reduce the model complexity. Moreover, skip connections (which we call the short path) between the encoder-decoder pairs are reserved to achieve the gradient flow in shallow layers, allowing the training of a very deep network. Finally, the segmentation results are obtained through the last sigmoid activation layer.

1) FEATURE ENCODER MODULE
The primary function of the encoder is to gradually reduce the spatial dimension of feature maps and capture more high-level semantic features. Since the proposed model is deeper than U-Net, we replace the ordinary convolution block with the residual block to avoid gradient degradation. Fig. 5 indicates the encoder structures of these two models. Both encoders are composed of two 3 × 3 convolution layers and activation layers. The activation layer contains a batch normalization layer [45] and a rectified linear unit (BN and ReLU). Compared with U-Net, our residual encoder has a shortcut connection and an addition operation that enable the model to learn residual features rather than entire features, which is easier to optimize. By learning existing various residual structures, we choose the pre-activation residual block [46] with activation layers in front of the weight layers to accelerate the training and improve the generalization ability. Besides, the convolution layers within the encoder have the same channel number. Since there is a maxpooling layer between adjacent encoders to reduce the feature map, the channel number of the next encoder is twice that of the previous encoder to compensate for information loss.

2) MULTIPATH FEATURE EXTRACTOR
After passing through five encoders, the input feature map of the main path is 1/32 of the original image size. However, the high-resolution information plays a decisive role in the restoration of the segmented region. Since the excessive increase of convolution kernel and channel number will bring too many parameters, we construct a modified Inception-ResNet block to obtain deep features under multiple receptive fields. As shown in Fig. 6, the extractor has three separate paths, whose convolution kernels are 1 × 1, 3 × 3, and 5 × 5. The 5 × 5 path is composed of two 3 × 3 convolution layers. The 1 × 1 convolution layers are used to adjust the input dimension to reduce the extra computation. In addition, the original Inception-ResNet structure is converted to the pre-activation form to keep consistent with the encoder. Then we use a concatenation operation to combine the multipath outputs with the same dimensions. Meanwhile, the feature concatenation causes no information loss. Finally, we use two 192-channel modules instead of a 384-channel module to build the main path.

3) FEATURE DECODER MODULE
In the maxillofacial segmentation experiments, we find that U-Net is more accurate and FCN-8s is faster with some performance loss. Based on the analysis of two models, it can be inferred that the symmetric decoders of U-Net bring plenty of information redundancy, which slows down the calculation. To reduce the complexity, we refer to FCN-8s and adopt 1 × 1 convolution layers to recover the object details and spatial dimensions before information fusion. In Fig. 7, our decoder has no common weighted layers, and an addition operation completes the feature combination of the two paths. Unlike the single-channel decoder in FCN-8s, the channel VOLUME 8, 2020 number in our decoder is determined by the number of the segmented objects to enhance the segmentation quality. Since each panoramic dental X-ray has only one target maxillofacial region, the object-oriented decoder is much lighter than U-Net. The activation layer is placed at the end of the decoder to ensure the pre-activated form.

C. WEIGHTED LOSS FUNCTION
As an end-to-end model, EED-Net is trained to predict whether each pixel belongs to the foreground or the background to extract the target region. Before the training, it is necessary to specify the task type and expected performance for choosing the loss function. Since regression functions are mainly used to predict dynamic variables, maxillofacial segmentation should be more a pixel classification problem than a regression problem. In general, the entropy loss function [47] is applied to train the multiple classification models by calculating the loss value of each pixel type. Considering that there are only two types of pixels in our task, we adopt the binary entropy loss function to estimate each pixel category. Equation (1) describes the loss calculation of a single image: where y is the ground truth of the input image and y i ∈ {0, 1}, p is the predicted values of the pixel classification and p i ∈ [0, 1], and N denotes the total number of pixels. The final loss value L bce is obtained by calculating the average loss of all pixels. Actually, maxillofacial segmentation is not only a pixel classification problem, but also a region segmentation problem, that takes a connected whole in the image as the target area. Moreover, the small difference in pixel intensity between the bone and tooth structures on panoramic radiographs increases the difficulty in recognizing the maxillofacial edges. Considering the boundary connectivity, we adopt the dice coefficient loss [48] to focus on the overlap between the ground truth and the predicted maps. In image segmentation, the ground truth and prediction can be viewed as two sets, and the dice coefficient is used to gauge their similarity. In this manner, the predicted map approaches the ground truth by minimizing the dice coefficient loss in (2).
where N i y i p i denotes the intersection of the ground truth y and the predicted results p, N i y 2 i and N i p 2 i are the quadratic sums of the pixel category of two samples. We set ω = 1 in this study, and L dice will come to zero when the predicted result is the same as the ground truth.
To balance the performance of EED-Net in pixel prediction and overall segmentation, we adopt the weighted loss function composed of binary entropy loss and dice coefficient loss. The final loss function is defined as where α ∈ [0, 1] is the weighted coefficient. This weighted loss function can be transformed into a single loss function by setting α = 0 or α = 1. The integrality of segmentation is emphasized with a large α. In our experiments, we set α = 0.8 to get better performance.

IV. EXPERIMENTS AND RESULTS
In this section, the extensive experiments are conducted to validate the effectiveness of our approach. Firstly, the experimental environments and the implementation details of model training are introduced. Then, the evaluation metrics are presented to quantify the performance of the models in various aspects. Finally, the proposed model is optimized from the structure and loss function, and compared with other methods.

A. EXPERIMENT ENVIRONMENTS AND IMPLEMENTATION DETAILS
The training and testing are implemented on a desktop server with an Inter(R) i9-9900K CPU, 64 GB memory, and two NVIDIA GeForce RTX 2080 Ti graphics cards. These GPUs are configured with CUDA 10.0 and cuDNN 7.6 to realize the fast parallel computation of convolutions. The software environment is built on the Keras 2.3.1 with Tensorflow 1.15.0 in Python language. In implementation, the pixel values of the input image are normalized to 0 to 1 before entering the model. The convolution kernel parameters are initialized using Tikhonov regularization method (more often known as L2 regularization). Moreover, an Adam optimizer [49] is employed to minimize the loss value with a multi-staged learning rate. The training epoch is 100, and the batch input size is 4. During the training, the learning rate is 5e − 4 in 1 to 10 epochs, 2e − 4 in 11 to 50 epochs, and 1e − 4 in 51 to 100 epochs.

B. EVALUATION METRICS
In order to make a comprehensive evaluation, it is necessary to measure model performance in terms of segmentation integrity. To this end, we add some widely accepted objective criteria to evaluate these segmentation methods.

1) JACCARD SIMILARITY COEFFICIENT
Similar to the dice coefficient, the Jaccard index (also referred to as IoU) is used to measure the similarity of two sets. Defined as the size of the intersection divided by the size of the union, the Jaccard index can quantify the overlap between the predicted map and the ground truth. In our experiments, its value ranges from 0 to 1, and a larger value indicates a better segmentation result. The Jaccard index is defined as: where X denotes the image array composed of the approximation of predicted results, and Y represents the image array of the ground truth. The elements in both X and Y are either 0 or 1. In practice, the intersection and the union are realized by logical matrix operations.

2) HAUSDORFF DISTANCE
The hausdorff distance (HD) [50] is widely employed to evaluate medical image segmentation methods as an indicator to measure the maximum boundary of the segmentation surface.
Since maxillofacial segmentation is a preprocessing step of tooth analysis in panoramic radiographs, HD provides a good measure of the usefulness of the segmentation results for the intended task. For overlapping binary images, HD can be obtained by calculating the distances from each point in one image to the nearest point in the other image, and then taking the largest distance. The bidirectional HD between two images is defined as follows: where X denotes the binary predicted map and Y denotes the ground truth. We employ Euclidean distance to compute the distance between the pixels of the same class. While hd(X , Y ) and hd(Y , X ) are the one-sided HD between the binary images, the maximum HD(X , Y ) represents the longest segmentation error.

C. EXPERIMENTS
To validate the effectiveness of the proposed model, we design a series of comparative experiments on our maxillofacial dataset. We first explore the performance of EED-Net with different depths and investigate the impact of the weighted loss function on segmentation accuracy, then demonstrate the differences between the proposed model and the baseline models by visualizing the predicted maps, and finally confirm the performance in comparison with the latest real-time segmentation models.

1) EVALUATION OF MODEL DEPTH
Based on the flexible structure, we construct the corresponding models according to the number of encode-decoder pairs. These models have the same multipath extractor. Using the same binary cross entropy (BCE) loss and gradient learning rate, we test four different versions of the network: EED-Net-3, EED-Net-4, EED-Net-5, and EED-Net-6.
To obtain a comprehensive evaluation, the frame per second (fps) is augmented to describe the real-time performance of the models. In particular, the output rate measurement is performed on the whole dataset with all 2602 images to enhance the credibility. The overall performance of EED-Nets is presented in Table 2. As the network gets deeper, the spatial and time complexity of EED-Net gradually increases, indicating that the model parameters are mainly derived from the bottom encoder-decoder pairs. Among these models, EED-Net-3 with the fewest parameters achieves  the highest output rate of 42.3 fps. In contrast, EED-Net-3 has the lowest segmentation accuracy, where the jaccard is 0.9790 and the hausdorff distance is 16.31. In general, the segmentation accuracy of the underfitting model benefits from additional parameters. However, the experimental results in Table 2 show that EED-Net-5 instead of EED-Net-6 yields the highest accuracy. As shown in Fig. 8, when the model depth exceeds three layers, the convergence trend of the loss values is roughly similar. For EED-Net-6, the increased parameters bring no improvement in segmentation accuracy, but lead to slight overfitting. Due to the small difference for EED-Nets in output rate, we can infer that a deeper network may not lead to better performance. Since both segmentation accuracy and speed must be considered in maxillofacial segmentation, the most accurate EED-Net-5 is selected for subsequent optimization.

2) EVALUATION OF THE LOSS FUNCTIONS
To verify the validity of the weighted loss function, we train EED-Net-5 with the other two classical losses under the same conditions. Specifically, the model performance of three typical weighted coefficients α are reported in Table 3. Besides, the loss curves of EED-Net-5 with different configurations are plotted in Fig. 9, where the weighted loss curves are located between the BCE curve and the Dice curve. With the increase of α, the curve gets closer to Dice, otherwise, it rises near BCE, which conforms to its definition. Table 3 shows that the weighted loss has a higher segmentation accuracy  than the classical losses, especially in the hausdorff distance. However, the performance improvement of the weighted coefficient is nonlinear. According to the experimental results of the typical weights, the boundary weights bring greater performance gain than the middle weights, approximating a U-shaped distribution. Compared with BCE and Dice, the model receives the biggest boost with a α of 0.8, where the jaccard increases by 0.15% and 0.14%, and the hausdorff distance reduces by 14.14% and 8.87%. As a result, this version of EED-Net-5 is chosen as our final model.

3) COMPARISON WITH THE BASELINE MODELS
Since the proposed model is built on the basis of FCN-8s and U-Net, it is necessary to analyze the changes brought about by our adjustments. Fig. 11 displays the visual segmentation results of FCN-8s, U-Net, and EED-Net-5. The corresponding accuracy values are presented in Table 4. The input images are the representative dental X-rays selected from the test images. On the whole, all three models are able to extract the basic contours of the maxillofacial region. However, there are significant differences in the details of the contour edge. FCN-8s has continuous pixel recognition errors in the maxillofacial edges, resulting in a jagged boundary. Compared with FCN-8s, the boundaries of U-Net are more distinct and smoother. Based on the structural analysis, TABLE 4. The accuracy performance of EED-Net-5 and the baseline models on the specified images. Image1 to Image5 are the test samples listed from top to bottom in X-ray column of Fig. 11. we consider that the encoder-decoder network form contributes to the fine-grained segmentation. However, there are still some regional pixel recognition errors around the edges, which looks like some bubbles. As for EED-Net-5, the residual encoders and the multipath feature extraction enable it to obtain a higher pixel accuracy and regional connectivity. Therefore, the results of EED-Net-5 are closer to the ground truth with smoother boundaries and fewer bubbles.
In addition to the recovery of image details, we illustrate the speed performance of the three models in Fig. 10. The concatenation operation of U-Net brings high computation while obtaining fine-grained segmentation, resulting in the longest training time and lowest output rate. The output form of FCN-8s enables it to lead U-Net 10 fps in speed with decreased accuracy. Due to the combination of channel pruning and decoder simplification, EED-Net-5 reduces the epoch time by 37.7% from 67.6 to 42.1 and increases the output rate by 79.1% from 22.9 fps to 41.0 fps compared to FCN-8s. In summary, the proposed model achieves significant improvements in accuracy and speed over the baseline models.

4) COMPARISON WITH THE LATEST METHODS
To further confirm the effectiveness of the proposed model, we design comparison experiments with the latest realtime segmentation models on the maxillofacial dataset. The weighted loss and Adam optimizer with stepped learning rate are employed for all models. It is worth noting that the model performance and properties may be slightly different from the original method because we have migrated the code of authors to our environment. From Table 5, it can be seen that except for the classical models, RefineNet and Fast-SCNN are representative of the other models. RefineNet achieves 0.9926, 0.9827, and 8.93 in the accuracy, the jaccard, and the hausdorff distance respectively, but this improvement is derived from the increase of model parameters, and its segmentation speed is unsatisfactory. Fast-SCNN achieves the output rate of 66.2 fps, but this real-time performance comes at the expense of segmentation delicacy, which is detrimental to subsequent image processing. The performance of other models on maxillofacial segmentation falls between these two models, and they fail to find a balance between speed and accuracy.
The results in Table 5 indicate that our model is a better implementation of maxillofacial segmentation. The proposed EED-Net-5 achieves 0.9928 in the accuracy, 0.9829 in the jaccard and 8.32 in the hausdorff distance with the fewest parameters, which outperforms the other methods. In the aspect of segmentation speed, our model can maintain an output rate of 41.0 fps, which meets the requirements of real-time segmentation. Since the environmental configuration of the models is identical, the highest accuracy and the increased speed further validate the efficiency of our network architecture. This is mainly due to the optimization of the encoder-decoder structure and the addition of the multipath feature extractor, which enables our model to focus on image details with fewer parameters. Considering both accuracy and speed performance, the conclusion can be reached that the proposed model achieves better overall performance than the other real-time segmentation methods for maxillofacial segmentation.

V. CONCLUSION AND DISCUSSION
An accurate and automatic extraction of the ROI is essential for the researches of panoramic dental X-rays. In this study, we establish a maxillofacial dataset and propose an efficient encoder and decoder network model named EED-Net to achieve maxillofacial segmentation. The dataset consists of 2602 panoramic X-ray images and the corresponding maxillofacial mask results. Based on the structure of U-Net and the decoding form of FCN-8s, the proposed EED-Net is composed of the residual encoders, the multipath feature extractors, and the object-oriented decoders. The encoders and extractors are used to capture deep features, and the channel number of the decoders is simplified to reduce the parameters. Moreover, we adopt a weighted loss function to further improve segmentation accuracy. The extensive experiments on the dataset demonstrate that EED-Net outperforms other methods in segmentation accuracy. In addition, our model can output an average of 41.0 fps segmented images per second, which is highly feasible to realize the preprocessing of panoramic X-rays. Based on the extensibility of the baseline models, it can be inferred that our method can be applied to the X-ray images with a single target type. For the multi-class segmentation, the effectiveness of the proposed method requires further study. Since we have to compress the original images to avoid insufficient computer memory in this work, how to directly extract ROI using high-resolution images also deserves further exploration.
ZHUOLIN FU received the B.Eng. degree from the Department of Artificial Intelligence and Automation, Wuhan University, Wuhan, China, in 2018, where he is currently pursuing the master's degree with the School of Electrical Engineering and Automation. His research interests include software development, few-shot learning, and artificial intelligence.
MAOQI ZHANG received the bachelor's degree in dental medicine from the School of Stomatology, Nanchang University, Nanchang, China, in 2019. She is currently pursuing the master's degree in periodontal science with the School of Stomatology, Wuhan University. Her main research interests include medical services, medical diagnostic imaging, medical diagnosis, medical treatment, and medical tests.
JINGXIN WENG received the bachelor's degree in medicine from the Guanghua School of Stomatology, Sun Yat-sen University, Guangzhou, China, in 2019. She is currently pursuing the master's degree with the School of Stomatology, Wuhan University. Her research interests include the medical diagnosis, medical diagnosis imaging, and medical services.
MINGZHE FAN received the bachelor's degree in oral medicine from the School of Stomatology, Shandong University, Jinan, China, in 2009. He is currently pursuing the master's degree with the School of Stomatology, Wuhan University. His research interests include the medical treatment, medical diagnosis, and medical services. VOLUME 8, 2020


s2fieldsofstudy:
Computer Science
Engineering

text:
A Novel Method for Diagnosis of Bearing Fault Using Hierarchical Multitasks Convolutional Neural Networks

Intelligent mechanical fault diagnosis has developed very fast in recent years due to the advancement and application of deep learning technologies. .us, there are many deep learning network models that have been explored in fault classification and diagnosis. However, there are still limitations in research on the relationship between fault location, fault type, and fault severity. In this paper, a novel method for diagnosis of bearing fault using hierarchical multitask convolution neural networks (HMCNNs) is proposed, taking into account the mentioned relationships. .e HMCNNmodel includes a main task and multiple subtasks. In the HMCNNmodel, a weighted probability is used to reduce the classification error propagation amongmultitasks to improve the fault diagnosis accuracy. .e validity of the proposed method is verified on bearing datasets. Experimental results show that the proposed method is very effective and superior to the existing methods.

Introduction
Rolling bearings, as the key parts of mechanical equipment, are widely used in rail transit equipment, construction machinery, precision machine tools, instrumentation, and other fields. According to statistics, about 40% of rotating machinery faults are caused by bearing faults. Once bearing faults occur, they will seriously affect the normal operation of equipment, and they may even cause accidents and economic losses. erefore, it is necessary to diagnose and monitor bearing faults before anything goes wrong [1,2]. At present, bearing fault diagnosis is usually based on data-driven methods. By collecting motor current signals or bearing vibration signals, fault diagnosis methods are applied to complete fault identification [3,4].
Data-driven fault diagnosis generally includes two steps: fault feature extraction and fault classification. e common methods of feature extraction include Fast Fourier Transformation (FFT) [5], Wavelet Transform (WT) [6], Empirical Mode Decomposition (EMD) [7], Local Mean Decomposition (LMD) [8], and Variational Mode Decomposition (VMD) [9]. e common fault classification algorithms include support vector machine (SVM) [10], BP neural networks [11], Bayesian classifier [12], K-Nearest Neighbor (KNN) [13], Random Forest (RF) [14], and Classification and Regression Tree (CART) [15]. Seryasat et al. [16] presented a diagnosis method based on wavelet transform and FFT to extract energy and root mean square of different frequency bands, which could accurately and effectively identify bearing faults. Yan and Jia [17] proposed a multidomain feature classification algorithm based on optimized SVM, which included three stages: multidomain feature extraction, feature selection, and fault recognition. e algorithm has high diagnostic accuracy for rolling bearings under different working conditions. Zhang et al. [18] proposed a new method of rolling bearing fault diagnosis based on Variational Mode Decomposition and compared the performance of VMD and EMD in extracting bearing defect features from rolling bearing simulation signals. e VMD method can accurately extract the main mode of bearing fault signal and is superior to EMD in bearing defect feature extraction. Liu et al. [19] presented a fault diagnosis method for wind turbine bearings based on integral extension local mean decomposition (IELMD), which could effectively process nonstationary signals. Kankar et al. [20] extracted the statistical characteristics of wavelet coefficients and completed the classification of bearing faults combined with an artificial neural network. Jiang et al. [21] proposed a fault diagnosis method for rolling bearings based on high-order cumulant and BP neural network in view of the fact that the vibration signals of rolling bearings were susceptible to the influence of Gaussian noise.
e basic steps of these traditional fault diagnosis methods can be summarized as follows: acquiring fault signals, analyzing the characteristics of fault signals, extracting appropriate features, and selecting appropriate classifiers according to the specific diagnosis problems. is process requires high professional knowledge and experience of signal processing for fault diagnosis personnel. With the development of modern industry, fault monitoring equipment obtains a large amount of data, and the data types are diverse, which brings great challenges to traditional fault diagnosis methods.
Deep learning has been widely used in recent years. e characteristic of the deep learning method is that it can automatically complete the task of feature extraction and classification [22]. Deep learning has also been introduced into the field of mechanical fault diagnosis to overcome the shortcomings of traditional methods recently. Zhang et al. [23][24][25] proposed a fault diagnosis method for rolling bearings based on deep convolution neural networks (CNN), which avoided manual feature extraction and realized automatic feature learning. Shao et al. [26] proposed an enhanced depth feature fusion method for fault diagnosis of rotating machinery. A new depth autoencoder method was constructed by combining Denoising Autoencoder (DAE) with Contractive Autoencoder (CAE) to improve the learning ability of features. Jia et al. [27] proposed a deep normalized convolution neural networks (DNCNN), which could effectively deal with unbalanced classification problems. Liu et al. [28] proposed an unsupervised fault diagnosis method for rolling bearings based on the generative adversarial networks. is method has higher generalization accuracy under noisy and varied workload situations. ese deep learning methods have been successfully applied to bearing fault diagnosis from different perspectives and application scenarios. Compared with traditional methods, they have higher diagnostic accuracy. However, the problem of low generalization ability of deep neural network model remains unsolved. e fault diagnosis of bearing includes fault location, fault type, and fault degree. In the existing fault diagnosis methods, all kinds of samples are generally used as training samples of the training model to achieve fault diagnosis, and the relationship between them and the impact on the final fault diagnosis results are less considered. For the hierarchical classification of deep learning, Yan et al. [29] first proposed the hierarchical deep convolution neural network model to classify images. It first classifies the easily separated classes roughly and then classifies them at a fine level [28]. Based on this idea, Guo et al. [30] and Qu et al. [31] proposed a hierarchical intelligent fault diagnosis algorithm based on an adaptive deep convolution neural network model (ADCNN), which classified bearing fault location first and then classified fault severity. e design of this hierarchical classification model requires training multiple CNN recognition models, taking pretraining, and fine-tuning. It can thus lead to a cumbersome training process, more training samples, interlayer error propagation, and difficulty in model level expansion. erefore, a hierarchical multitask bearing fault diagnosis method based on the deep convolution neural networks is proposed in this paper. By adding multilearning tasks to the convolution neural network, the multitasks learning of bearing fault diagnosis is realized, and the generalization performance of the proposed model is improved. e main contributions of this paper are as follows: (1) Based on the CNN fault classification model, the HMCNN model is formed by adding several related classification tasks representing different dimension information for parallel auxiliary diagnosis. In the proposed model, final classification results are obtained by fusing the classification results of the main task and subtasks of different dimensions according to the weight obtained by training, which can reduce the interlayer error propagation of tasks. (2) e proposed model can extract more valuable features from fewer training samples for fault classification and improve the classification accuracy of the model. In the proposed model, multiple tasks can share network parameters and information, and only one network structure needs to be trained that reduced computational consumption and training complexity. e parallel structure of multiple tasks also has good scalability. e remainder of the paper is structured as follows. e structure of the hierarchical multitask convolutional neural network (HMCNN) and some main techniques used in HMCNN is introduced in Section 2. In Section 3, experiments are carried out to prove that HMCNN has better performance than traditional intelligent methods and some typical deep learning models. After that, the structure of HMCNN model is extended to demonstrate its ability to extend. Lastly, the diagnostic results of HMCNN model are compared with CNN model and analyzed visually to explore its mechanism. e conclusion of this paper is presented in Section 4.

Proposed Method
In this paper, a novel method called hierarchical multitask convolutional neural network (HMCNN) is proposed for the intelligent fault diagnosis of bearings. e proposed model includes four parts: CNN model, hierarchical classifiers, multitask learning, and hierarchical multitask convolutional neural network. e HMCNN model only needs to train a network model to realize multitask classification, in which the sharing layer can reduce the number of network parameters, thereby reducing the computational load. e multitask learning, hierarchical classification, and joint classification layer design in the HMCNN model can improve network generalization ability. More details are described in the following sections.

Introduction to CNN Model.
Convolutional neural network (CNN) is a kind of feed-forward multistage neural network. It mainly contains three kinds of layers: convolutional layer, pooling layer, and fully connected layer. e convolution layer is designed to extract different features of input data. e pooling layer following the convolutional layer is to reduce the parameters of the network through extracting the local mean or maximum value of input data. A fully connected layer is usually built in the last part of the hidden layer of the convolutional neural network. Its main function is to connect all features and send the output value to the classifier. Convolutional neural network (CNN) is one of the common deep learning models. It is used to extract features and classify vibration signals in bearing fault diagnosis.

Hierarchical Classification.
e main idea of hierarchical fault diagnosis based on convolutional neural network is proposed in this paper, as shown in Figure 1. e main structure of hierarchical classification includes the sharing layer, coarse classification layer, fine classification layer, and joint classification layer, in which the sharing layer can reduce the number of network parameters and thus reduce the computation. e coarse classification layer is mainly used for coarse classification of bearing fault location, such as bearing recognition as health, inner ring fault, and outer ring fault. e fine classification is achieved through the fine classification layer. e joint classification layer which receives fine classification results as well as coarse classification results produces a weighted probability as the final classification results; it can be described as follows: where p c (x j ) is the probability of coarse classification made by the coarse classification layer. p F (x j ) is the fine classification made by the fine classification layer. N is the number of hierarchical tasks. e coarse classification layer, fine classification layer, and joint classification layer are all based on softmax classification function for the final classification tasks. In this way, the output of the network is transformed into a probability distribution; the softmax function is described as follows: where z j is the logits of the j th output. n is the number of categories.

Multitask Learning.
In this paper, bearing fault diagnosis has multiple learning tasks, such as fault location, fault type, and fault severity. From the perspective of machine learning, multitask learning can be regarded as inductive transfer learning, which can improve the learning performance of the model by using multiple related tasks, including improving generalization accuracy, learning speed, and comprehensibility of the learning model. In this paper, the related learning tasks are fault or fault location, which helps the final classification of severity tasks of fault categories. In the training process of the model, the joint training method is adopted, which combines the loss functions of multiple tasks and carries out the optimization training together. e loss function is described as where k i is the coefficient, loss i is the loss function for each task, N is the number of hierarchical tasks, m is the size of training minibatch, p j k is the true predicted output value, and q j k is the one-hot type vector with target distribution.

Hierarchical Multitask Convolutional Neural Network (HMCNN) Model.
e HMCNN model is shown in Figure 1, which consists of the following four parts. e first part is the sharing layer, which consists of two modules. e two modules are composed of 2 convolution layers and 1 pooling layer. e 3 × 1 size convolution kernel with stride of 2 is used in all convolutional layers. For pooling layers, the 8 × 1 sized max-pooling with the stride of 8 is done. e second part is the coarse classification layer, which is connected to the shared layer. It consists of 2 full connection layers and 1 softmax layer, which are used to complete the classification of bearing fault location, fault type, and fault severity. e third part is the fine classification layer, which is connected to the shared layer and consists of 3 convolution layers, 1 pooling layer, 2 full connection layers, and 1 softmax layer. It is used to complete the fine classification of bearing fault. e fourth part is the joint classification layer, which receives fine classification results as well as coarse classification results and produces a weighted probability as the final classification results. e HMCNN model training parameters are shown in Table 1.

Data Description.
Experimental data were collected from the bearing test rig of Paderborn University in Germany [32]. e experimental data were obtained by the test rig for condition monitoring of rolling bearings. e test rig consisted of several modules: an electric motor (1), a torque-Shock and Vibration measurement shaft (2), a rolling bearing test module (3), a flywheel (4), and a load motor (5), as shown in Figure 2. e test bearing was ball bearings of type 6203. Bearings are run at a rotational speed of 900 rpm with a load torque of 0.7 Nm and a radial force on the bearing of 1000 N. e frequency of the data acquisition system is 64 kHz. e bearing temperature was kept roughly at 45-50°C. ree kinds of bearing states are used in this experiment: inner ring damage, outer ring damage, and healthy. e detailed situation of data is shown in Table 2. In Table 2, the bearing fault location, damage method, and fault severity are listed. For fault location, H is the bearing with no fault, IR is the bearing with an inner race fault, and OR is the bearing with an outer race fault. e damage methods of bearing are shown in Figure 3. e bearing damage used in this paper was caused by three different methods: electric discharge machining (trench of 0.25 mm length in rolling direction and depth of 1-2 mm), drilling (diameter: 0.9 mm, 2 mm, and 3 mm), and manual electric engraving (damage length from 1 to 4 mm).
As described in the document of the dataset, each bearing acquired 20 original vibration time-series signals, each of which recorded about 256,000 data points. In this experiment, the 2048 data points were used to construct a sample. For each health condition of bearings, 2000 samples were used in the training set and 500 samples were used in the test set. e vibration signals of each health state are shown in Figure 4. e experimental data are normalized by maximum and minimum normalization, and the normalization formula is as follows: where x i is the value of the i-th point of sample data. x max is the maximum value of sample data. x min is the minimum value of sample data. In this paper, the proposed model is based on tensorflow deep learning framework. e experiment was completed on a computer with CPU i7 8700, 16 GB memory, and NVIDIA GTX 1070 GPU.

Diagnosis Results of HMCNN.
e experiments are divided into three parts. e first part is a comparison among the proposed model, traditional method, and intelligent algorithm based on deep learning to demonstrate the superiority of the proposed model in terms of generalization performance. e second part is to extend and compare the     Figure 4. From Figure 5, it can be seen that the classification accuracy of traditional models such as SVM and DNN is less than 80%, while the classification accuracy of deep learning model such as CNN and LSTM is about 90%. e bearing fault diagnosis accuracy of HMCNN model reaches 99.7%. Compared with traditional bearing fault diagnosis models, it does not need to extract features, has higher diagnosis accuracy, and has better generalization ability than the current deep learning models.
e HMCNN model and CNN model are also compared and analyzed. e CNN and HMCNN models use the same optimization method and training parameters in this paper. e accuracy of per 50 steps in HMCNN and CNN model is shown in Figure 6. Figure 6 shows that the bearing fault diagnosis accuracy of HMCNN model is 99.7% and that of CNN model is 92.1%. Compared with CNN model, HMCNN not only has higher bearing fault diagnosis accuracy but also has fewer training steps to achieve the highest diagnosis accuracy. e confusion matrix of experimental results is compared as shown in Figure 7. From Figure 7, compared with CNN model, HMCNN model mainly reduces the confusion degree between outer ring bearing faults, so as to improve the diagnosis accuracy of bearing faults.
In the training processes, the learning speed of HMCNN model and CNN model is compared, as shown in Figure 8. e convergence rate (learning speed) of HMCNN model is about twice that of CNN model.
In order to further verify the generalization performance of HMCNN model, the diagnosis accuracy of HMCNN with other models under different training sets is also compared here. e comparison results are shown in Table 3 and Figure 9. As the number of training samples decreases, the recognition accuracy of all methods decreases to varying degrees. From Table 4, it can be seen that the HMCNN model has better recognition accuracy in fewer training sets, and the recognition accuracy can reach 96.7% in the case of only 500 training samples.
In addition, the performance differences between HMCNN model and SVM, BPNN, CNN, and LSTM model under noise are compared. Comparison results are shown in Table 3 and Figure 10. It can be seen that the diagnosis accuracy of HMCNN is 99.1% in noise environment (SNR � 10 dB), while the diagnosis accuracy of other models is not more than 90%. At the same time, the diagnosis accuracy of HMCNN model in noise environment (SNR � −2 dB) is more than 90%. So HMCNN has good antinoise performance.

e Comparative Analysis of HMCNN with Different
Tasks Numbers. In the second part, we study and analyze the relationship between the hierarchical tasks' number of HMCNN model and its diagnosis accuracy. According to the dataset, the HMCNN model is used to learn one, two, and three classification tasks (bearing fault location, fault type, and fault severity), which are named HMCNN1, HMCNN2, and HMCNN3, respectively. e comparison results are shown in Figures 11 and 12. e results show that both HMCNN3 and HMCNN2 model can achieve a high diagnosis accuracy. e accuracy of HMCNN2 model is 0.8% higher than that of HMCNN1 model. Compared with CNN, HMCNN contributes more to the accuracy improvement by adding the task of bearing fault location. In HMCNN3 model, the diagnosis accuracy of fault type reaches 99.7%, which shows that the task of bearing fault position, fault type, and fault severity diagnosis is effective and feasible, and the final diagnosis accuracy is improved to a certain extent.  Shock and Vibration e comparison between HMCNN1, HMCNN2, and HMCNN3 models proves that the proposed model can be extended to diagnosis multiple tasks. In practical application, the location, type, and severity of bearing fault can be output in HMCNN model, which provides more detailed guidance for fault maintenance.

e Comparative Analysis of HMCNN Model with ADCNN Model.
In the third part, the ADCNN model proposed by Guo et al. [30] is compared with HMCNN model. e idea of ADCNN model for bearing fault diagnosis is to identify the location of bearing fault first and then identify the fault severity of each location of bearing on this basis. Accuracy results of ADCNN and HMCNN models for diagnosis bearing fault locations and final fault severity are shown in Tables 5 and  6, respectively. We can see that the ADCNN model's hierarchical diagnosis of the bearing will make the error of bearing fault location diagnosis spread to the result of bearing fault severity diagnosis, and the more the number of tasks, the more serious the error propagation. e HMCNN model has a shared layer and a weighted joint classification layer, which can solve the problem of error propagation and make the model more scalable.     Shock and Vibration 7

Visualization Analysis.
e principle of HMCNN model is further analyzed by t-SNE visualization technology. In this paper, the test set data are used as input, and the output data of the pooling layer in the HMCNN model are extracted as output. ese output data are reduced to two-dimensional feature vectors by t-SNE, and then these outputs are plotted as scatter plots, representing their classes with different colors, as shown in Figure 13. e visualization results show that, with the increase of network layers of HMCNN model, the separation degree of features extracted from original signals becomes more and more obvious. At last, the output features of softmax classifier have seven distinct distributions (the final classification of bearing faults).
By comparing the output of two identical pooling layers of HMCNN and CNN models as shown in Figure 14, it can be seen that HMCNN has a better classification effect than CNN       e visual output of HMCNN and CNN third pool layer shows that CNN model has no obvious effect on the diagnosis of KA07 and KA08 bearings, while HMCNN model has an obvious effect on the diagnosis of KA07 and KA08 bearings. is proves that, in the third pooling layer of HMCNN model, the bearing fault severity is clearly classified. From the network of HMCNN, the second pooling layer is essentially the last layer of shared layer.
In the shared layer, the proposed model learns the features of bearing fault location, fault type, and fault severity.
HMCNN model can share more fault information than CNN model through multitasks learning. In particular, the task of fault location focuses the network attention on the possible neglected fault location information, which enhances the classification ability of bearing fault location of HMCNN model and the KI07 bearing can be separated obviously by HMCNN model in the second pool layer. After shared layer, the HMCNN model only needs to recognize KA07 and KA08 bearings, but CNN model also needs to recognize KI07, KA07, and KA08 bearings, which may lead to the final recognition accuracy of the CNN model which is lower than that of the HMCNN model. From the analysis of the learning process of HMCNN model, multitask learning in HMCNN model may improve the generalization accuracy of the model by using the information hidden in training signals of multiple tasks as an inductive bias. Multitask learning plays the same role as regularization and reduces the risk of model overfitting. At the same time, it reduces the ability to fit random noise and makes the model have better generalization performance.

Conclusions
e hierarchical multitask learning CNN model (HMCNN) is proposed, which reflects hierarchical classification. Only one model needs to be trained to achieve a multitask classification. Compared with the experimental results of other models, the HMCNN model can improve the accuracy of the final fault diagnosis, and the diagnosis accuracy reaches 99.7%. Compared with CNN model, the HMCNN model has a faster learning speed. We compare the diagnosis accuracy of HMCNN with other models in different training samples and noise environments. HMCNN model has better diagnosis accuracy than other models. It is proposed that the HMCNN model can be extended to diagnosis multiple tasks. e fault location, fault type, and fault severity of bearing fault diagnosis are given, which can provide more detailed guidance for fault maintenance. Compared with the ADCNN model, the HMCNN model solves the problem of error propagation and makes the model scalable.
rough the visual analysis of the HMCNN and CNN model learning process, the reason why HMCNN has higher generalization accuracy is further explored. HMCNN model shares more fault information than the CNN model. In

Data Availability
e data that support the findings of this study are available at https://mb.uni-paderborn.de/en/kat/mainresearch/datacenter/bearing-datacenter/data-sets-anddownload/?tdsourcetag�s_pcqq_aiomsg. At the same time, the data used to support the findings of this study are available from the corresponding author upon request.

Conflicts of Interest
e authors declare that they have no conflicts of interest.


JCT at SemEval-2020 Task 12: Offensive Language Detection in Tweets Using Preprocessing Methods, Character and Word N-grams

In this paper, we describe our submissions to SemEval-2020 contest. We tackled subtask 12 - “Multilingual Offensive Language Identification in Social Media”. We developed different models for four languages: Arabic, Danish, Greek, and Turkish. We applied three supervised machine learning methods using various combinations of character and word n-gram features. In addition, we applied various combinations of basic preprocessing methods. Our best submission was a model we built for offensive language identification in Danish using Random Forest. This model was ranked at the 6 position out of 39 submissions. Our result is lower by only 0.0025 than the result of the team that won the 4 place using entirely non-neural methods. Our experiments indicate that char ngram features are more helpful than word ngram features. This phenomenon probably occurs because tweets are more characterized by characters than by words, tweets are short, and contain various special sequences of characters, e.g., hashtags, shortcuts, slang words, and typos.

Introduction
Nowadays, social networks occupy an increasingly important place in our life. However, their rapid development has also allowed the development of increasingly offensive, hateful content that takes advantage of the anonymous nature of the cyber world. This bullying, trolling, and harassment content has become a growing concern for governments, organizations, and individuals. The use of machine learning (ML) and natural language processing (NLP) solutions to find this offensive content has been surprisingly useful. Still, the detection of offensive language from social media is not an easy research problem due to the different levels of ambiguities present in natural language and the noisy nature of social media language. In addition, social media subscribers come from linguistically diverse communities. Up to now, most research has focused on the detection of offensive language in English but works have also been written for other languages like Struß et al. (2019) for German and Basile et al. (2019) for Spanish. Task 12-A of SEMEVAL 2020 deals with the detection of offensive language in tweets in five languages including English. In this paper, we describe our models submitted to subtask 12-A for tweets written in four languages Arabic, Danish, Greek, and Turkish. The full description of task 12 is given in . The structure of the rest of the paper is as follows. Section 2 introduces a background concerning offensive language, tweet classification, and data preprocessing. Section 3 describes subtask 12 and its datasets. In Section 4, we present the submitted models and their experimental results. Section 5 summarizes and suggests ideas for future research.

Offensive language
In recent years, more and more workshops on offensive content, hate speech, and aggression have been organized. This phenomenon shows the growing interest in the field. Schmidt and Wiegand (2017) noted in their survey paper that for hate speech detection the most used approach is the supervised one with a focus on support vector machines (SVM) and recurrent neural networks (RNN) (Pavlopoulos et al., 2017). Zampieri et al. (2019) showed that n-grams can perform well for hate speech detection using SVMs with different surface-level features, such as surface n-grams, word skip-grams, and word representation n-grams induced with Brown clustering. They also noticed that these features reached their limits for more complex tasks, e.g., distinguishing profanity and hate speech. In such tasks, more in-depth linguistic characteristics may be required. Most of the existing work essentially concerns the English language (Xu et al. (2012), Burnap and Williams (2015), Davidson et al. (2017)). However, the discussed competition has provided datasets not only for English, but also for four other languages Arabic, Danish, Greek, and Turkish. Given the large size of the English dataset and the resources required to train models on it, we chose to focus on the four other languages. The datasets for these four languages are relatively small and unbalanced. Therefore, we prefer using classical ML methods and avoid using deep-learning ones such as RNN or CNN.

Tweet Classification
Tweet classification means the classification of a tweet into one of a set of predefined classes. The most common general method for automatically performing this task is through supervised ML methods. We also adopted this general method. Tweets are a challenge for classification as they are informal, short, and contain various misspellings, shortenings, and slang words (HaCohen-Kerner et al., 2017).

Text preprocessing
Text preprocessing is an important step in text mining and ML processes. In text documents in general and in social text documents in particular, it is common to find typos, emojis, slang, HTML tags, spelling mistakes, and repetitive letters. Analyzing text that has not been carefully cleaned or preprocessed might lead to misleading results. Not all of the preprocessing types are considered effective by all text classification (TC) researchers. For instance, Forman (2003), in his study on feature selection metrics for TC, claimed that stop words are ambiguous and therefore should be removed. However, HaCohen-Kerner et al. (2008) demonstrated that the use of word unigrams including stop words lead to improved TC results compared to the results obtained using word unigrams excluding stop words. In our system, we applied various combinations of preprocessing types depending on the language. For all the languages we convert the tweet's characters to lowercase and add a start token and an end token. We first tried to remove stop words but probably due to the already reduced size of the tweet datasets, this led to poorer results. We also tried to use a list of offensive words but in most cases, these words already appeared in the words (or sometimes even in the character n-grams) that we collected in the first step. For Arabic, given the different forms that many words can have, we convert each word to its standard form and remove the noise caused by different vowels. To do so we applied various python functions for developed by Maxim Romanov 1 .

Task Description
SemEval-2020 Task 12  addresses the problem of detecting offensive language in tweets.It is a based on the last edition: SemEval-2019 Task 6. (Zampieri et al., 2019). For Arabic (Mubarak et al., 2020), Danish (Sigurbergsson and Derczynski, 2020), English , Greek (Pitenis et al., 2020), and Turkish (Çöltekin, 2020), task 12 deals with the identification of offensive language in tweets. That is to say, we have to classify each tweet as offensive (OFF) or not offensive (NOT). For English, there were two additional tasks: Automatic categorization of offense types and Offense target identification. As mentioned and explained above, we did not participate in tasks for the English language. A development dataset has been provided for Arabic but not for the other languages in which we split the provided training dataset and allocate 20% of this data for development. Table 1 presents the size and distribution of each subset we used for the applied languages.

The Submitted Models and Experimental Results
The two co-authors of this paper are from different academic places: Bar-Ilan University (BIU) and Jerusalem College of Technology (JCT). Different models have been applied and submitted from two users BIU and JCT. In this paper, we will present the best submitted JCT models. We applied the following three supervised ML methods on the training datasets: Random Forest (RF), Support Vector Classifier (SVC), and Multi-Layer Perceptron (MLP).

Language
Train subset Dev subset Test Subset # of tweet # of offensive one # of tweet # of offensive one # of tweet Random forest (RF) is an ensemble learning method for classification and regression (Breiman, 2001). Ensemble methods use multiple learning algorithms to obtain improved predictive performance compared to what can be obtained from any of the constituent learning algorithms. RF operates by constructing a multitude of decision trees at training time and outputting classification for the case at hand. RF combines Breiman's "bagging" (Bootstrap aggregating) idea in (Breiman, 1996) and a random selection of features introduced by Ho (1995) to construct a forest of decision trees. SVC is a variant of the support vector machine (SVM) ML method (Cortes and Vapnik, 1995) implemented in SciKit-Learn. SVC uses LibSVM (Chang and Lin, 2011), which is a fast implementation of the SVM method. SVM is a supervised ML method that classifies vectors in a feature space into one of two sets, given training data. It operates by constructing the optimal hyperplane dividing the two sets, either in the original feature space or in higher dimensional kernel space. Multi-layer perceptron (MLP) is a deep, artificial neural network (Pal and Mitra, 1992). This model is based on a network of the computational unit, called perceptron, interconnected in a feed-forward way. The network is composed of layers of perceptron that each one has directed connections to the neurons of the subsequent layer. Usually these units apply a sigmoid function, called the activation function, on the input they get and feed the next layer with the output of the function. This model is very useful especially when the data is not linearly separable. We preferred to use the Relu function as activation function which show better convergence performance than sigmoid (Krizhevsky et al., 2012) and logistic loss as loss function. We also Adam solver for weight optimization (Kingma and Ba, 2014) and a batch size of 200. These ML methods were applied using the following tools and information sources: • The Python 3.7.3 programming language 2 • Scikit-learn -a Python library for ML methods 3 • Numpy -a Python library that provides fast algebraic calculous processing especially for multidimensional object 4 . Table 2 presents the Macro F1-Score results of our best JCT's models in task 12-A. On each language, we applied these three different supervised ML methods for various combinations of character and word n-gram features from unigram to 5-gram. We, then, selected the model that gives the best result and we try to combine them. Finally We only submit one model per language, so the results of the other models on the test are not available (marked as NA in the table). Our submitted models for at least three languages (Arabic, Danish, and Greek) show that the most helpful type of feature sets for the tweet classification  tasks is the char ngram features. Specifically, the best char ngram features are usually thousands of bigram and/or trigram and/or 4-grams and/or 5-grams features. These findings are because tweets are more characterized by characters than by words, tweets are relatively short, and contain also emojis, hashtags, onomatopoeia, slang words, shortcuts, and typos. Only for the Turkish language, the two submitted models used word n-grams (in addition to char ngrams). However, the results of these two models are relatively low. Like we can see with the results of the majority baseline for each language the presence of offensive tweet in the submission set is disproportionate compared to this one in the training and development set. This can explain the differences between the results obtained by some models on the development set and on the submission, like for example the SVC on Arabic that reached a F1 of 0.8065 on the result.

Conclusions and Future Research
In this paper, we describe our submissions to subtask 12 of SemEval-2020 contest. Like said before, we developed several models for each language for four languages: Arabic, Danish, Greek, and Turkish, after what, we select the two best models while trying to choose two models that use a different method of supervised learning and submit them. Our best submission was a model we built for offensive language identification in Danish using Random Forest. This model was ranked at the 6th position out of 39 submissions. Its Macro F1-Score result (0.7741) is lower by only 0.003 than the result of the team that won the 4th place. It is interesting to note that this is the only model among those in the top which does not use a neural method which show that classical methods still remains relevant.
Many tweets contain acronyms that can be presented in different forms. These acronyms can lead to ambiguity. Future research may seek to lessen this ambiguity. Acronym disambiguation, e.g. HaCohen-Kerner et al. (2010a), will extend and enrich the tweet's text and might enable better classification. We also suggest examining the usefulness of skip character n-grams because they serve as generalized ngrams that allow us to overcome problems such as noise and sparse data (HaCohen-Kerner et al., 2017). Other ideas that may lead to better classification are to use stylistic feature sets (HaCohen-Kerner et al., 2010b), key phrases (HaCohen-Kerner et al., 2007), and summaries (HaCohen-Kerner et al., 2003). We may also apply advanced deep learning methods and especially RNN, which is more suitable for text applications.