s2fieldsofstudy:
Computer Science
Medicine

text:
Cervical Cancer Diagnosis Using Very Deep Networks Over Different Activation Functions

Cancer prevention is mainly achieved by screening the transformation zones. Cervical pre-cancerous stages can be seen in three different types, and all can transform into cancer. Thus, it is crucial to intelligently screen cervical abnormality and have a robust system for detecting whether a cervix is in normal (healthy) or at a pre-cancerous stage. Deep learning showed great potentials when applied to biomedical problems, including medical image analysis, disease prediction, and image segmentation. Hence, in this paper, very deep residual learning based networks are designed in order to perform cervical cancer screening. Moreover, in this work, we highlight the importance of the activation functions on a residual network (ResNet)’s performance. Thus, three residual networks of the same structure are built with different activation functions. The employed models are trained and tested using a dataset of colposcopy cervical images, and the experimental results showed that designed residual networks with leaky and parametric rectified linear unit (Leaky-RELU and PRELU) activation functions performed almost equally in terms of accuracy where they reached accuracies of 90.2 and 100%, respectively. This achieved high accuracy was compared to other related works’ results, and it showed an outperformance in screening the pre-cancerous and healthy colposcopy cervical images. Such an earlier and accurate diagnosis may help in preventing cervical cancer transformation.

I. INTRODUCTION
Cervical cancer develops in the cervix, the narrow entrance of the uterus. This cancer can mainly affect sexually active women aged between 30 and 45 [1]. It is estimated that 13800 of women will be diagnosed with cervical cancer in the United States [2] in 2020. The leading cause of this cancer is the human papillomavirus (HPV), i.e., a sexually transmitted virus [1]. Cervical cytology, or the so-called smear test, is the most common test used to detect cervical cancer. This test can help in detecting the cancer in its earlier stages, which helps in reducing the number of deaths [1]. This cancer can have five different stages and detecting it in its earlier stages improves the survival rate [2].
The associate editor coordinating the review of this manuscript and approving it for publication was Charith Abhayaratne .
Over the past few decades, studies have been extensively conducted for the creation of computer-aided diagnosis (CAD) systems to help diagnosing diseases and analyzing medical images, and computer-assisted reading systems have been proposed for cervical cancer cell segmentation and classification [3]- [5]. These systems are based on automated image analysis algorithms and designed to select potential abnormal cells in a given cytology specimen to be classified. However, this task includes cytoplasm and nuclei segmentation, handcrafted features extraction, and cell classification. Cell segmentation is crucial in designing a CAD system; however, the large variations between normal and abnormal nuclei and the presence of cell clusters can form a significant obstacle to reaching an accurate segmentation of cells [3].
Moreover, engineered, and handcrafted features representing morphological characteristics of cells are costly and time inefficient. In addition to that, they may be error-prone as they are limited to the understanding of the cervical cancer cytology and the finite imagination and visualization of human experts [6]. Overall, it has been noted that the current cervical cancer diagnosis systems that depend on handcrafted features are hindered by these aforementioned limitations, and others like ignoring some clue features and removing complementary information of the input image. Thus, there has been an urgent need for developing an automated technique for extracting different levels and discriminative features that can represent complex and distinguished information to describe cell abnormality. This technique of feature extraction has been developed in the deep learning networks, convolutional neural networks, which are biologically inspired structures, consist of powerful feature extraction tools [7]. These networks consist of several convolutions and pooling layers that act as feature extraction tools of effective capability in extracting features in a hierarchical way and at multiple levels, in a similar manner from the input space.
Recently, deep convolutional neural networks (DCNN) have shown remarkable performance in several medical tasks such as medical image analysis and understandings [4], [5], medical image classification [6], and cancer detection and identification [5], [6]. These networks have undergone significant upgrades and improvements over the years. This upgrade was mainly in terms of depth and performance, and it was found that the performance of CNNs varies proportionally to their corresponding depth, i.e., the number of hidden layers [4]- [6]. Hence, deep networks of various depths were proposed over the years, starting from the AlexNet [8] which consists of 8 layers, to VGGNet of 16 layers [9] and GoogleNet of 22 layers [10]. This increase in depth makes it difficult to train a network in terms of time, but the evolution of computers minimized this difficulty. On the other hand, going deeper and deeper, another problem has been encountered. It was noticed that training a very deep network (over than 18 layers) is associated with a vanishing gradient phenomenon due to the backpropagation algorithm used for training [11]. Thus, to suppress the issue, residual learning was proposed in 2016 [12]. This new concept of deep learning presented the shortcut connection or residual blocks in building the network. This concept is based on skipping connections between layers within the residual unit and an identity map of the input data instead of consecutive connections similar to AlexNet and GoogleNet.
Generally, all proposed residual networks (18,50,101, and 152 layers) use a non-saturating rectified linear unit (ReLU) as an activation function that allows complex relationships to be learned by the network [11], [12]. This function has been verified to a great outcome in many works [13]- [15]; however, researchers found that it can be associated with a problem called ''dying ReLU,'' i.e., it outputs zeros for the negative input values it receives [16]. This, therefore, badly affects the learning of the network. Hence, other activations networks were proposed in order to avoid the 'zeros' problem caused by ReLU and improve the learning process of the network. For such purposes, the Leaky rectified linear unit (Leaky-ReLU) [17] and Parametric rectified linear unit (PReLU) [17] were proposed in 2015 and 2016, respectively. This paper addresses the cervical cancer screening using deep learning. The studies of diagnosing cervical cancer using deep learning are quite limited. Hence, there is a need for applying deep networks in detecting the pre-cancerous cervical colposcopy as this may prevent their transformation into cancer. In this context, a classification of colposcopy cervical images into pre-cancerous and healthy colposcopy images is carried out using a ResNet18 structure's inspired network. The network is built from scratch and trained on precancerous and healthy images in order to detect patients that are more likely to develop cervical cancer. As this network is a newly designed and built, we attempted to investigate the best activation function that fits our model. Hence, we duplicated the same network's structure over three different activation functions (ReLU, Leaky-ReLU, and PReLU) and evaluated its performance. The developed residual network with ReLU activation function is denoted as ReLU-ResNet, and the network with Leaky-ReLU function is denoted as Leaky-ReLU-ResNet. The last is the residual network with Parametric ReLU function and denoted as PReLU-ResNet. Thus, in this paper, three networks with the same structure but with different activation functions were considered. As aforementioned, the developed networks are inspired by the ResNet18 architecture; however, more layers have been added in order to make them deeper, for optimal classification. The performance of the three designed networks is validated and tested on a cervical cancer diagnosis task, which is considered a challenging classification task due to the complexity and similarity of the pre-cancerous and healthy cervical colposcopy images.
The rest of the paper is organized as follows: Section 2 presents a deep insight into the residual learning, activations functions and dataset description. Section 3 discusses the problem formulation and the proposed models, in addition to the results discussion. Section 4 is a conclusion of the work.

A. RESIDUAL LEARNING
Recently, deep networks started to go deeper, i.e., several hidden layers. This ''very'' depth seemed to be associated with some optimization difficulties during the learning process of networks. This problem is called the vanishing gradients, and it arises when a network is trained with a stochastic gradient descent algorithm [13], [17]. The idea is that as the networks go deeper, its calculated gradient of the loss function starts to decrease exponentially while it back propagates to initial layers. Hence, it may approach zero at some layers, and this makes the network hard to train. Small or zero gradient means that weights and biases may not be adequately tuned during every training pass, and this consequently leads to less convergence and high error value.  To overcome this issue, He et al. [12] proposed a residual learning concept designed for building and training very deep networks easily, without confronting the vanishing gradients problem. Residual networks (ResNets) introduced a new concept of connections called the skip or identity connections over some layers instead of connecting layers in a shallow way. This is based on residual units within the network, where the identity of input is mapped from the precedent hidden layers to higher ones. Hence, instead of learning the underlying mapping of input, the network is allowed to fit its residual mapping, as seen in Fig. 1(a).
There can be two types of residual mappings. The first one is the identity mapping-based shortcut connectivity ( Fig. 1.a), in which the input encounters skip or short connection directly without passing through the convolution layer [12], [13]. The second residual block is the convolutional based identity mapping, i.e., short connection of input encounters a 1 × 1 convolution layer in the shortcut path ( Fig. 1.b). Fig. 1 shows the residual units of a residual network structure.
The use of more identity mapping-based shortcut connectivity leads to better performance of the network in terms of computational complexity and training time; also, the convolution-based identity mapping results in a more straightforward propagation of information acquired during the forward and backward passes of the network learning [12].
Moreover, Fig. 1 shows that each residual unit contains two convolutional layers of 3 × 3 filter's size. These filters store the learnable parameters that are optimized during the training process of the whole network. In addition, each unit also has one batch normalization (BN) that normalizes the features map and one rectified linear unit (ReLU) layer that acts as an activation function for the network. The inputs and outputs are computed as follows [12]: where h (X i ) represents the identity mapping of input X i , and F is the residual function. x, w, and y represent the input, weight, and output, respectively. Therefore, in the case of identity mapping-based shortcut connectivity, the identity mapping ( Fig. 1.a) is defined as h (X i ) = X i . However, in the case of convolutional based identity mapping ( Fig. 1.b), the identity mapping is defined as h (X i ,W i ) and it is not the same as input as it is surpassed by a 1 × 1 convolution layer. In this case, input-output computations are defined as follows: where h (X i , W i ) represents the convolutional based identity mapping.

B. ACTIVATION FUNCTIONS
In this section, the three activation functions employed in this study, namely, ReLU, Leaky-ReLU, and PReLU, will be discussed in detail. Activation functions are the functions that set a specific output or ''activation'' for a given input, considering its weight. Preferred transform functions are the non-linear functions that can learn complex mapping functions [19]. Traditional and shallow neural networks with one or a very few hidden layers are used to perform well with the Sigmoid activation function. This non-linear activation function transforms the input values into a range of 0 and 1. Therefore, for any given positive or negative input, the output is between 0 and 1. However, one major problem is encountered when using the Sigmoid function. The sigmoid function saturates with a given large positive or negative input; hence, the local gradient is minimal (almost zero). Therefore, during the backpropagation of error, this gradient is multiplied by that local gradient, and as a result, the gradient is vanished or killed. This is then called the vanishing gradient problem.

1) RECTIFIED LINEAR UNIT (ReLU)
A rectified linear unit (ReLU) was then proposed to overcome the problem caused by the Sigmoid activation function [20]. It has become very popular in the deep learning networks and leads to many breakthrough applications in different fields [8]- [10]. It is merely thresholded at zero input values, i.e., it zeros the negative input values, while keeping the positive input values, as seen in equations 5 and 6.
Which has the gradient of where F is the ReLU activation output, and x is the input value.
ReLU boosted the performance of the deep convolutional neural networks as it helps networks to converge faster and not to be stuck in the vanishing gradient problem. On the other hand, ReLU can sometimes fall into ''dying ReLU,'' where the input values of the ReLU neuron are stuck to negative values. In this case, ReLU output is always zero, and this deactivates the majority of the neurons. This, therefore, affects the learning of the network and results in a poor convergence and performance [21]. Setting the learning rate to minimal values can somehow solve this problem; however, other modified versions of ReLU were also proposed to avoid the zeros outputting of the negative input values.

2) LEAKY-ReLU
Xu et al. [22] proposed another activation function, namely, Leaky ReLU (Leaky-ReLU), to improve and modify the traditional ReLU and to solve more complex and non-linear functions. The main aim of Leaky-ReLU was to solve the problem associated with ReLU, i.e., ''dying ReLU.'' Thus, this method suggests a slight leak of information in the part where output is always 0. This means that the gradient will be small but not zero; therefore, neurons won't be inactive as the weights will be adjusted. This function proposes that in the case of negative input, the corresponding output will be the input multiplied by a small number (0.01) in order not to shut off the neurons (Equation 7 and 8).
Which has the gradient of 3

) PRELU
Applying Leaky-ReLU on CNNs [19], [22] showed that this function could have some benefits over ReLU, such as solving the vanishing gradient problem by avoiding the zero gradient's part and speeding up the network's learning and convergence. On the other hand, it was noticed by some studies [20] that Leaky-ReLU's impact on the network's accuracy is negligible; therefore, a new modified version of Leaky-ReLU was proposed by He et al. [17]. This new method is called a parametric rectified linear unit (PReLU), and it is similar to Leaky-ReLU. However, instead of having a predefined slope value of 0.01 for negative inputs, PReLU allows this coefficient to be learnable by the network, during training just like weights and biases. The output of this function is: This function has the gradient of where F is the PReLU activation function, x is the input value, and α is the learnable coefficient. Fig. 2 shows the VOLUME 9, 2021 three different activation functions. It is seen that PReLU can include them all, depending on the coefficient value, which is learnable in this case: If α = 0; F becomes ReLU If α > 0; F becomes Leaky-ReLU If α is a learnable parameter, F becomes PReLU In this paper, these three activation functions are applied in a residual network built for the diagnosis of cervical cancer. The network's performance is compared over the three aforementioned functions in terms of accuracy in order to select the optimal function for such residual network's classification task.

C. EXPERIMENTAL ANALYSIS
Dataset: Three residual networks of 18 layers are built, each with different activation functions. The networks are trained and validated with cervical pre-cancerous datasets that contain colposcopy images of healthy and pre-cancerous cervixes. The pre-cancerous stage cervical images are collected from the Intel and MobileODT Cervical Cancer Screening, which is a competition launched on Kaggle [18] to classify three types of cervical pre-cancer images (earlier stages of cervical abnormalities that may transform into cancer). This dataset provides three types of pre-cancer cervical images. In this study, all three types are considered as one class, which is pre-cancerous since our aim is to classify pre-cancerous or healthy cervixes. Normal or healthy images are collected from the Tripoli Hospital center, Libya [19]. Table 1 shows the number of healthy and pre-cancerous cervical images collected from both datasets. As it is seen in Table 1, 4,000 pre-cancerous images are collected from the first dataset, and 800 healthy images are collected from dataset 2. The learning scheme used for training and testing the network is 50:10:40, i.e., 50 % of the images are used for training the network, 10% for validation, and the remaining are used for testing purposes.
As noticed, the number of abnormal cervical images number is ∼ three times more than normal images in the created dataset. Hence, our classifier may tend to exhibit a bias towards the majority class (abnormal cervices), which is practically not desirable. Therefore, balancing the proportions of both classes is considered a common solution for this issue. Data augmentation is then applied only to healthy images in order to increase the number of images to ∼ 4000. Hence, shift translation and scale invariance are employed to balance the classes' proportions and to provide the network with the power of detecting the condition of disease at different angles and shifts. Therefore, the 800 original healthy cervical images are rotated at angle 90 • and 180 • and randomly translated up to two pixels horizontally and vertically. In total, a dataset of 3920 healthy cervical images is formed and shown in Table 2. As for preprocessing, our collected and augmented images were first converted to three-channel PNG format. Moreover, images were normalized to scale their pixel values to the range of 0 to 1, and resized to 224 × 224 pixels to fit the employed models' inputs. Fig. 3 shows a sample of images used for training the networks. The first row shows the pre-cancerous cervical images, which are three types, while the second row shows the healthy cervixes. As mentioned above, our aim in this work is to create a deep network from scratch and train it to classify cervical pre-cancerous and healthy images in order to diagnose cervical cancer in its earlier stages; so that it can be treated before transforming into cancer. Thus, we adopted the ResNet18's structure to build our network. Moreover, we investigated the performance of this model using three types of activations in order to discover which function can cause a slight impact on the network's accuracy using a specific dataset.

1) DEEP RESIDUAL NETWORK-BASED CERVICAL DIAGNOSIS
As outlined, three deep residual networks with three different activation functions are designed to diagnose cervical cancer. Our network architecture is inspired by the ResNet18 structure and is comprised of 50 layers with four residual and convolution blocks, as shown in Fig. 4. Each residual and convolution block contains 1 × 1 and 3 × 3 convolution operations (Conv) and rectified linear unit functions (ReLU). Moreover, batch normalization (BN) and dropout layers are also used for building more feasible designs and improving model generalization. Note that some layers are not shown in the figure due to simplicity.
The whole output of the residual blocks of all these layers undergoes a 3 × 3 max-pooling operation and then sent to the global averaging pooling layer [25]. This layer replaces the fully connected layer, and it converts every feature map into a value. The computed values are then fed into a Sigmoid function as our task is a binary classification task. Note that several studies have found that global average pooling reduces overfitting as, in this layer, there is no parameter to be optimized [12], [13], [25]. Moreover, this approach is more robust and efficient to spatial translation, and the shift of the input for it sums the spatial information of the input data.
This research addresses the activation functions of a convolutional network on a particular medical classification task; hence three ResNets are built, however, each with different activation functions. Fig. 4 shows only the ReLU based ResNet architecture; however, two other similar networks are also built with the same architecture but with different activation functions, one with Leaky-ReLU and the other with PReLU. Note that these functions are employed in the residual blocks, whereas a Sigmoid function is used for the fully connected layer since our cervical classification task is considered a binary task, i.e., pre-cancerous or healthy cervices.

2) TRAINING
As mentioned above, networks are trained using the same healthy and pre-cancerous images. A learning scheme of 60:40 is used to train and test the three developed models. This means that 60% (4752 images) of the data are used for training, and 40% are used as a held-out test (3168). The 10% of the train data was used to validate the networks to fine-tune their hyperparameters and reduce overfitting.
All employed ResNets were trained on the training dataset with a learning rate of 1e−4, 0-1 input image normalization, cross entropy loss function, growth rate of 12, block depth of 6, for 20 epochs on a MATLAB 2020a. This selection of hyperparameters was based on the grid-hyperparameters search, inspired by the work of Huang et al [31].
The networks were simulated using Windows 64-bit desktop computer with an Intel Core i7 4770 Graphical processing unit (GPU) and 8 GB random access memory.
Networks are evaluated by calculating their training and testing accuracy using the following formula: (11) where N denotes the total number of correctly classified images, and T represents the total number of images. The loss function used in this work is a binary cross-entropy as cervical cancer screening is considered a binary classification problem in this study. Note that more evaluations metrics such as sensitivity, specificity, and area under curve (AUC) are calculated as follows: Specificity = TN TN + FP (13) where TP stands for true positive, and it indicates the number of correctly predicted positive classes. TN stands for true negative, and it indicates the number of correctly predicted negative classes. FP is the false positive, and it shows the number of incorrectly predicted negatives as positives by the VOLUME 9, 2021  model, while FN is the false-negative, indicating the number of incorrectly predicted positive data as negative. AUC is the area under the Receiver Operating Characteristic curve (ROC), which is a graph that shows the performance of the network at thresholds. ROC plots the True positive rate versus the false positive rate. Table 3 shows the setting of the learning parameters of the three different networks. A Stochastic gradient descent method is used to train the networks with a minibatch size of 64 images for every iteration to optimize the proposed models. Moreover, the learning rate and reducing factor of the fully connected layers of the three networks are set to 0.001 and 0.1, respectively. Epoch's number is selected based on the variations of the accuracy and validation error. In other words, the number of epochs is selected based on monitoring the validation error and performance of the networks as it can be associated with optimization problems, i.e., if it is high, overfitting may occur. Hence, a maximum number of 30 epochs is selected, and networks can stop learning whenever validation error starts to increase, and accuracy starts to saturate (Early stopping). Thus, we draw the learning curves of the three networks (Fig. 5). These curves show the variations of training classification accuracy against the increase of epochs. It is seen that the three networks reached their highest accuracies at epoch 20 and then started to saturate. Therefore, learning stops at epoch 20, the optimum epoch. It is also noted that the networks required approximately the same time to reach such maximum accuracies, which is 125 minutes, 119 minutes, and 122 minutes for ReLU-ResNet, Leaky-ReLU, and PRELU, respectively. This similarity, in time and optimum epochs, is because all models share the same structure with different activation functions. In terms of errors, PReLU_ResNet reached the lowest mean square error compared to other models. Table 4 shows the loss values at the end of every residual bock of every employed network being trained, at epoch 2.

III. RESULTS AND DISCUSSION
In order to evaluate the feasibility of the three ResNet models in classifying cervical cancer, we conducted an experimental test of 40% of the remaining cervical data that were not seen before by the networks. Table 4 shows some testing results, such as classification accuracies and evaluation metrics.
As seen in Table 4, the ReLU-ResNet achieved the lowest classification accuracy, sensitivity, specificity, and area under curve (AUC) of 98.3%, 91.2%, 96.2%, and 96.9%, respectively. This may be due to its ReLU activation function, which, as discussed in Section 2, causes 'dying ReLU' during training that leads to a poor performance compared to other employed activations like Leaky-ReLU and PReLU. It is noticed that Leaky-ReLU and PReLU helped in improving the generalization capability of the networks by making the gradient small but not zero, which activates neurons and allows weights to be adjusted. Fig. 6 shows the confusion matrix and ROC graph of the PReLU-ResNet. Fig. 7 shows the learned activations of the network that outperformed all other networks that is PReLU-ResNet. It shows the learned activations at the first convolution and maxpooling layers. It is seen that the network learned gradients and levels of abstractions in these two layers, but those features are kept to color, orientations, and edges since learning is still in the first layers.   Nevertheless, once we go deeper (Fig. 7), networks seem to learn more complex and meaningful features such as objects and complete parts of cervices; as in deeper layers, learned features are built up by combining features from previous layers. In Fig. 8, we opt to show the learned features at a deeper layer and the layer of activations of each network. Thus, we choose the seventh activation function layer of each network and visualize its leaned features. As seen in the figure, the network with ReLU activation function ( Fig. 8.a) appears to learn no meaningful features when compared to other networks of leaky-ReLU (Fig. 8.c) and PReLU functions ( Fig. 8.b), which seem to learn more abstract and complicated features. This, therefore, affects the learning performance of the networks because, as seen in Fig. 7, ReLU-ResNet seems to be incapable of extracting the right features of cervical images that help in classifying whether the cervix is healthy or pre-cancerous. This is most likely the reason why this network achieved a lower accuracy and higher error than other networks. Fig. 9 shows examples of some pre-cancerous (abnormal) cervical images which are misclassified by ReLU-ResNet as normal and correctly classified as pre-cancerous by Leaky-ReLU-ResNet and PReLU-ResNet. Fig. 10 shows samples of some misclassified healthy (normal) cervical images by ReLU-ResNet.
For more interpretability of what the network relies on to make a final decision concerning the class of an image, we visualize the Gradient Weight Class Activation Mapping (Grad-Cam) of some pre-cancerous images. Grad-Cam is a method for visualizing the highest activations' regions that were the reason for the network's final decision. The suspected regions associated with the predicted class are highlighted by heatmaps, in which a jet colormap shows the highest activation regions as deep red and the lowest activation regions as deep blue. Figure 10 shows the Grad-Cam of some testing pre-cancerous colposcopy images that were correctly classified by PReLU-ResNet. 46620 VOLUME 9, 2021

A. MODELS COMPARISON WITH EARLIER WORKS
Several studies have been conducted to diagnose cervical cancer [5], [14], [15], [26], [27]; however, colposcopy based cervical images classification using deep learning is quite limited. Generally, most of the published studies used microscopic data as inputs for their systems [5], [25]. These studies produced significant results, but they cannot be compared with ours as their database used for training and testing the VOLUME 9, 2021  network's performances is of different type. On the other hand, some studies have used the same Kaggle database used in our study but for different purposes, i.e., to classify the three types of cervical cancer [14] or segment the uterine 46622 VOLUME 9, 2021  cervix [28]. Thus, the two studies that used the same dataset and task were considered here to perform a robust comparison. Mustafa and Dauda [29] discussed the classification of colposcopy cervical images into cancerous or healthy. Their work utilized three different deep convolutional neural networks (DCNNs), each with different optimizers such as stochastic gradient descent (SGD), Root Mean Square Propagation (RMSprop), and Adaptive Moment Estimation (Adam). Those networks were all trained and tested using cancerous and healthy cervical images, and the result was compared to select the best optimizer. As the authors claimed, the Adam based CNN achieved the highest accuracy (90%) of classifying normal and cancerous cervices.
In another study, Yuan et al. [30] discussed the recognition of cervical squamous intraepithelial lesions recognition in colposcopy images. This study presented different applications using different neural networks, such as the U-Net model for segmenting the lesion in the cervix and the ResNet VOLUME 9, 2021    Table 5 shows a comparison of the proposed networks discussed in this paper with the aforementioned related works.
It can be seen from Table 5 that the developed residual learning based networks outperform the plain and traditional Adam convolution neural network [25] in terms of accuracy. It is concluded, by such comparison, that the residual learning approach can provide a great positive impact on a CNN as it boosts its generalization capability.

IV. CONCLUSION
In this paper, a very deep network is developed for the purpose of diagnosing cervical cancer using colposcopy images. The designed network is a residual learning-based network (ResNet) inspired by the ResNet18 architecture. Activations function drawbacks are also discussed in this work, and three different activation functions are employed to investigate the impact of activation function on the ResNet's performance. Hence, three networks were designed with three different networks. All networks were trained and tested using a dataset of raw cervical images.
The experimental results showed that all designed networks achieved high-accuracy generalization of cervical cancer diagnosis. However, the ReLU activation function network produced lower accuracy than other networks with the Leaky-ReLU and PReLU activation functions. Moreover, this network, ReLU-ResNet, also showed a weakness in extracting the rightful and complex features that distinguish the two cervical classes, leading to a poor generalization.
Finally, it was proved that Leaky-ReLU and PReLU activation functions could serve as an important booster for a designed residual learning-based network's performance. This performance improvement is mainly due to their ability to solve the 'dying ReLU' problem associated with the ReLU activation function.
Overall, this research shows that a deep learning system can achieve complex medical classification tasks like cervical screening. Such systems can help medical experts in the diagnosis of this disease and finding its pre-shape before its transformation into cervical cancer. Furthermore, this system can be extended to diagnose the three different types of cervical cancer. Such a developed system could be very significant as it can stage cervical cancer, which helps develop the treatment plan tailored to patients with respect to their specific cancer type.
KHALED MABROUK AMER ADWEB received the master's degree from Near East University, Turkey, in 2015, where he is currently pursuing the Ph.D. degree with the Department of Management of Information Science. He has a strong background in machine learning, image processing, and computational intelligence.
NADIRE CAVUS is currently a Lecturer and the Chairperson with the Department of Computer Information Systems, Near East University, Turkey. She has got many scientific articles published by the worldwide famous journals. She is also in the editorial board and advisory board of several scientific journals. She also acts as a referee to these journals. Her research interests include information systems, health systems, networks, e-learning systems, and learning management systems (LMSs). He is the author of more than 50 articles, conference papers, and book chapters. His research interests include AI, machine learning applications, deep learning, image processing, and computer vision. VOLUME 9, 2021


extfieldsofstudy:
Physics
Computer Science

text:
Signal Detection and Identification in an Optical Camera Communication System in Moving State

Visible Light Communication (VLC) technology uses light-emitting diodes (LEDs) as a signal transmitting source to simultaneously realize lighting and data communication, and the spectrum resources are abundant. Since the visible light communication system with a photodiode (PD) as the receiver has some weaknesses, such as narrow field of view (FOV), inaccurate alignment in moving state, and inadaptability for indoor communication environments, an optical camera communication (OCC) system based on target detection algorithm is proposed in this paper. The system applies a LED array mounted on a moving AGV (automated guided vehicle) as the signal transmitter and a CCD camera as the receiver to realize data transmission in an outdoor environment. In this paper, a VLC MIMO channel model is constructed to verify that simultaneous communication of multiple LEDs through MIMO channels can effectively reduce bit error rate. The designed OCC system utilizes an image processing algorithm based on target detection to detect and identify light signals, realizing relatively stable and reliable communication in the state of motion. The distance from the signal transmitter to the receiver ranges from 6m to 14m.

Introduction
As wireless communication has been widely applied, there is a potential shortage in radio frequency resources to meet the current need. Visible light generally has a wavelength of 400nm-780nm and a frequency of 380THz-750THz. With abundant spectrum resource and unregulated frequency bands, visible light communication has great development potential. High-intensity light emitting diodes (LEDs) are high in luminous efficiency and strong in energy conservation and durability, and are thus gradually replacing conventional incandescent light bulbs in a wide range of fields, such as indoor and outdoor lighting, traffic lights, automobile lights and sign boards. The popularization and wide application of LED drives the development of visible light communication, which fosters both illumination and communication [1][2] [3] .
Visible light communication (VLC), a wireless communication technology, does not require transmission media such as optical fiber or physical lines. It utilizes visible light to transmit information and data, instead of occupying radio frequency resources. Communication systems based on visible light commonly use LEDs with high response sensitivity as the transmitter signal source, and loads signals on the LED-driven circuit through intensity modulation [4] . Zhang Long et al. proposed a VLC system utilizing one single RGB LED as signal source [5] , which receives optical signals by capitalizing on a CMOS rolling shutter camera. The receiver of a VLC system is generally a photodiode (PD) which converts optical signals into electrical ones and realizes point-to-point wireless communication. Such systems tend to require pretty high power for long-distance communication, and are thus more  [6] . Liu Ying et al. analyzed the application of orthogonal frequency division multiplexing (OFDM) and carrierless amplitude phase (CAP) modulation in VLC systems [7] , which can effectively improve BER performance and increase spectral efficiency.
However, VLC systems with PD as receiver have a very narrow field of view (FOV) and are susceptible to the disturbance of other optical signals. The transmitter and receiver need to be accurately aligned, and thus such systems are not ideal for mobile environments. To increase the signal-to-noise ratio of VLC and broaden the receiver's FOV in mobile environment, some researchers from home and abroad considered optical camera communication (OCC) systems using imaging sensor as the receiver, based on the technology of multiple-input-multiple-output (MIMO) wireless communication [8] . OCC technology inherits the advantages of VLC and digital image processing technology, realizing a pixelated MIMO VLC model that filters out noise through optical imaging of the image sensor [9] .
The application of MIMO technology realizes simultaneous transmitting and receiving of parallel multi-channel signals, which effectively enhances the utilization of the light source and reduces fading channels through spatial multiplexing. Since a single LED's power is limited and illumination intensity generally ranges from 300-1,500 LM, the communication distance is greatly limited. By forming a visible light MIMO system deploying multiple LEDs to create an environment with high signal-to-noise ratio, the transmission distance is increased. An OCC system may utilize several LED arrays as the signal source to transmit signals parallelly, capture optical signals through pixels on the image sensor, and realize the diversity reception of source information over MIMO system, thus identifying signal sources at different locations in a space [10] . In this paper, we discuss simulation results of MIMO VLC channels and study how MIMO VLC systems based on imaging sensor as receiver realize simultaneous transmission and reception of multi-channel signals. To filter background noise while the transmitter is in the state of motion and identify effective optical signals from the LED light source, a signal detection model based on adaptive target detection algorithm for OCC systems is proposed. The designed OCC system applies LED array as transmitter and Basler CCD camera as receiver to identify and decode optical signals in moving state.

The Model of OCC System and LED Light Intensity Distribution
2.1. OCC system architecture The designed OCC system uses a LED array as transmitter and a CCD camera as receiver. The architecture framework is presented in Figure 1. First, raw binary data is input into the microcomputer of the transmitting end and turned into parallel multi-channel signals through serial-parallel conversion. Then, after coded modulation, the signals are loaded on a LED-driven circuit and gated to LEDs at corresponding locations in the array through electrical addressing. Lighted LEDs, as the light source, transmits optical signals to free-space optical channels. Through on-off keying (OOK) modulation, the luminous intensity of LEDs changes at high speed to transmit information. Since light travels in straight line (rectilinear propagation of light), the multiple parallel light paths going from the LED array can be deemed as MIMO channels. The imaging sensor at the receiving end captures optical signals through densely-distributed PDs on it and outputs current. The current intensity changes in line with luminous intensity [11] . The system's receiver is acA1300-30gc Basler ace camera composed of lens and imaging sensor chip. The control chip of the camera system uses control signal circuit in the photosensitive component to control the current generated from PDs. The signals are then output through the current transmission circuit. CCD collects all electrical signals generated from imaging and exports them to the amplifier. The obtained signals will be stored in the imaging plane in the form of pixel gray value. As shown in Figure 2, the light emitted from a LED is projected through the lens on an imaging plane composed of pixel blocks. The information contained in one obtained frame of image includes the object's spatial information and its luminous intensity in the space. The size of a LED projection on the imaging plane depends on the number of pixels. Each lighted LED at least covers one pixel.

Figure 2. Projection of a LED on the imaging plane
When the transmitter is a LED array composed of multiple LEDs, the lighted LEDs and the camera's imaging plane form multiple independent free-space communication links, which determine that every object in the field of view shall occupy a certain pixel region in the imaging plane, and the area covered by pixels and spatial coordinates can be identified through image processing. In this way, even if the transmitter moves at a certain speed against the receiver, the light source still can be detected and located in the imaging plane. Then, upon obtaining coordinates of the light source, image information can be identified and decoded.

Line-of-sight (LOS) links of VLC systems
If we use a Lambertian model to describe the LED as transmitter, it is a Lambert radiator, whose luminous intensity's spatial distribution accords with Lambert's cosine law. A typical point-to-point VLC link is shown in Figure 3. There are two ways for optical signals in VLC systems to travel from the light source to the receiver, i.e. line-of-sight propagation (LOS) and non line-of-sight propagation (NLOS).

Figure 3. LOS link in visible light communication
As the OCC system uses imaging sensor as receiver and the imaging only considers available signal sources in the field of view, we only consider LOS links in the OCC system. The transmission function of optical signals emitted from the LED light source and going through LOS channels is: where A refers to the effective receiving area of the photoelectric detector of the receiver, d is the distance from the transmitter to receiver, and ( ) t R  is the radiant intensity of LED, which is: where m is Lambertian order ( 1/2 ln 2 ln(cos ) m    ), and 1/2  refers to half-power angle. The luminous pattern of the OCC system's light source, i.e. a LED array composed of multiple LEDs, is shown in Figure 4. After the imaging sensor captures image signals, the distribution of luminous intensities of corresponding LEDs in the imaging plane can be obtained，which is shown in Figure 5.

VLC MIMO channel model
Since one single LED's power is limited and cannot support long-distance data transmission, our OCC system uses several LEDs at the transmitting end. Each LED is connected to the receiver through an independent LOS link, and thus the LED array is connected to the imaging sensor's plane through multiple links, as shown in Figure 6. These independent channel links can be regarded as a certain VLC MIMO system. MIMO is a wireless communication technology that uses multiple antennas at both the transmitting and receiving ends. VLC MIMO system splits data up into several segments and transmits different data through different antennas independently. At the same time, the receiving end combines mixed information from diverse channels, distinguishes data from parallel data streams and restores raw information by utilizing the different fading characteristics of spatial channels [12] , thus acquiring higher data transmission rate while consuming the same amount of frequency resource. The VLC MIMO channel model constructed through experiment is illustrated in Figure 6. We suppose there are 8 LEDs in the LED array as transmitter of the VLC MIMO system and 8 PDs at the receiving end. The LED array and the PD array at the receiving end are parallel. Every link from LED to PD is an independent channel. In such a system, ( ) y t is the signal matrix at the receiving end: where r is the opto-electronic conversion factor of light sensors on the imaging sensor at the receiving end; ij H is the transmission coefficient of the subchannel from LED j to PD i . All subchannels are independently and identically distributed.
where  refers to the emission angel of the LED light source,  is the incidence angle of PD receiving optical signal, and ( ) g  refers to the gain of light passing through the imaging lens. ij H = 0 when the light source is out of the field of view. We use MATLAB to simulate communication systems with 2*2 and 2*1 antenna structures. For different antenna structures, we introduce convolutional error correcting code to the transmitting end for channel coding. The signals pass through additive white noise channels that subject to Gaussian distribution and then go to the receiver for maximum likelihood judgment and Viterbi decoding. Data is then output from the receiver. Our simulation results are shown in Figure 7. The 2Tx-2Rx configuration is slightly lower than the 2Tx-1Rx configuration in bit error rate when the signal-to-noise ratio (SNR) is approximately 5dB. However, when SNR is above 10dB, 2Tx-2Rx system that applies channel coding has a significantly lower bit error rate than other systems. MIMO uses multiple antennas to transmit information simultaneously, which realizes space multiplexing and space-diversity reception, effectively enhances channel reliability and decreases bit error rate.

Synchronization between transmitter and receiver
To guarantee reliable communication, the transmitter and receiver need to be synchronized in a VLC system. The data frame the transmitter sends shall contain one set of synchronized signals, which is a unique bit combination that is easy to be identified and decoded. In the designed OCC system, the LED array at the transmitting end flickers in a special way, and the signal is marked as a flag frame. To detect and locate the LED zone in an OCC system based on LED array, the flag frame representing the beginning of signal reception can be set as the LED array with four corners lit up, as a mark symbol. This pattern (with four corners lit up) doesn't serve the function of data transmission, while the other LEDs transmit information through being turned on and off. Since OOK modulation is adopted at the transmitting end, a LED turned on transmits the code element "1", and a LED turned off transmits "0". Serial input of a data stream composed of "0" and "1" at the input end, after finishing serial-parallel conversion and passing a LED-driven circuit, result in corresponding Ons and Offs of LEDs in the array. The receiving end verifies the synchronization between transmitter and receiver through detecting the flag frame periodically.

Quick detection and locating of the LED zone
To simulate real communication conditions, the transmitter and receiver can either be fixed or moving independently. Thus, the imaging sensor at the receiving end shall be able to identify the LED array's location quickly from certain frames and keep locating the array in subsequent frames while continuously decoding the transmitted information.
The method to locate the LED array is to add a special, easily recognizable image sequence segment into the image sequence of the LED array transmitted from the signal source. As shown in Figure 8, by using the aforementioned flag frame, i.e. four lit-up corners, the LED array zone can be accurately located. After synchronizing the transmitter and receiver, the OCC system with CCD camera as receiver will pre-process loaded image information in two steps, histogram equalization and adaptive threshold segmentation. The obtained image information will only be saved. Then, through recording multiple continuous frames and detecting necessary optical signals in background noise, the system will locate the LED array zone in the imaging plane and identify and decode LED image information through the support of target detection algorithm. The receiver works as follows: a) Image preprocessing. b) LED array detection and locating. c) LED array signal extraction. d) Judgment (whether the image frame is tilted). e) LED signal decoding.  Figure 9. Signal detection and identification

Inter-frame difference algorithm based on adaptive threshold segmentation
Since the receiver of the OCC system receives continuous frames, there are relative changes in the position of a moving target object in consecutive frames, and in the luminous state of LEDs. Inter-frame difference algorithm is used in target objection. It is applied to obtain the profile of a moving target through computing the difference between consecutive frames in a video image sequence. In an OCC system, inter-frame difference algorithm can be applied to perfectly detect the moving target through locating changes in the LED array's relative position in different frames, thus detecting and identifying optical signals.
The inter-frame or three-frame difference approach computes the difference between two or three frames. By subtracting the same information while saving different information, the difference between two images can be revealed. The inter-frame difference approach works as follows: First, a differential image is calculated from two consecutive gray images by subtracting gray values of the two frames. Then, binarize the differential image and set a binarization threshold [13] . If the subtracted result is very low in value, the differential image can be deemed as the background. If the value is high, the differential area can be regarded as the transmitter (LED array) in the state of motion, and needs to be marked. The location of the LED array can be identified from analyzing the marked pixel zones.
By applying the inter-frame difference approach, compute the difference between Image k (P k ) and Image k-1 (P k-1 ) and obtain a binary image. T refers to the preset threshold, which depends on the distribution of grey values in the image. By dividing images into two categories by applying T, images with grey value higher than T are regarded as foreground (i.e. the target LED array), while those with grey value lower than T are background. If T is set too high, the detected target may be hollow, and the target profile may be incomplete. If T is too low, there will be pretty much noise information. Since the illumination of adjacent LEDs in the LED array will result in overlayed luminous intensities, using single threshold method to process images with unevenly distributed illumination will result in relatively huge error. Thus, we apply adaptive threshold approach to determine the T value in our OCC system: Figure 10. AGV-mounted transmitter By conducting morphological operations such as logic and operation, and dilation and erosion, over the obtained two difference results, the target image is acquired as shown in Figure 11. The LED array can be extracted from the background according to coordinates L X , R X , T X and B X . Figure 11. Detected LED array

Image information decoding
A single LED may cover multiple pixels in the imaging plane of the imaging sensor. Through applying the approach of connected components labeling, connected components labelled as finished are analyzed to identify the lighted LED and corresponding coordinates on the imaging plane are output. By going over all pixel points, adjacent pixels with the same value are labelled as a set. Equivalence pairs of the same set are then be labelled as a connected component. Repeat the process until all equivalence pairs are labelled as finished, i.e. all marks of pixel points are decoded. By using connected components labeling, the detected LED zone is decoded and the following binary data matrix is obtained. We can see that the results shown in Figure 12 are exactly the same as the raw data sent by the transmitter, as presented in Figure 4. According to experiment results, when the transmitter of our OCC system is moving at the speed of 25 km/s, the bit error rate is -4 6.5 10  , and the distance from the transmitter to the receiver ranges from 6m to 14m.

Conclusion
Considering the huge application prospects of visible light, the paper proposes an OCC system based on MIMO channel model, which expands the field of view and meets the requirement of relatively longdistance outdoor communication. Our OCC system based on LED array and CCD camera utilizes image algorithms related to target detection to code and decode optical signals and realize data transmission in the state of motion. In the future, more state-of-the-art coded modulation techniques and target detection approaches based on deep learning algorithms may be utilized to further enhance the speed of visible light communication.


Sequence-to-Sequence Lexical Normalization with Multilingual Transformers

Current benchmark tasks for natural language processing contain text that is qualitatively different from the text used in informal day to day digital communication. This discrepancy has led to severe performance degradation of state-of-the-art NLP models when fine-tuned on real-world data. One way to resolve this issue is through lexical normalization, which is the process of transforming non-standard text, usually from social media, into a more standardized form. In this work, we propose a sentence-level sequence-to-sequence model based on mBART, which frames the problem as a machine translation problem. As the noisy text is a pervasive problem across languages, not just English, we leverage the multi-lingual pre-training of mBART to fine-tune it to our data. While current approaches mainly operate at the word or subword level, we argue that this approach is straightforward from a technical standpoint and builds upon existing pre-trained transformer networks. Our results show that while word-level, intrinsic, performance evaluation is behind other methods, our model improves performance on extrinsic, downstream tasks through normalization compared to models operating on raw, unprocessed, social media text.

Introduction
Social media is a pervasive part of our modern lives and provides us with a rich source of information and insight into human behaviour. Usergenerated content has been a valuable resource for the research community, especially in the form of text, but it is notoriously noisy and non-standard. Models that operate on social media posts go beyond marketing and advertisement applications, and have the potential to impact real human lives through, for instance, detecting loneliness (Guntuku et al., 2019), stress (Winata et al., 2018), life satisfaction (Yang and Srinivasan, 2016), suicidal ideation (Matero et al., 2019;Cao et al., 2019), and mental health problems such as depression (Yates et al., 2017;Bucur et al., 2021a;Tadesse et al., 2019) and PTSD (Coppersmith et al., 2014;Amir et al., 2019).
Outside of a formal setting, users communicate freely in text form, resorting to abbreviations, slang or plain spelling mistakes or typos. Eisenstein (2013) further explored bad language on social media, in the sense of language that defies our expectation of good spelling, vocabulary and syntax. He identified several underlying factors for the cause of non-standard text: user illiteracy, length limits imposed by social media sites (i.e. Twitter), text input affordances (i.e. standard mobile keyboards or predictive entry), pragmatics (emoticons/emoji, abbreviations and expressive lengthening), and a social component. Nguyen et al. (2021) further explored the latter, concluding that some types of non-standard text have strong social meaning, and normalization could induce a loss of meaning.
However, it is well known that for most benchmark tasks, noisy/non-standard text has proven to be a real problem to NLP models, such as BERT (Kumar et al., 2020), trained on clean or curated data, but fine-tuned on tasks with noisy and inconsistent format.
To overcome this predicament, Eisenstein (2013) proposes two possible approaches: either domain adaptation or normalization. While domain adaptation is not specific to natural language processing, text normalization and cleaning have always been a central part of any modern text processing pipeline. Text normalization is the process of adapting an input text to a more standard form. It has proven to be effective in increasing performance on tasks such as POS tagging (van der Goot and Çetinoglu, 2021), dependency parsing (van der Goot, 2019a) and sentiment analysis (Mandal and Nanmaran, 2018). Naturally, most text normalization pipelines are based on supervised models, which require carefully annotated data. However, annotating a large corpus of text in multiple languages is often cumbersome and expensive, and some approaches rely on synthetically generating corrupted text (Dekker and van der Goot, 2020;Ma, 2019).
Commonly, approaches are based on word-level normalization. One of the most prominent methods is MoNoise (van der Goot and van Noord, 2017), in which the text correction pipeline is similar to a classic ranked retrieval. However, MoNoise operates at the individual word and uses a spelling correction module and a word embedding module. While word embeddings can be made to account for a specific sentence context, it is mostly discarded.
Different from current methods, we aim to perform text normalization at a sentence level. This approach has several advantages, compared to word or subword methods: i) it can be naturally framed as a sequence-to-sequence type problem, ii) it is more straightforward, as it requires only one module, as opposed to a multi-stage pipeline (i.e. complex candidate generation and ranking), and iii) the same model can be trained on multiple languages at the same time, without increasing in size and computational processing.
In this edition of The Workshop on Noisy Usergenerated Text (W-NUT), organizers propose the shared task of multilingual lexical normalization 1 , in which participants are required to perform lexical normalization on 12 different languages (van der Goot et al., 2021a).
As such, we use the state-of-the-art multilingual sequence-to-sequence transformer model mBART (Tang et al., 2020) and fine-tune it for our task. mBART is one of the first models that can be fine-tuned simultaneously on multiple languages without performance loss. We show that framing text normalization as a neural machine translation problem is a viable method for text normalization, improving performance on extrinsic, downstream tasks compared to models that operate on raw, unprocessed social media text. We made the code publicly available on github. 2

Related Work
The W-NUT workshop hosted a shared task on lexical normalization of user-generated content from English tweets in its first edition (Baldwin et al., 2015a). The task received from the competing teams two categories of submissions, from constrained (using only the training data provided by the organizers) and unconstrained systems (using other publicly available data or tools).
The best model, from Jin (2015), generated candidates from the most similar canonical forms from the training data evaluated with the Jaccard Index. A random forest classifier was used to predict the suitable canonical form from all the candidates using features such as support and confidence, string similarity, and part of speech tags. The model was a constrained system, suggesting that the quality of the proposed model is more important than using additional data and tools. Other approaches were based on conditional random fields (CRF) (Akhtar et al., 2015;Supranovich and Patsepnia, 2015;Akhtar et al., 2015) and recurrent neural networks (RNN) (Min and Mott, 2015;Wagner and Foster, 2015) among others.
Notably, MoNoise (van der Goot and van Noord, 2017) has long been considered state-of-the-art in lexical normalization. MoNoise is a normalization model using spelling correction and word embeddings for candidate generation and a feature-based random forest classifier for candidate ranking. It is a modular normalization system easily reusable and adaptable (van der Goot and van Noord, 2017). The model was at the beginning developed only for English text. Still, then it was later expanded for multi-lingual lexical normalization covering languages such as Dutch, Spanish, Turkish, Slovenian, Croatian and Serbian (van der Goot, 2019b).
The lexical normalization task can also be formulated as a machine translation (MT) task. The noisy user-generated content is the source language, and the canonical form is the target language. Veliz et al. (2019) compare the MT approaches for lexical normalization, focusing on statistical neural translation (SMT) and neural machine translation (NMT) and obtaining better results using the SMT method. Furthermore, the authors show that the SMT approach works better in a low-resource setting than an NMT approach which requires a lot of data.
With the rise in popularity of pre-trained language models for natural language understanding and natural language generation, their ability to perform lexical normalization was also studied. By transforming the task into a token prediction one, Muller et al. (2019) demonstrate that a BERT model can be used as a lexical normalization model in low resource settings.
We propose to make use of the latest transformer models that are capable of multilingual translation in a sequence to sequence manner, namely mBART (Tang et al., 2020). However, we do not perform translation between languages, but instead, we use mBART as a denoising autoencoder, i.e. translating from bad English to good English. This way, we take the whole sentence into consideration when correcting the text. Moreover, this method is more straightforward and can scale to multiple languages without increasing computational demands.

Data and Evaluation
We further describe the dataset for this task and evaluation procedures. MultiLexNorm Dataset The data provided by the organizers includes texts from 12 languages: Croatian, Danish, Dutch, English, German, Italian, Serbian, Slovenian, Spanish, Turkish and codeswitched data for Indonesian-English and Turkish-German, as seen in Table 1. Some examples from the training data are shown in Table 2. For some languages in the dataset, the capitalization (Caps column) is also corrected, and words are split or merged (1-N/N-1 column). The dataset comprises Twitter posts from all languages, but some languages also have texts from additional sources. For example, Danish also has texts from Arto, Denmark's first large-scale social media (Plank et al., 2020) and Dutch texts were also gathered from public Internet forums, and SMS messages (Schuur, 2020). W-NUT Evaluation Methodology The organizers of the W-NUT workshop propose two types of evaluation procedures: intrinsic, word-level and extrinsic, downstream task performance (i.e. dependency parsing). introduced by van der Goot (2019b) is proposed:

As intrinsic evaluation, the Error Reduction Rate
Because accuracy is hard to compare across datasets with different numbers of raw words which have to be normalized, the ERR proposes an evaluation metric that can be used to compare the performance of systems across multiple datasets. It is computed as accuracy normalized for the number of raw words normalized in the gold dataset.
A system that always keeps the raw words has an ERR score of 0.0, while a perfect system will have ERR precisely 1.0. The ERR has a negative value when the system normalizes more words with a wrong form than the correct canonical form.
However, one downside of the ERR is that it fails to distinguish between FP and FN. Thus, in the case of FP, the system may provide a correct normalization, even if the annotators did not normalize the raw word.
Further, two baselines are provided: Leave-As-Is (LAI) -the output is the same as the raw input, the normalization is not performed -and Mostfrequent-Replacement (MFR) -the output is the most frequent replacement from the training data. If the raw word is not found in the training set, no normalization is performed.
As a secondary evaluation, the organizers propose an extrinsic evaluation of the effect of normalization on the task of dependency parsing, previous research showing that lexical normalization improves the performance for this task (van der Goot, 2019a). A dependency parser is trained on
For evaluating on sentiment analysis (SMILE) and offensive language identification (OLID), we trained a simple word-level TF-IDF model together with a linear SVM with balanced weights. For SMILE, we report average macro F 1 score across 5 folds, and for OLID, we report macro F 1 score on the test set. Figure 1: Fine-tuning a mBART model for lexical normalization on all available languages. We use the same model for all languages simultaneously. Lewis et al. (2019) proposed BART in 2019, as a way to pre-train large-scale transformers for sequence-to-sequence tasks. Initially, the authors pre-trained an encoder-decoder transformer only for English, obtaining good results on multiple downstream NLP tasks. Further, mBART (Tang et al., 2020), follows the same procedure, but for multiple languages. The pre-training stage for both BART and mBART is akin to a denoising autoencoder, in which the model receives a noisy (in this case masked) sentence, and it learns to reconstruct it.

Method
While mBART is fine-tuned on multiple language pairs, it is pretrained monolingually, and is capable of acting as an autoencoder for the same language. In our case, we make use of a pretrained mBART on 50 languages 3 from the transformers library (Wolf et al., 2020), and employ a procedure similar to the pre-training stage: a noisy sentence is   Figure 1 showcases our fine-tuning procedure. We tried two different approaches for fine-tuning: Frozen Encoder and Frozen Decoder because, with a fixed encoder, the model suffers from the same OOV-type problems as a typical transformer. However, training with a fixed decoder allows the model to better adapt its representations to each language's noisy version while maintaining its generative properties. For both approaches, we train a single model for all languages. Moreover, we also trained a separate mBART for each language, monolingually.
Training details For all runs, we fine-tune mBART for 50 epochs, using a batch size of 256 and with a cyclical learning rate scheduler (Smith, 2015) that linearly increases the learning rate from 0.00001 to 0.0001 and back across 5 epochs. The workshop organizers provided both the training data and the validation data on most languages. We omit validation on languages where the validation data is missing. The training was performed on an NVIDIA RTX 2070 graphics card. Since the memory requirements of an mBART model are quite high, we employed gradient accumulation to increase the batch size. In addition, we employed early stopping when the validation loss increased for more than 3 epochs. Post-processing Since our model outputs a whole sentence directly, the word-level evaluation requires the noisy input words to be aligned to their normalized counterpart. This phase is essential for sequence-to-sequence text normalization, as bad alignments will reduce the overall word-level performance score, especially in the 1-N/N-1 languages. As such, for the post-processing phase, we aligned input words with their normalized counterparts based on the Levenshtein distance between them. We used a linear sum assignment on the distance matrix to perform matching. Additionally, we matched the capitalization between corrections and left links, hashtags, and user mentions as they are.

Results
We further showcase the results of the pretrained mBART models fine-tuned on the available data: firstly, we kept the transformer encoder fixed and trained only the decoder, and secondly, we kept the decoder fixed and trained the encoder. During this fine-tuning process, we trained a single model for all languages. Further, for the CodaLab submission, we fine-tuned multiple models, one for each language, in the "fixed encoder" regime. Intrinsic Evaluation Table 3 showcases intrinsic, word-level evaluation across languages. Our best model obtained an average ERR across languages of 10.65, corresponding to a separate mBART trained for each language, with the additional postprocessing described in Section 4. In our case, training multilingually did improve performance on some languages (i.e. DE, EN, NL, TR), but over-    all achieved lower scores, especially in the case for HR and SL. For those languages, the model severely diverged, and its output consisted only of repeating the first word in the sentence. As per our intuition, fixing the decoder results in better perfor-mance when compared to fixing the encoder: the model learns to adapt its representations to account for the noisy text.
However, since our method is sentence-based, perfect alignment between words is cumbersome, with many cases of mismatch between punctuation. Moreover, merging or splitting words for normalization is also not taken into account in the post-processing phase. Extrinsic Evaluation For extrinsic evaluation, we showcase the results for our best model in Table 4 on the dependency parsing downstream task from the workshop challenge. Even though our model is not in the top-performing models, the absolute difference in performance is minimal.
Moreover, we also evaluated the effect of lexical normalization on two other tasks -sentiment analysis on the SMILE dataset and offensive language identification on OLID (Table 5). We trained a word-level TF-IDF and a linear SVM with balanced weights for both datasets and reported a macro F 1 score. Our lexical normalization improves results on both these tasks, compared to modelling the raw, unprocessed social media posts. This is because lexical normalization results in a smaller vocabulary for the documents, allowing the SVM model to operate on smaller dimensional data. Moreover, this evaluation procedure is arguably more realistic, as it does not require accurate post-processing to precisely align noisy words with their corrected version and match punctuation. Discussion In Table 6 we showcase some examples for correct, incorrect and questionable text normalizations. The model is able to easily grasp contractions such as u → you and expressive lengthening such as Juhuuuuu → Juhu. However, more complex word abbreviations such as nvm are quite challenging to generate, as the model only outputs the first part (i.e. never). Moreover, code-switched languages are an inherent problem to our approach, as mBART is only trained to receive input from a single language and not code-switched. Interestingly, even though we specified the langauge code for German in the phrase "Why Germany says "nein", the model actually translates the English part into German: Warum Deutschland sagt "nein".
However, as the organizers have pointed out, there are inconsistencies in the training and testing data annotations. In some cases, some words are not normalized (i.e. jajajajaj / w/ / n) even though they were clearly lengthenings or contractions. Despite this, in some of these cases, our model was able to provide correct normalizations.
There also appears to be no correlation between training dataset size and final normalization performance. For example, in the case of Croatian, even though the dataset is the second largest, the performance is lower than for other languages. Thus, the lower performance in some languages may be a cause of the complexity of the language; for English, our model obtained the best results.

Conclusions
In this work, we presented a method to perform lexical normalization by fine-tuning a multilingual machine translation model on pairs of noisy and normalized sentences from various languages. We employed mBART, as it is currently the state-ofthe-art in transformer-based multilingual machine translation, allowing us to fine-tune on all available languages simultaneously. Furthermore, we used mBART as a denoising autoencoder and tuned it in a supervised fashion.
As opposed to current two-stage methods for word candidate generation and ranking, our approach is more straightforward. Moreover, it scales to multiple languages without increasing computational demand (i.e. not increasing vocabulary size, increasing search space and others). Evaluation results show that our method, even though it lacks behind current methods on intrinsic, word-level evaluation, improves performance on downstream tasks.
For future work, we aim to develop our method for better post-processing of the output and increasing augmentation levels -i.e. injecting more noise in the form of spelling mistakes, backwards translations etc. Moreover, since our method is supervised, the quality and quantity of training data play an essential role in the final performance. In this regard, we aim to explore ways to take into account inconsistent annotations. Ana-Maria Bucur, Marcos Zampieri, and Liviu P. Dinu. 2021b. An exploratory analysis of the relation between offensive language and mental health.