Probing into the Root: A Dataset for Reason Extraction of Structural Events from Financial Documents

This paper proposes a new task regarding event reason extraction from document-level texts. Unlike the previous causality detection task, we do not assign target events in the text, but only provide structural event descriptions, and such settings accord more with practice scenarios. Moreover, we annotate a large dataset FinReason for evaluation, which provides Reasons annotation for Financial events in company announcements. This task is challenging because the cases of multiple-events, multiple-reasons, and implicit-reasons are included. In total, FinReason contains 8,794 documents, 12,861 financial events and 11,006 reason spans. We also provide the performance of existing canonical methods in event extraction and machine reading comprehension on this task. The results show a 7 percentage point F1 score gap between the best model and human performance, and existing methods are far from resolving this problem.

Introduction
Why does the event happen? People are always eager to find the reasons for an event. Automatically extracting the causal explanations of the given events from texts is useful and important for common users and downstream applications. For example, in the financial domain, returning the reasons of a concerned financial event in an Information Retrieval system can free analysts from reading the enormous company announcements and help investors make financial decisions.
Previous work on event causality (Do et al., 2011;Riaz and Girju, 2013;Mirza and Tonelli, 2014;Caselli and Vossen, 2017) mainly focus on * Most of the work was done when the first author was a research engineer in the Institute of Automation, CAS. the identification of causal relations between two given events that are usually presented as event trigger words. However, in reality, users may only know a particular event happened but without knowing its mention or trigger in the documents, and they just wonder the reasons for it. Therefore, we propose a new task aiming at extracting the causal explanations of the given structurally presented events from document-level texts. Specifically, a Structural Event defined here is a structural description that contains all necessary roles for an event type. Such a description can completely represent a specific happened event in reality. For example, in Figure 1, the PLEDGE event has four predefined roles NAME, ORG, NUM, BEG to represent an occurred PLEDGE event. Then, our task is to extract the reasons of the structural events as textual spans from the document.
To investigate the solution for this challenging task, we construct a large-scale Chinese dataset FinReason 1 . Specifically, we automatically collect the formal financial documents with their corresponding structural events the same as Yang et al. (2018). Then, crowd workers are employed to annotate the reasons in the documents for each structural event. In order to guarantee annotation quality and high inter-annotator agreement (IAA), we set several annotation principles and define 3 types of possible causal explanations (MOTIVATION, CAUSE, ENABLE) as Reasons for the events in company announcements to guide annotators. Finally, there are 8,794 documents, 12,861 collected financial events, and 11,006 reason spans in total. The Cohen's kappa of annotations is 83.87%.
Moreover, to understand this task's difficulties, we regard this task as an Event Extraction (EE) or a Machine Reading Comprehension (MRC) task. We also try some canonical models, such as BiL-SRM+CRF (Ma and Hovy, 2016), and BERT (Devlin et al., 2019) on this task and set benchmarks. Empirical results show that this task is challenging, and there is still an overall gap of 7pp (percentage points) in the F1 score between the best model and human performance.

Related Work
Much NLP research has focused on identifying causality relations from text, including knowledge bases (WordNet (Miller, 1998), FrameNet (Baker et al., 1998) and ConceptNet (Speer et al., 2017)), semantic related evaluations (SemEval-2007 task 04 (Girju et al., 2007), COPA (Roemmele et al., 2011), RED (Ikuta et al., 2014)), and event-related systems (Beamer and Girju, 2009;Do et al., 2011;Riaz and Girju, 2013;Hu and Walker, 2017;Caselli and Vossen, 2017). These work tried to identify real-world causality in lexicons or texts from different aspects. However, they have found it is difficult to agree on if a causal relationship exists in reality due to the ambiguity of causality definition. Our dataset mitigates this problem by only identifying contextual causality and do not check with reality.
In addition, plenty of work also only identify context-level causal relationships, such as general causality detection tasks PDTB (Prasad et al., 2007) and BECauSE 2.0 (Dunietz et al., 2017), and emotion causality detection task ECA (Lee et al., 2010). Some work (Radinsky et al., 2012;Mirza and Tonelli, 2014;Zhao et al., 2017) also tries to identify the causal relations between events at the contextual-level. However, our task is different because we focus on extracting the reasons for well-defined structural events, which is more close to practice scenarios.

Task Description and Data Collection
Task Description Our task is to extract the corresponding causal explanations for given structural events in a document. The inputs are a document with corresponding structural events described in it. The outputs are the causal text spans for the given events. For a given event in the document, there may be zero, single, or multiple causal explanations that need to be identified. To construct this dataset, we first collect a corpus of structural events with their corresponding documents following Yang et al. (2018). The collected documents are constrained to company financial announcements, which are relatively formal documents. Such a setting could improve annotation IAA because of the logical consistency and clarity. In specific, we crawl the public company financial announcements as documents from sohu.com 3 and the structural events from eastmoney.com 4 . Since the documents are not in line with their corresponding structural events, we leverage key event items (see more details in Appendix B) matching to align them. Same as Yang et al. (2018), we assume that if the key event items of a structural event appear in a document, the document mentions the target structural event. This alignment method has a high precision of 94.5% as evaluated by Yang et al. (2018).

Data Collection
In total, as in Table 5, we align 8,794 documents with corresponding 12,861 structural events of 3 types in financial domain, namely Pledge of Shares ( Pledge), Overweight and Underweight of Shares (O/U), Lawsuit and Arbitration ( Lawsuit). To the best of our knowledge, this is the largest dataset in the event reason extraction task.

Event Reason Annotation
Annotation Principles: To construct a corresponding dataset with high IAA, we follow the two principles in the annotation. First, we annotate the event reasons according to the contextual expressions. We do not check with reality, even if it is obviously a false statement (e.g., the stock market falls because of intense sunspot activity). Second, we specifically define 3 types of possible causal explanations as reasons for the financial events in announcements following previous work (Trabasso et al., 1989;Van den Broek, 1990;Dunietz et al., 2015): MOTIVATION, CAUSE, ENABLE (see details in Appendix A). This provides a clear guideline to annotators to decide what to annotate and what not to. Because it is also ambitious to differentiate those reason types from texts (Dunietz et al., 2015), we do not require the annotators to distinguish them but just require them to confirm that the reasons annotated at least belong to one of the 3 types. Quality Control: Besides the aforementioned 2 principles, we adopt several more rules to control data quality as follows.
(1) Each member should find as many reasons for a target event as possible.
(2) Each reason annotated should be as short as possible but with complete expressivity. (3) When explicit causal relation terms such as 因为 (because), 为了 (in order to) are mentioned, they should be included in the annotated reasons. (4) For each reason annotated, the annotator should confirm it by doing a why test (Grivaz, 2010), which means the reason should answer the question why the event happened.
Then, we employ crowdsourcing to annotate the reasons for each event. Specifically, 9 workers are divided into 3 teams to annotate each event type separately. Each team is trained to acquire the domain knowledge of the target event type so they can figure out the possible reasons for the events. Within each team, 2 members are responsible for annotating the reasons independently, and the 3rd member will be activated to make a judgment when 2 annotators have inconsistent annotations. Be-cause the alignment in the first step may not be perfectly accurate, annotators are also responsible for removing those wrongly aligned cases in the annotation to maintain data quality. Finally, as shown in Table 5, there are totally 8,794 documents, 12,861 collected financial events and 11,006 annotated event reason. And, approximately 73.58% of the documents are annotated with event reason. The Cohen's kappa of IAA is 83.87%.  From the annotation results, we could briefly conclude that extracting the reasons for given structural events in a document is not an easy task. First, a document may mention multiple events like the 2 events in the example of Figure 1. As in Table 5, approximately 20.67% documents mention more than one events. Without event mention assignment, discriminating the corresponding reasons for different events within the same document is difficult. Second, about 13.25% of documents mention multiple reasons for an event. Finding all reasons out is also not easy. Thirdly, 71.74% of the documents mention the reason for the events in an implicit way. There are only 28.26% reasons mentioned with explicit modifiers, like 因 为(because),由于(since),原因(cause),为(in order to),目的(aims to), etc. Such implicitly mentioned reasons are harder to be identified because they do not have any syntactic clue and require deep reasoning.

Task Challenges
We regard the average performance of the two annotators with respect to the final golden standard in the test set as human performance 5 . We can see in Table 2, the human performance on the test set is in line with intuition. Compared with simple cases (Single-Event, Single-Reason, Explicit-Reason), identifying reasons in multipleevent, multiple-reason, and implicit-reason cases are more challenging.

Evaluation Criterion
To evaluate the solution on this task, we follow a similar paradigm of SQuAD 2.0 (Rajpurkar et al., 2018) but also with several differences. In general, we get precision/recall/f1 scores of every event in the test set and calculate the macro-average of all events as the overall performance. However, there are multiple reason cases in FinReason, and we try to evaluate the ability of multiple reasons identification. As a result, we do not fully follow SQuAD-style evaluation by selecting the best prediction but considering all predictions to avoid the systems cheating by predicting all possible causal expressions in the text. For each case, we compute the scores as follows. 1) When there is no reason annotated for an event, the prediction should be Null string so as to get precision/recall/f1 scores of all 1; otherwise, all scores will be assigned 0.
2) When there is only one reason annotated for an event, we calculate the precision/recall/f1 scores based on the overlapping strings of prediction and ground truth. 3) When there are multiple reasons for a target event, we first calculate each reason's scores with corresponding predictions as in situation 2 and then calculate the macro-average of all reasons as the final scores for the target event.

Baselines
FinReaon is a new task but similar to several existing tasks, such as event extraction (EE) or machine reading comprehension (MRC). So we apply existing canonical methods for those similar tasks on FinReason as benchmarks for future research. The selected baselines are as followed (see more details in Appendix C): Regular Expressions (RegExp): In this setting, we regard the FinReason task as a causal sentence detection problem and employ some ad-hoc regular expressions to solve it. Specifically, we use five modifiers (因为(because),由于(since),原 因(cause),为(in order to),目的(aims to)) as causal clues to detect the sentence as the reasons for an event.
BiLSTM-CRF (BiLSTM): We can take the reasons as one part of the event description and regard the task as an EE task. Similar to Yang et al. (2018), we employ a BiLSTM-CRF (Ma and Hovy, 2016) to predict the start and end positions of each reason. Specifically, We simply get the event participants in the documents via string matching between the documents and the given structural events. Such information is used as features in a BIO tagging format. BERT-QA: We can take this task as an MRC problem if the structural event is regarded as a query and the target reason as the answer. In particular, we use templates to turn each structural event into a why-question and employ BERT-QA (Devlin et al., 2019)

Results
We split the dataset into train/dev/test sets with a ratio of 8:1:1 for experiments. From the results in Table 2, we can see that there is still an average of 7pp (82% vs. 89%) F1 score gap between the best model (BiLSTM) and human performance. Besides, we can see that human performance is relatively low for Lawsuit. This is because the reason for a lawsuit usually lies in a whole story between the plaintiff and the defendant, and it is hard to agree on the boundaries of the span. Furthermore, the BiLSTM model generally performs better than BERT-QA. The reason may be that the BiLSTM model knows the positions of event mentions by using the event BIO features, but BERT-QA only uses the structural event as the query. So it may be easier for BiLSTM to locate the correct reasons. Besides, we also evaluate the 3 challenges on the whole test set. As from Table 2, the average F1 gap between the best model and human for the 3 challenges are 12pp, 16pp, 28pp, respectively, which is much larger than the overall average gap of 7pp. This demonstrates that the challenges are also the bottlenecks of the models to reach comparable human performance, especially the implicit cases.

Conclusion
In this work, we propose a dataset FinReason for a new event causality extraction task. Our experiments show that this task is still challenging for current models. Future work may consider breaking the challenging cases (multiple-events, multiplereasons, and implicit-reasons) to achieve a more satisfying performance.


LIIR at SemEval-2021 task 6: Detection of Persuasion Techniques In Texts and Images using CLIP features

We describe our approach for SemEval-2021 task 6 on detection of persuasion techniques in multimodal content (memes). Our system combines pretrained multimodal models (CLIP) and chained classifiers. Also, we propose to enrich the data by a data augmentation technique. Our submission achieves a rank of 8/16 in terms of F1-micro and 9/16 with F1-macro on the test set.

Introduction
Online propaganda is potentially harmful to society, and the task of automated propaganda detection has been suggested to alleviate its risks (Martino et al., 2020b). In particular, providing a justification when performing propaganda detection is important for acceptability and application of the decisions. Previous challenges have focused on the detection of propaganda techniques (Martino et al., 2020a), based on news articles. However, many use cases do not solely involve text, but can also involve other modalities, notably images. Task 6 of SemEval-2021 proposes a shared task on the detection of persuasion techniques detection in memes, where both images and text are involved. Substasks 1 and 2 deal with text in isolation, but we focus on subtask 3: visuolinguistic persuasion technique detection.
This article presents the system behind our submission for subtask 3 (Dimitrov et al., 2021). To handle this problem, we use a model containing three components: data augmentation, image and text feature extraction, and chain classifier components. First, given a paired image-text as the input, we paraphrase the text part using back-translation and pair it again with the corresponding image to enrich the data. Then, we extract visual and textual features using the CLIP (Radford et al., 2021) image encoder and text encoder, respectively. Finally, we use a chain classifier to model the relation between labels for the final prediction. Our proposed method, named LIIR, has achieved a competitive performance with the best performing methods in the competition. Also, empirical results show that the augmentation approach is effective in improving the results.
The rest of the article is organized as follows. The next section reviews related works. Section 3 describes the methodology of our proposed method. We will discuss experiments and evaluation results in Sections 4 and 5, respectively. Finally, the last section contains the conclusion of our work.

Related work
This work is related to computational techniques for automated propaganda detection (Martino et al., 2020b) and is the continuation of a previous shared task (Martino et al., 2020a).
Taks 11 of SemEval-2020 proposes a more finegraind analysis by also identifying the underlying techniques behind propaganda in news text, with annotations derived from previously proposed propaganda techniques typologies (Miller, 1939;Robinson, 2019).
This current iteration of the task tackles a more challenging domain, by including multimodal content, notably memes. The subtle interaction between text and image is an open challenge for state of the art multimodal models. For instance, the Hateful Memes challenge (Kiela et al., 2020) was recently proposed, as a binary task for detection of hateful content. The recent advances in pretraining of visuolinguistic representations (Chen et al., 2020) lead the model closer to human accuracy (Sandulescu, 2020).

Methodology
In this section, we introduce the design of our proposed method. The overall architecture of our method is depicted in figure 1. Our model consists of several components: a data augmentation component (Back-translation), a feature extraction component(CLIP), and a chained classifier. Details of each component are described in the following subsections.

Augmentation Method
One of the challenges in this subtask is the low number of training data where the organizers have provided just 200 training samples. To enrich the training set we propose to use the back-translation technique (Sennrich et al., 2016) for paraphrasing a given sentence by translating it to a specific target language and translating back to the original language. To this end, we use four translation models, English-to-German, German-to-English, Englishto-Russian, and Russian-to-English provided by (Ng et al., 2019). Therefore, for each training sentence, we obtain two paraphrased version of it. In the test time, we average the probability distributions over the original and paraphrased sentenceimage pairs.

Feature Extraction
Our system isProbabilities of a combination of pretrained visuolinguistic and linguistic models.
We use CLIP (Radford et al., 2021) as a pretrained visuolinguistic model. CLIP provides an image encoder f i and a text encoder f t . They were pretrained on a prediction of matching image/text pairs. The training objective incentivizes high values of f i (I).f t (T ) if I and T are matching in the training corpus, and low values of they are not matching 1 . Instead of using a dot product, we create features with element-wise product f i (I) f t (T ) of image and text encoding. This enables aspect-based representations of the matching between image and text. We experimented with other compositions (Sileo et al., 2019b) which did not lead to significant improvement.
We then use a classifier C on top of f i (I) f t (T ) to predict the labels.

Chained Classifier
In this task, we are dealing with a multilabel classification problem, which means we need to predict a subset of labels for a given paired imagetext sample as the input. We noticed that label  co-occurrences were not uniformly distributed, as shown in figure 2. To further address the data sparsity, we use another inductive bias at the classifierlevel with a chained classifier (Read et al., 2009) using scikit-learn implementation (Pedregosa et al., 2011). Instead of considering each classification task independently, a chained classifier begins with the training of one classifier for each of the L labels. But we also sequentially train L other classifier instances thereafter, each of them using the outputs of the previous classifier as input. This allows our model to model the correlations between labels. We use a Logistic Regression with default parameter as our base classifier.
Our chain classifier uses combined image and text features as the input. We transfer the predicted probabilities of the classifier via the sigmoid activation function to make the probability values more discriminating (Ghadery et al., 2018). Then we apply thresholding on the L labels probabilities since the task requires a discrete set of labels as output. We predict a label when the associated probability is above a given threshold. We optimize the threshold on the validation set by a simple grid search using values between 0.0 and 0.9 with a step of 0.005.

Datasets
We use the dataset provided by SemEval-2021 organizers for task 6. The dataset consists of 687 (290)   is an image and its corresponding text. We use 10% of the training set as the validation set for hyperparameter tuning.

Results
In this section, we present the results obtained by our model on the test sets for Subtask 3. Table 2 shows the results obtained by the submitted final model on the test set. All the results are provided in terms of macro-F1 and Micro-F1. Furthermore, we provide the results obtained by the random baseline, the best performing method in the competition, and median result for the sake of comparison. Note that, we used the first released training set at the time of final submission which contained just 290 training samples. Therefore, we also provide results obtained by our model after using all the provided 687 training samples. Results show that LIIR has achieved a good performance compared to the majority class baseline and the median result which demonstrates that our model can effectively identify persuasion techniques in text and images. Also, we can observe LIIR has achieved a competitive performance compared to the best result obtained by the best team in the competition when it uses all the training samples.

Ablation Analysis
In this part, we provide an ablation study on the effect of different components of our proposed method on the dev set. First, we show the effect of using just visual features, just textual features, and both. Furthermore, we examine how well the final results of our model was influenced by the augmentation method. Table 3 shows the ablation study on the effect of using different features. The first observation is that image features contain more information compared to the textual features. Also, we can observe that the best Micro-F1 score is obtained when we combine both visual and textual features. These results show the effectiveness of our method in making use of both visual and textual information.  In Table 4, the effect of the augmentation technique is shown. As the results show, the augmentation approach is quite effective in improving the model performance by a high margin.

System
Macro-F1 Micro-F1 LIIR w/o Augmentation 0.25090 0.54952 LIIR w Augmentation 0.29972 0.58312 Table 4: Ablation analysis for the effect of augmentation method on the dev set.

Negative Results
We also tried to use CLIP as a zero-shot classifier for propaganda technique detection. To do so, we constructed prompts such as : ( For each input image/text, we generated a prompt for each labels, and used CLIP to estimate the estimate an affinity score between the prompt and the image. CLIP is designed to predict relatedness between the input image and text, and we expected that an input text mentioning the relevant propaganda technique should be associated with higher probabilities that the others.
However, this method did not seem to perform better than chance. This suggests that propaganda detection technique task might be too abstract for CLIP in zero-shot settings.

Conclusion
We described our submission for the shared task of multimodal propaganda technique detection at SemEval-2021. Our system performances that are competitive with other systems even though we used a simple architecture with no ensemble, by leveraging non-supervised learning. We believe that further work on zero-shot learning would be a valuable way to improve propaganda detection techniques for the least frequent labels.

Acknowledgments
This research was funded by the CELSA project from the KU Leuven with grant number CELSA/19/018.


Fractional-Order Colour Image Processing

: Many image processing algorithms make use of derivatives. In such cases, fractional derivatives allow an extra degree of freedom, which can be used to obtain better results in applications such as edge detection. Published literature concentrates on grey-scale images; in this paper, algorithms of six fractional detectors for colour images are implemented, and their performance is illustrated. The algorithms are: Canny, Sobel, Roberts, Laplacian of Gaussian, CRONE, and fractional derivative.

Introduction
Conventional edge detectors are based on integer derivatives. With the introduction of fractional calculus, some new non-integer edge detectors were created [1] and the existent integer ones adapted [2][3][4].
The aforementioned detectors were developed for grey-scale images. Since the input is often a coloured image, two options are available. One is to convert the image into grey-scale and then apply the detector. This may compromise the performance of edge detection. Thus, the other approach is to create colour-based edge-detection detectors [5,6]. This paper presents, in sequence for six different detection algorithms, three variants: conventional grey-scale integer detectors, grey-scale fractional detectors, and colour-based fractional detectors. The former two can be found in the literature; to the best of our knowledge, colour-based fractional detectors presented in this paper are the first ever used, with the exception of Canny, already given for integer orders, in [5]. While fractional derivatives are linear operators [7], the algorithms for image processing include other steps and operations, resulting in a non-linear treatment of the input data.
This paper is organised in the following manner. Section 2 explains the theoretical formulations for the edge detectors. Section 3 presents an application of the formulated methods to satellite images and the results which serve as an illustration of their performance. In Section 4 discussion of the results is made, and conclusions are drawn.

Grünwald-Letnikov Definition
The first-order derivative of a function is given by Iterating, it is possible to arrive to the nth derivative of a function. A general expression can be deduced by induction: Combinations of n things taken m at a time are given by The Grünwald-Letnikov (GL) derivative is a generalization of the derivative in (2). The idea behind it is that h should approach 0 as n approaches infinity. However, before doing so, binomial coefficients must cope with real numbers to extend this expression. For that, the Euler gamma function is used, instead of the factorials in (3): Combining (2) with (4) provides the main justification for the following extension of the integer-order derivative to any real-order α, which was proposed independently by Grünwald [8] and Letnikov [9]: For practical reasons, a truncation of the expression above was introduced, corresponding to an initial value c. This version of the derivative starting in c can be applied to functions that are not defined in the interval from −∞ to c: Expression (6) is the formulation that was applied to the different integer edge detectors in order to adapt them to fractional orders. The corresponding code is available from a public repository [10].

Derivative Filters
Derivative filters measure the rate of change in pixel value of a digital image. When filters of this kind are used, the result allows for enhancement of contrast, detection of boundaries and edges, and measurement of feature orientation.
Convolution of the specimen image with derivative filters is known as a derivative filtering operation. In most of the times, there is a filter to each direction; thus, convolution is performed twice. Using two dimensions, a gradient can be measured from the combination of the convolutions in x and y: The gradient points in the direction of the largest intensity increase. The magnitude (8) and orientation (9) of this gradient are frequently used to combine the two convolutions in the processing of images with edges with different orientations:

Canny Edge Detector
Original grey-scale: The Canny edge detector is a very popular edge detection algorithm. It was developed in 1986 by John F. Canny [11]. The algorithm is composed of the following steps: noise reduction, gradient calculation, non-maximum suppression, and hysteresis thresholding.
The first step of the Canny algorithm is noise reduction. Since image processing is always vulnerable to noise, it is important to remove or reduce it before processing. This is done by the convolution of the image f (x, y) with a Gaussian filter, defined as Then, a simple 2D, first-derivative operator (which, in the case of the algorithm used in this work, is the derivative of the Gaussian function used to smooth the image) is applied to the image already smoothed G(x, y, σ) * f (x, y). This highlights the zones of the image where first spatial derivatives are significant: Finally, the gradients in each direction E x and E y are given by This step of the process is called gradient calculation. After computing gradient magnitude and orientation with (8) and (9), a full scan is performed in order to remove any unwanted pixels which may not constitute edges (non-maximum suppression).
The final step of the algorithm is hysteresis thresholding. In this phase, the algorithm decides which edges are suitable for the output image. Each edge has an intensity proportional to the magnitude of the gradient. For this, two threshold values are defined, the minimum and maximum values. All gradients higher than the maximum threshold are considered "sure-edges". In contrast, the gradients that are lower than the minimum threshold are considered "non-edges". For the gradients that are between the two values, two instances may occur [12]:

•
If the pixels in question are connected to "sure-edge" pixels, they are considered to be part of the edges; • Otherwise, these pixels are also discarded and considered non-edges.
Fractional grey-scale: When adapting the grey-scale fractional Canny, the only change was that, instead of calculating the first-order gradient of the Gaussian kernel, the GL derivative was applied to the Gaussian function with the desired order α. This means that for each point of the Gaussian mask, its fractional derivative is obtained: where α k is given by (4).
After computing the fractional derivative, the algorithm follows the same steps of the conventional Canny, including non-maximum suppression and hysteresis thresholding.
Colour: In 1987, Kanade introduced an extension of the Canny operator [5] for colour edge detection. The operator is based on the same steps as the conventional Canny, but the computations are now vector-based. This means that the algorithm determines the first partial derivatives of the smoothed image in both x and y directions.
A three-component colour image assumes, for each of its points in the plane, a value which is a vector in the colour space. In the RGB space, which is a three-dimensional (3D) space to represent colour by a mixture of red (R), green (G), and blue (B), this corresponds to a function C = (R, G, B). It is possible now to define the Jacobian matrix, which is the matrix that contains the first partial derivatives for each component of the colour vector: Indexes x and y are used to represent partial derivatives: The direction along which the largest variation in the colour image can be found is the direction of the eigenvector of J T J that corresponds to the largest eigenvalue: In order to calculate the magnitude, one has to compute det(J T J − λI) = 0, which yields The orientation θ of a colour edge is determined in an image by After the magnitude is determined for each edge, non-maximum suppression is used. This eliminates broad edges, thanks to a threshold value.
According to the literature [13], even though colour edges and intensity edges are identical in over 90% of the cases, the former describes object geometry in the scene better than the latter.

Sobel Edge Detector
Original grey-scale: The Sobel operator measures the spatial gradient of an image. In this way, regions where there are sudden increases of pixel intensity are highlighted. Such regions correspond to edges.
The operator consists of two masks, one for each direction of x and y (G x and G y , respectively). Note that the mask, or kernel, for G y is nothing more than that for G x rotated 90 degrees [14]: Those edges that are vertical and horizontal in relation to the pixel grid cause a maximal response of these kernels. They can be applied to the input image separately. The resulting measurements of the gradient component in each direction (G x and G y ) are thereafter combined, so as to find at each point both the magnitude of the gradient and its orientation, using (8) and (9), respectively.
The resulting image gradient components can be expressed as Fractional grey-scale: Following the same line of thought, Yaacoub [2] presented a fractional Sobel operator.
Applying the GL definition (6) to G x (21), the fractional α-order derivative of G x yields This gradient is obtained convolving the image f (x, y) with a filter mask: Since the mask has an even number of rows, the origin is not centered. In the mask above, the origin is considered to be located on the fifth row, in the second column, shown in bold.
Similar reasoning can be applied to the y-direction. That is why, in this case, the mask in y is not the mask in x transposed.
According to the authors, this edge detector, compared with the conventional Sobel edge detector, resulted in thinner edges and reduced the number pixels of false edges.
Colour: A novel, colour-based, fractional Sobel was introduced by applying the same colour-based formulation of Section 2.3, only this time with the mask in (24). A Jacobian was constructed, and the largest eigenvalue of J T J computed; this allows for discovery of the direction in the image, along which the largest variation in the chromatic image function occurs.

Roberts Edge Detector
Original grey-scale: The Roberts Cross operator [15] is a simpler, quick way to find the gradient of an image. It also finds zones with great variations in pixel intensity that correspond to edges.
The Roberts operator consists of a pair of 2 × 2 masks, again, one for each direction. Here, the mask to compute the gradient in one direction is the other mask rotated by 90 • [14]: The combination of the two gradients in order to find the magnitude and orientation is performed once more using the above-mentioned expressions.
Fractional grey-scale: The authors of [3] presented the application of the GL derivative to the integer Roberts edge detector and arrived to a kernel for a fractional-order operator. It is known that the Roberts expression for the gradients stands: Combining (6) with (4), the authors arrived at expressions for the gradient's components: Referring to (27) and (28), the 3 × 3 fractional differential mask can be constructed in the eight central symmetric directions, viz. positive and negative x and y coordinates, and left and right downward and upward diagonals. The sum of the eight directional masks yields Combining the fractional mask with the Roberts operator defined by (26), the authors arrived at a solution for edge detection in which the texture of the image is enhanced and small edges are also detected. The mathematical formulation for this combination is where f (x, y) is the input image, and g(x, y) is the output image using an integer Roberts operator. From the experimental results in [3], and comparing this fractional algorithm with the original Roberts algorithm, it was concluded that edge detection was enhanced, while the thinner edges of the original algorithm are preserved.
Colour: The reasoning used to implement the colour-based Sobel operator was also used here. The fractional Roberts requires convolutions with two masks-first with the integer masks, and then with the fractional-derivatives operator. The colour-based vector convolution and Jacobian computations were applied only to the first one with the integer Roberts. Then, the output of this first integer colour-based edge detection serves as input to the fractional derivative operation.

Laplacian of Gaussian Detector
Original grey-scale: The Laplacian is a measure of the second spatial derivative of an image. It allows for the identification of zones where intensity changes fast. It is thus often used for edge detection. The Laplacian of a 2D image is given by In the discrete domain, the simplest approximation of the continuous Laplacian is the numerical first-derivative of the numerical first-derivative, yielding Substituting (34) and (35) in (33), the first kernel of (36) is obtained. The second, a non-separable eight-neighbor Laplacian defined by the gain-normalized impulse response array, was suggested by Prewitt. The third mask in (36) is a separable eight-neighbor version of the Laplacian [16].
To tackle sensitivity of second-order derivatives to noise, the image is smoothed with a Gaussian filter, and only then is the Laplacian filter reducing high-frequency noise applied. The smoothing filter can also be convolved first with the Laplacian kernel, and only then is the result convolved with the input image.
The Laplacian of Gaussian (LoG) operator of a 2D image is illustrated in Figure 1 and defined by [14] : LoG(x, y) = − 1 Fractional grey-scale: In 2014, the authors of [4] presented a fractional adaptation for the first operator in (36) (using the symetric mask).
In a discrete function ( f ), the operator corresponds to approximation Decomposing and noting that for this case, h = 1, (39) By generalizing the order from integer to fractional, a fractional-order differential form of the Laplacian operator can be obtained: Using the GL definition for the fractional-order derivative as it was used for the other operators, one may arrive at With the definition above, the mask that performs the calculation of the fractional Laplacian may be built: Experiments with (42) show that, the larger the order of differentiation is, the better the image feature is preserved, but the more noise that appears too.
Colour: A novel, colour-based, fractional LoG operator was implemented and tested in this study. The conventional algorithm finds edges searching for zero-crossings. This means that previous formulations cannot be adapted. According to [17], a pixel of a colour image is considered as part of an edge if zero-crossings are found in any of the colour channels. Thus, the fractional grey-scale operator formulated in [4] can be applied to each colour channel of a colour image. Then, a search for zero-crossings in the convolution outputs may be performed. If a zero-crossing is found in any of the channels, the corresponding pixel is flagged as part of an edge. The output of this algorithm is a binary image with all edges found.

CRONE
Original fractional grey-scale: In 2002, Benoît Mathieu wanted to prove that an edge detector based on fractional differentiation could improve edge detection and detection selectivity in the case of parabolic luminance transition.
The first derivative of a function f (x), calculated with increasing x, can be defined by with decreasing x, with h being infinitesimally small. A shift operator q is consequently introduced, defined by Using the shift operator on the directional derivative yields From the expressions above, it is clear that Generalizing to an order n, D − → n and D ← − n can be defined as As explained before, the bidirectional detector can be constructed by a composition of the two unidirectional operators using the following expression: Expanding (1 − q −1 ) n and (1 − q) n using Newton's binomial formula, the expression above can be rewritten: Applying the operator to a function, such as the transition studied before, where a k = (−1) k n k = (−1) k n(n − 1) · · · (n − k + 1) k! .
In order to detect edges on images, the formulated detector must be designed in two dimensions. Two independent vector operators for x and y, each of them a truncated CRONE detector, given respectively by +a m · · · +a 1 0 −a 1 · · · a m (57) are used for this purpose. The detector was experimented in artificial and real images, and performance compared with Prewitt operators. In all cases, the CRONE detector showed better immunity to noise. Colour: In this paper, a novel, colour-based, fractional CRONE was implemented, following the same steps as in the colour Canny already formulated. The colour channels are convolved with the masks for each direction constructing a Jacobian. Then, the maximum eigenvalue of J T J is computed in order to find the pixels where the variation in chromatic image is higher than a designated threshold.

Fractional Derivative Operator
Original fractional grey-scale: In the fractional Roberts, the fractional mask is used in addition to the conventional integer mask. In this study, the Fractional Derivatives mask in (29) was implemented individually as a fractional edge detector.
Colour: A new colour-based version for this detector was also implemented. In this case, there is only one mask to detect edges in eight different directions. Thus, the Jacobian is reduced to a vector (one dimension): The magnitude of the gradients was computed with the 3D vector formula:

Implementation
All edge detectors were implemented in MATLAB to perform fractional detection. A low-nebulosity image from ESA's Sentinel-2 satellite [18] is used below to illustrate performance. The image retrieved from the website was analysed, and a ground truth was manually taken using GIMP. Both the selected photograph and its corresponding ground truth are given in Figure 2. Additional results for a greater number of images from the same source are available in Excel files from our public repository [10]. Performance was checked with the usual metrics of which the first two are, respectively, the Jaccard coefficient and the Dice similarity coefficient. In (61)-(64), true-positives, false-positives, true-negatives, and false-negatives are defined as usual (see Table 1).

Discussion
Fractional-order adaptations of grey-scale edge detection algorithms are known to usually have better performance than the original method [19]. This happens in the example shown, where the results for Canny serve as an illustration: the integer algorithm achieves a Jaccard coefficient of 51.80%, the fractional method increases this result to 73.56%, and the colour version was able to almost entirely close the smooth inner-land gradients, achieving a Jaccard coefficient of 96.32%. Even when the increase in performance is low in percentage, since we are dealing with images that are composed of more than 120 million pixels, a 1% increase corresponds to more than one million correctly identified pixels. On the other hand, it is true that the colour-based detector is also heavier computationally, since it usually requires more than one convolution (at least one for each colour channel).
To summarize, in this study, seven novel fractional edge-detection methods were introduced (viz. one grey-scale and six colour-based versions). Future work includes a statistical assessment of their relative performance, in different types of images and applications, to statistically measure how their use improves performance.  Acknowledgments: The authors would like to acknowledge ESA, Copernicus for kindly providing access to the database for this research.

Conflicts of Interest:
The authors declare no conflict of interest.

Abbreviations
The following abbreviations are used in this manuscript:  Table 1  TP  True positive  Table 1


Wireless sensor network calibration technique for low-altitude unmanned aerial vehicle localization in paddy field

Received Apr 6, 2020 Revised Jun 13, 2020 Accepted Jul 31, 2020 This paper presents the use of the received signal strength indicator (RSSI) from the RF signal to estimate the distance from a point where the signal is transmitted to the point where the signal is received. This can be a challenge as in the paddy field, the watery and dry conditions, as well as the height of the paddy plant can affect signal transmission during this estimation process. Two low-cost ground beacons, Beacon1 and Beacon2 (The coordinator), are used and placed in a known location with a fixed distance across the paddy field, which becomes the reference point during the distance estimation for the unmanned aerial vehicle (UAV). These signals are analyzed by using the non-right-angle trigonometry computation, to estimate the distance between the transmitter and the receiver. The estimated distance is compared with the measured value to determine the efficiency of this approach. The calibration trendlines of these beacons in the open, watery and dry paddy fields are discussed and presented. It is found that the dry paddy field gives less RSSI mean error and proved that humidity can contribute to the distance estimation error.

INTRODUCTION
The breakthrough in wireless sensor network (WSN) has made the use of the RSSI signal to become the main focus in range measurement by the researchers [1] either for industry or domestic application [2,3]. This technique integrates the wireless network with sensors and other embedded systems that can give valuable information [4] either for indoor [5][6][7][8] or outdoor environment [9,10]. The WSN using the RSSI technique also increase the mobility and the coverage of the sensor's area [11]. The RSSI signal as a range measurement technique faces a huge challenge. The effect from the signal noise, signal refraction, reflection, multipath propagation and other interferences may have a great impact on the received RSSI signal [12]. Therefore, in real-time applications with the RSSI, the signal must have an accurate and reliable calibration technique to ensure that the received RSSI signal is a high-quality signal and stable [5].
A tracking strategy to calibrate the position of the XBee wireless adaptor has been proposed [12,13]. With the help of the trilateration algorithm and the least mean square (LMS) method, a grid for defining the possible locations of the XBee is built. Error correction is minimizing using the grid correction method [12,14]. The limitation of this technique is when the localization error is bigger than the grid step size, the correction strategy becomes inefficient. To overcome this issue, the environmental factor is introduced as the environmental coefficient in the transmitter [15]. Based on this technique, an initial estimated object location value must be given and the distances between the object location and the transmitter are used together with the environmental corrected distance between the transmitters and the object in estimating its localization [6]. The disadvantage of this technique is that there must be at least 3 transmitters to transmit the RSSI values to the object receiver. Interesting work has been done by implementing a fingerprint-based location determination using the cascade artificial neural network (ANN) [16]. In this method, the dimension area covered in the experiment is quite small and more works need to be done and adjusted to be readily implemented in a larger area. Using the fingerprint-based system, the algorithm requires too much computation effort but resulting a great efficiency. This technique requires a database to collect and process the fingerprint data. In another method, combined the fingerprint-based system is combined with an improved trilateration method that is called as a unilateral technique [17]. Even though this technique works very well, undeniably this technique requires a lot of WSN in the area for the system to function. The available systems as discussed previously are focusing on a very complicated system and use a lot of resources. On top of that, an excessive algorithm is needed to process the complex conditions. Therefore, in this paper, the RSSI based localization is introduced by using the minimum resources between the transmitter and the receiver. The RSSI signal on the wireless adaptor calibration is presented and evaluated and so that, the distance estimation by using this signal is obtained with a minimal error.

RESEARCH METHOD
In this research, a system based on the RF signal has been chosen. The system is controlled by using the Arduino Uno R3 microcontroller development platform. For the RF signal at the transmitter and receiver channels, the XBee wireless modules are used. The XBee modules are based on the IEEE 802. 15.4-2003 standard that is designed to work on the wireless network for point to point communications and star communications for baud rates of 250 kbit/s. The system is divided into 2 parts, namely, the former is set as the coordinator that is replicating the onboard UAV and the latter part is becoming the beacon. Each part is equipped with a set of Arduino Uno R3 microcontroller board with the XBee module.
The system architecture for the calibration technique is depicted as in Figure 1. Referring to this figure, the architecture of the system is developed by using two XBee Pro S1 modules, which one acts as the transmitter and the other one serves as the receiver. The transmit module is called Beacon1, and the other module, which calls the signal is called as Beacon2 or coordinator. The coordinator works as the main controller for the whole WSN architecture. The coordinator calls the beacon whenever it needs to read the RSSI signal. Both XBee modules in the coordinator and Beacon1 are controlled using the Arduino Uno R3 microcontroller.  Table 1 shows the configurations of the Arduino Uno R3 development board in this implementation. There are different setup for the coordinator and Beacon1. Basically, both of the controller boards are installed with the expansion Arduino board to easily interface with the Arduino development board with the XBee modules. As the whole system is not connected directly with any personal computer or laptop, therefore, a 16x2 array liquid crystal display (LCD) is used to show all the signals information on the related boards. To make possible for the wireless system configuration to be used wirelessly and portable, XBee S1 Pro module both the coordinator and Beacon1 use rechargeable batteries as the main power source. Table 1 shows the specifications of the Arduino Uno R3 microcontroller.
Referring to this table, the Arduino Uno R3 uses 5 Volts as its main power supply and supports the voltage needed by the whole system. The small current usage by the microcontroller platform gives the advantage to the UAV as the power supply is very critical for the UAV implementation. The XBee Pro S1 provides RF communications within the range of approximately 100 meters. It has the ability of WSN, which can be set point to point communication or mesh network [18]. The performance of the XBee Pro S1 is shown in Table 2.  According to Table 2, the signal can be received up to 1600 meters in the open space. The XBee Pro S1 board requires 3.3 Volts of voltage supply and it also has good receiver sensitivity. It can be set up using 2 types of configuration, either by using the Application Transparent (AT) mode or by using the Application Programming Interface (API) mode [19,20]. Using the AT mode, the XBee can do a simple network communication between point to point of two XBee modules directly. This feature is useful to read any wireless sensor value from a distance. In this work, the API mode is selected. This mode allows for more complex communications between 1 or more XBee modules [21]. The user can detect the ID of the transmitter together with the content of the transmitted data. To use XBee communication, firstly, it must be configured by using the XCTU software [22]. Table 3 shows the network configuration and setting for both XBee modules for the coordinator and Beacon1. The Arduino Uno R3 and the XBee Pro S1 are then put together in a plastic box for easy management and carry. Figure 2

The signal calibration step
The communication technique for system calibration is depicted in Figure 3. According to this figure, the whole calibration test is set up in the area that has a range of about 50 meters long. For a paddy field that has different field conditions, the tests were done in 3 different paddy field types, namely the dry paddy field, like a football field, a watery paddy field with the trees that were 1 meter above the ground and a dry paddy field with the matured paddy plant. In this setting, the coordinator XBee is the represents the UAV that will be moving freely on the paddy field. The XBee Beacon1 is the XBee module that has been placed on the ground at the start or the end of the paddy field. It acts as the on-ground beacon that stationed in that point of location and will transmit RF signal to the coordinator. The measurement is done in a straight-line measurement method. A measuring tape that has a length of more than 50 meters long is placed at the starting point in front of Beacon1. From this point, the measuring tape is pulled in a straight line forward from 0 meter to 50 meters. The coordinator and Beacon1 are placed on top of a tripod stand that has a height of about 1.3 meters from the ground to represent the low-altitude UAV platform. Beacon1 is placed at the start of the measuring tape or 0-meter point. The coordinator is put in front of Beacon1 at the range of 1 meter. Then, the coordinator is moved along the measuring tape and stopped at every 1 meter of distance. RSSI value received by the coordinator is recorded at every stop. The flow of the sequence on how was the coordinator read the RSSI value from the Beacon1 and change it into range estimation in centimeter is shown in Figure 4.
The coordinator stores 10 RSSI values before taking the average value to reduce the detection error. This method is used to filter any spike that disturbs the RSSI efficiency which was read by the coordinator. Thus, for every ten RSSI values, one average RSSI value is produced. For calibration purposes, this signal then will be saved manually by the user and then the value will be calculated using regression analysis in MS Excel to create distance measurement.

Regression analysis
After the RSSI value is collected, the analysis of the data must be completed before starts formulate the equation that can be used for the calibration. Regression analysis is utilized to find a trend in the collected RSSI data. It can provide equation and prediction about the group of data [23,24] and help to set accuracy to the data group. Figure 5 shows the process steps of the reference development technique for 1 beacon in one of the paddy field conditions. After building the graph according to the sequence in Figure 5, it is found that the graph is in the form of a logarithmic model. The logarithmic regression equations as given by the author in [25,26] can be defined in a general form of (1).
where, x = Distance (meter) Y = RSSI value From the graph, a trend line is plotted to normalize the available spiky form at different intervals of distance. The normalized value is in a modified form of the formula in (1). From this value, the mean of error is calculated to create a single reference value for the RSSI. 5 Figure 5. Process steps of the reference development technique for one beacon in one of the field conditions

Distance range calculation
The formula to calculate the range is found using the regression analysis as in (2). Using this formula, the distance between the coordinator and Beacon1 can be determined. This technique is used as the calibration value for a single beacon signal that is received by the coordinator.

RESULTS AND DISCUSSION
The RSSI signal calibration tests were executed to all 3 paddy field conditions that were mentioned earlier. A measuring tape is installed on the ground as the measurement guideline, which facilitates the comparison between the measured value and the estimated value obtained from the coordinator. The coordinator XBee is moved along the measuring tape line. For every 1 meter of range recorded using the measurement tape, the RSSI reading at the coordinator XBee that has been transmitted by Beacon1 is observed and recorded manually. Each one test will measure the signal strength of every 1 meter for the range of 50 meters. For the reliability purpose, every 1 meter of the movement of the coordinator, five readings have been taken. A total of 250 readings have been taken over the 50 meters of coordinator movement. Figures 6-8 show the RSSI signal that has been received from the test in an open field, dry and watery paddy field conditions. From these figures, five RSSI test signals and their trend line using regression analysis formula that have been extracted from the graph can be seen. From these trend lines, the mean values are calculated.  Table 4 shows the reference value that has been used in calibration for the different types of fields. The field r1 represents the value of the RSSI signal and the calculated distance between the coordinator and Beacon1 in a meter. This formula is used in the Arduino Uno R3 algorithm to calibrate the distance between the coordinator and Beacon1. From these graphs, it shows that the distance estimation in the dry field has less error compared to the watery field by approximately 20 percent. This result is supported by the fact that humidity, which available in the watery field has disturbed the accuracy.

CONCLUSION
RSSI has been used as the medium of range measurement in this research. This calibration technique is targeted to be used as a measurement technique for measuring distance using the RSSI signal received by the XBee modules. The target of this research is to be deployed on a UAV platform that moves in a paddy planting field. In the end, the UAV can use this technique basically as their localization. Although a direct RSSI signal received can be used as range estimation, the effect of the environment on the received signals can confuse with the received signals. Using this calibration to the received signals, a more accurate and reliable range measurement can be obtained from the RSSI signals. Using a low-cost and lightweight component such as the Arduino Uno R3 and the XBee module will give the positive impact on the result. This technique is to help for more advance development base on the RSSI signal from the XBee in the future.


s2fieldsofstudy:
Computer Science
Linguistics

text:
mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs

Multilingual T5 pretrains a sequence-to-sequence model on massive monolingual texts, which has shown promising results on many cross-lingual tasks. In this paper, we improve multilingual text-to-text transfer Transformer with translation pairs (mT6). Specifically, we explore three cross-lingual text-to-text pre-training tasks, namely, machine translation, translation pair span corruption, and translation span corruption. In addition, we propose a partially non-autoregressive objective for text-to-text pre-training. We evaluate the methods on seven multilingual benchmark datasets, including sentence classification, named entity recognition, question answering, and abstractive summarization. Experimental results show that the proposed mT6 improves cross-lingual transferability over mT5.

Multilingual pretrained models are typically trained on multilingual unlabeled text with unsupervised language modeling tasks, e.g., masked language modeling (Devlin et al., 2019), causal language modeling (Conneau and Lample, 2019), and span corruption (Raffel et al., 2020). These unsupervised tasks are built upon large-scale monolingual texts. In addition, several studies propose cross-lingual tasks that utilize translation data from multilingual parallel corpora, such as translation language modeling (Conneau and Lample, * Contribution during internship at Microsoft Research. 2019), cross-lingual contrast (Chi et al., 2021a), and bidirectional word alignment (Hu et al., 2020a). Thanks to the translation data, the pretrained models produce better-aligned cross-lingual representations and obtain better cross-lingual transferability.
Recently, the multilingual text-to-text transfer Transformer (MT5; Xue et al. 2020) achieves stateof-the-art performance on several cross-lingual understanding benchmarks. MT5 inherits the benefits of T5 (Raffel et al., 2020) that treats every text processing problem as a text-to-text problem, i.e., the problem of generating some target text conditioned on the input text. Despite the effectiveness of MT5, how to improve MT5 with translation data is still an open problem.
In this paper, we present MT6, standing for improving multilingual text-to-text transfer Transformer with translation data. MT6 differs from MT5 in terms of both pre-training tasks and the training objective. We present three cross-lingual tasks for text-to-text Transformer pre-training, i.e., machine translation, translation pair span corruption, and translation span corruption. In the translation span corruption task, the model is trained to predict the text spans based on the input translation pair. The cross-lingual tasks encourage the model to align representations of different languages. We also propose a new objective for text-to-text pre-training, called partially non-autoregressive (PNAT) decoding. The PNAT objective divides the target sequence into several groups, and constrains that the predictions should be only conditioned on the source tokens and the target tokens from the same group.
We conduct experiments on both multilingual understanding and generation tasks. Our MT6 model yields substantially better performance than MT5 on eight benchmarks. We also provide an empirical comparison of the cross-lingual pre-training tasks, where we evaluate several variants of MT6 under the same pre-training and fine-tuning procedure.
Moreover, our analysis indicates that the representations produced by MT6 are more cross-lingual transferable and better-aligned than MT5. The contributions are summarized as follows: • We introduce three cross-lingual tasks for textto-text Transformer pre-training, which improves MT5 with translation data.
• We propose a partially non-autoregressive objective that pretrains the decoder to use more information from the source sequence.
• We provide extensive evaluation results of various pre-training tasks and training objectives.

Background on T5 and MT5
Multilingual text-to-text transfer Transformer (MT5; Xue et al. 2020) is the multilingual variant of T5 (Raffel et al., 2020) pretrained on the mC4 (Xue et al., 2020) dataset, which consists of natural text in 101 languages drawn from the public Common Crawl web scrape. The backbone architecture of MT5 is the simple encoder-decoder Transformer (Vaswani et al., 2017), which is trained in a unified text-to-text manner. In specific, text-based NLP problems are formulated as text-to-text transfer, i.e., the model is trained to predict the target text conditioned on the input source text. For example, in text classification, the model predicts the label text rather than a class index. This feature enables the MT5 to be fine-tuned with the same training objective for every task. Formally, let x and y denote the input sequence and the output sequence, the loss function of training the x → y transfer is where y <i = y 1 , · · · , y i−1 . With the unified textto-text formulation, the pre-training task can be designed by constructing the input and output text sequences. Specifically, MT5 employs the span corruption task as the pre-training task, which is an unsupervised masked language modeling task. As shown in Figure 1, we provide an example of constructing the input and output sequences for span corruption. Given a natural sentence s, it first randomly selects several spans of s as the spans to be masked. Then, the input sequence is constructed by replacing the selected spans with unique mask  Figure 1: Example of the span corruption task (Raffel et al., 2020) used in T5 and MT5. tokens. The output sequence is the concatenation of the original tokens of the masked spans, each of which starts with a unique mask token to indicate the span to be decoded. We denote the above two operations as g i and g o , standing for converting the original sentence s into the input or the output formats of span corruption. Thus, the loss function of the span corruption task can be written as

Methods
In this section, we first present three text-to-text pre-training tasks for improving MT5 with translation data. Then, we introduce the partially nonautoregressive decoding objective, and provide the detailed fine-tuning procedures for the classification, question answering, and named entity recognition tasks.

Cross-lingual Pre-training Tasks with Translation Pairs
As shown in Figure 2, we illustrate an overview of our cross-lingual text-to-text pre-training tasks. Given the same translation pair, the three tasks construct different input and output sequences.

Machine Translation
Machine translation (MT) is a typical text-to-text task with the goal of translating a sentence from the source language into a target language. It is a natural design to use MT as a text-to-text pre-training task for sequence-to-sequence learning (Chi et al., 2020). Let e and f denote a sentence and its corresponding translation. We directly use e and f as the input and output sequences, respectively. The loss function of MT is Thanks for your invitation last week .

Thanks for your invitation last week .
Merci pour votre invitation la semaine dernière .

Thanks for your invitation last week .
Merci pour votre invitation la semaine dernière .  Figure 2: Overview of three cross-lingual text-to-text pre-training tasks. For each task, we provide an example of the input and target text. The words marked with "×" are randomly replaced with unique mask tokens like [M 1 ].
Notice that in the translation span corruption task, we mask tokens only in one language.

Translation Pair Span Corruption
Inspired by the translation masked language modeling (Conneau and Lample, 2019) task, we propose the translation pair span corruption (TPSC) task that aims to predict the masked spans from a translation pair instead of a monolingual sentence. Let e and f denote a sentence and its corresponding translation.
We concatenate e and f as a single sentence, and perform the span corruption on the concatenated sentence. Formally, we construct the input and output sequences by

Translation Span Corruption
A potential issue of translation pair span corruption is that the spans in the target sequence can be organized in unnatural word order. As shown in Figure 2, the output sequence of TPSC is organized as It can be found that the French word "invitation" is after the English word "week", which could harm the language model of the decoder. This motivates us to propose the translation span corruption (TSC) task where we only mask and predict the spans in one language. Given a translation pair (e, f ), we randomly select the e or f to perform span corruption. Without loss of generality, we consider e as the sentence for span corruption. Then, the input and output sequences are constructed by [g i (e); f ] and g o (e), respectively. With the resulting input and output sequences, the loss function of TSC can be written as

Pre-training Objective: Partially Non-autoregressive Decoding
Recall that the predictions in MT5 are conditioned on both the source tokens and the target tokens to the left. When predicting the tokens closer to the end, the model can use more information from the target sequence, resulting in the insufficient training of the encoder. To encourage the model to utilize more information from the encoding side while preserving the ability of autoregressive decoding, we propose a new training objective for text-to-text training, called partially non-autoregressive decoding (PNAT). In Figure 3, we provide an example for PNAT. Specifically, given a target sequence containing several spans, we divide the target sequence into groups, and train the model to decode each group separately. With the PNAT objective, a prediction is only conditioned on the source tokens and the target tokens from the same group. Consider the target sequence consisting of m spans. We divide the spans into n g groups, each of which contains m/n g consecutive spans. For the j-th group, we denote l j and r j as the start position and the end position, respectively. The PNAT objective is defined as The text-to-text loss L(x → y) is a specially case of L PNAT (x → y) with n g = 1.
The MT6 model is jointly pretrained on both monolingual and parallel corpora, where we use the span corruption and one of the three cross-lingual text-to-text tasks. For both tasks, we use the partially non-autoregressive decoding as the training objective where we divide the target sequence into  n g groups. The overall pre-training objective is to minimize where L PNAT X stands for the one of the loss functions of machine translation (MT; Section 3.1.1), translation pair span corruption (TPSC; Section 3.1.2) and translation span corruption (TSC; Section 3.1.3), with PNAT as the training objective.

Cross-lingual Fine-tuning
We fine-tune all parameters of the MT6 model with Equation (1) regardless of the end task. Unlike language generation tasks, language understanding tasks should be pre-processed as the text-to-text format. We introduce how to convert the following three types of the language understanding task into the text-to-text format, i.e., constructing the input and output sequences from the original examples.
Classification The goal of the text classification task is to predict the label of a given text. Following T5 (Raffel et al., 2020), we directly use the label text as the output text sequence. We provide an example for the MNLI natural language inference task . Given an input sentence pair of "You have access to the facts ." and "The facts are accessible to you .", the goal is to classify the input into the relationships of "entailment", "contradiction", or "neutral". The input and target sequences are constructed as Input: bos You have access to the facts. eos The facts are accessible to you. eos Output: bos entailment eos Since multi-task fine-tuning is not the focus of this work, we do not prepend a task prefix in the input text. We also adopt a constrained decoding process, where the decoded text is constrained to be one of the labels.
Question Answering For the extractive question answering (QA) task, we concatenate the passage and the question as the input, and directly use the answer text as the target instead of predicting the answer span positions. We provide an example of converting a QA training example into the text-totext format.
Input: bos It has offices in Seoul, South Korea. eos Where is the office in South Korea? eos Output: bos Seoul eos We use the constrained decoding for the QA tasks where we use the tokens shown in the input passage as the decoding vocabulary.
Named Entity Recognition In named entity recognition (NER), we do not directly use the original tag sentence as the output. We find that the model tends to repeat decoding the "O" tag if the model directly learns to decode the tag sequences. Alternately, we construct the target text by concatenating the entity spans, each of which starts with the entity tag and ends with the entity tokens. We show an example of converting a NER training example into the text-to-text format.
Input: bos Italy recalled Marcello Cuttitta . eos Output: bos loc Italy sep per Marcello Cuttitta sep eos loc and per are entity tags denoting location and person. The sep tag means the end of entity span. We use the following constrained decoding rules: (1) The model should decode entity tags or the end-of-sentence tag ( eos ) after a bos token or a sep token; (2) Otherwise, the model should decode the tokens from the input sentence or the sep token for the other situations.

Setup
Data Following previous work on cross-lingual pre-training (Conneau et al., 2020;Chi et al., 2021a), we use the natural sentences from CC-Net (Wenzek et al., 2019) in 94 languages for monolingual text-to-text tasks. For crosslingual text-to-text tasks, we use parallel corpora of 14 English-centric language pairs, collected from MultiUN (Ziemski et al., 2016), IIT Bombay (Kunchukuttan et al., 2018), OPUS (Tiedemann, 2012), and WikiMatrix . Details of the pre-training data are described in Appendix.

Training Details
In the experiments, we consider the small-size Transformer model (Xue et al., 2020), with d model = 512, d ff = 1, 024, 6 attention heads, and 8 layers for both the encoder and the decoder 1 . We use the vocabulary provided by XLM-R (Conneau et al., 2020), and extend it with 100 unique mask tokens for the span corruption tasks. We pretrain our MT6 for 0.5M steps with batches of 256 length-512 input sequences. The model is optimized by the Adam optimizer (Kingma and Ba, 2015) with a linear learning rate scheduler. The pre-training procedure takes about 2.5 days on an Nvidia DGX-2 Station. Details of the pre-training hyperparameters are described in Appendix.

XTREME Cross-lingual Understanding
To validate the performance of MT6, we evaluate the pretrained models on XTREME (Hu et al., 2020b), which is a widely used benchmark for cross-lingual understanding. Following MT5 (Xue et al., 2020), we consider six downstream tasks included by XTREME: the named entity recognition (NER) task on the WikiAnn (Pan et al., 2017;Rahimi et al., 2019) dataset in 40 languages, the question answering (QA) task on MLQA (Lewis et al., 2020b), XQuAD (Artetxe et al., 2020), and TyDiQA-GoldP (Clark et al., 2020), the cross-lingual natural language inference task on XNLI (Conneau et al., 2018), and crosslingual paraphrase adversaries on PAWS-X . The models are evaluated under the cross-lingual transfer setting (Conneau et al., 2020;Hu et al., 2020b). Under this setting, the models should be fine-tuned only on English training data but evaluated on all target languages. Moreover, for each pretrained model, only one model is used for all languages rather than selecting fine-tuned models separately. Details of the fine-tuning hyperparameters are described in Appendix.
As shown in Table 1, we present the evaluation results of the pretrained models on the XTREME benchmark. We observe that MT6 achieves the best performance on XTREME, improving the average score from 45.0 to 50.4, as we go from MT5 to MT6. It is worth mentioning that pre-training the model only with the machine translation task performs even worse than MT5. We have noticed that several target languages in TyDiQA and WikiAnn are not covered by our parallel corpora. However, the NMT pretrained model still shows poor results on the other four tasks, where all target languages are covered by the training data. Detailed results can be found in Appendix.

Comparison of Pre-training Tasks
To provide a clear comparison among the pretraining tasks, we implement the text-to-text pretraining methods presented in Section 3, and pretrain variants of MT6 with the same training data and resources for fair comparisons. Table 1 compares the evaluation results of the models pretrained with seven different combinations of span corruption (SC), machine translation (MT), translation pair span corruption (TPSC), translation span corruption (TSC), and partially non-autoregressive decoding (PNAT). It can be observed that jointly training SC+TSC with PNAT achieves the best overall performance on the XTREME benchmark, with substantial gains over the models trained on monolingual data only. The same trend can be observed for the other models pretrained on both monolingual data and parallel data. This demonstrates that introducing translation data to text-to-text pre-training can improve the performance on the end tasks of cross-lingual understanding. Moreover, PNAT provides consistent gains over SC and SC+TSC, showing that PNAT is effective on both monolingual and cross-lingual tasks. Surprisingly, SC+PNAT obtains comparable results to SC+MT without any parallel data. Comparing TSC with MT and TPSC, we observe that SC+TSC brings noticeable improvements on question answering tasks. Although SC+MT shows competitive results on XNLI, the results on the other tasks are relatively low, indicating that simply jointly training SC with MT is not the most effective way to pretrain MT6.

Abstractive Summarization
Multilingual Summarization In addition to language understanding tasks, we also evaluate our MT6 model on the abstractive summarization task. Abstractive summarization aims to generate a summary of the input document while preserving its original meaning. We use the Gigaword dataset provided by Chi et al. (2020). The dataset is constructed by extracting the first sentences and head-  lines as the input documents and summaries, respectively. The dataset consists of examples in the languages of English, French, and Chinese. For each language, it contains 500K, 5K, and 5K examples for the training, validation, and test, respectively. We fine-tune the models for 20 epochs with a batch size of 32 and a learning rate of 0.00001. During decoding, we use the greedy decoding for all evaluated models. As shown in Table 2, we report the ROUGE (Lin, 2004) scores of the models on Gigaword multilingual abstractive summarization. We observe that MT6 consistently outperforms MT5 on all the three target languages. Comparing with the XLM (Conneau and Lample, 2019) and XNLG (Chi et al., 2020) models with 800M parameters, our MT6 model achieves a similar performance with only 300M parameters. Besides, under the setting with fewer training data, MT6 shows more improvements over MT5.

Cross-Lingual Summarization
The crosslingual summarization task aims to generate summaries in a different language. We use the   Wikilingua (Ladhak et al., 2020) dataset containing passage-summary pairs in four language pairs. We fine-tune the models for 100K steps with a batch size of 32 and a learning rate of 0.0001. We use the greedy decoding for all evaluated models. The evaluation results are shown in Table 3, where MT6 outperforms MT5 on the test sets of four language pairs.

Cross-lingual Transfer Gap
To explore whether our MT6 model achieves better cross-lingual transferability, we compare the crosslingual transfer gap scores of our MT6 with MT5. Cross-lingual transfer gap (Hu et al., 2020b) is defined as the difference between the performance on the English test set and the average performance on the non-English test sets. The transfer gap indicates how much the end-task knowledge preserves when transferring from English to the other target languages. Empirically, a lower transfer gap score indicates better cross-lingual transferability. Following Hu et al. (2020b), we compute the transfer gap scores over the sentence classification and question answering tasks. As shown in Table 4, MT6 consistently reduces the transfer gap across all the five tasks, demonstrating that our model is more effective for cross-lingual transfer than MT5.

Cross-lingual Representations
We analyze the cross-lingual representations produced by our MT6 model. Following Chi et al. (2021a), we evaluate the representations on the Tatoeba (Artetxe and Schwenk, 2019) cross-lingual sentence retrieval task. The test sets consist of 14 English-centric language pairs covered by the parallel data in our experiments. Figure 4 illustrates the average accuracy@1 scores of cross-lingual sentence retrieval. The scores are averaged over 14 language pairs and both the directions of xx → en and en → xx. From the figure, we observe that MT5 shows a parabolic trend across different layers, which also appears in other cross-lingual encoder models (Jalili Sabet et al., 2020;Chi et al., 2021a). Differently, we obtain better performance  Table 5: Evaluation results on word alignment. We report the alignment error rate scores (lower is better). We use the hidden vectors from the last encoder layer, and apply the SimAlign (Jalili Sabet et al., 2020) tool to obtain the resulting word alignments.  Table 6: Effects of noise density. We report the average results over different task types and the average results over all the six tasks on the XTREME benchmark. We vary the noise density of the translation span corruption task from 15% to 100%. All results are averaged over five runs.
as we use higher layers of our MT6 model. At layer-8, our MT6 model achieves an average ac-curacy@1 of 43.2, outperforming the MT5 model by 35.6, which means our MT6 model produces better-aligned text representations. We believe the better-aligned representations potentially improve the cross-lingual transferability. Furthermore, the results also indicate that our pre-training objective is more effective for training the encoder than MT5.

Word Alignment
In addition to cross-lingual sentence retrieval that evaluates sentence-level representations, we also explore whether the representations produced by MT6 are better-aligned at token-level. Thus, we compare our MT6 with MT5 on the word alignment task, where the goal is to find corresponding word pairs in a translation pair. We use the hidden vectors from the last encoder layer, and apply the SimAlign (Jalili Sabet et al., 2020) tool to obtain the resulting word alignments. Table 5 shows the alignment error rate (AER) scores on the test sets provided by Jalili Sabet et al. (2020). Among the three language pairs, MT6 achieves lower AER scores than MT5, indicating that the cross-lingual representations produced by MT6 are also betteraligned at token-level.

Effects of Noise Density
In the translation span corruption (TSC) task, the input parallel sentences provide redundant information in two languages, which is different from the standard monolingual span corruption task. Thus, we explore the effects of noise density by varying the noise density in the translation span corruption task, with the other hyperparameters fixed. To reduce the computational load, we do not apply the partially non-autoregressive decoding, i.e., we pretrain the models with the original text-to-text objective. We pretrain MT6 models with the noise density of 0.15, 0.3, 0.5, and 1.0 respectively. It means 15%, 30%, 50%, or all of the source or target tokens are replaced with the masked tokens. Notice that setting the noise density as 1.0 is identical to machine translation, where the decoder is required to decode the whole target sentence.
In Table 6, we report the average scores on the XTREME benchmark. From the results, we observe that MT6 achieves the best results with the noise density of 0.5, rather than a higher noise density such as 1.0. The results indicate that the TSC task prefers a higher noise density, so that the model can learn to use more cross-lingual information. This finding is different from that reported by T5 (Raffel et al., 2020), where the span corruption task works better with the noise density of 0.15 under the monolingual setting.

Related Work
Cross-lingual LM Pre-training Cross-lingual language models are typically built with the Transformer (Vaswani et al., 2017) architecture, and pretrained with various pre-training tasks on largescale text data. Multilingual BERT (mBERT; Devlin et al. 2019) and XLM-R (Conneau et al., 2020) are pretrained with masked language modeling (MLM; Devlin et al. 2019) on large-scale unlabeled text in about 100 languages. MASS (Song et al., 2019) and mBART  are pretrained in an auto-encoding manner, which provides improvements on the neural machine translation tasks. MT5 (Xue et al., 2020) is pretrained with the span corruption (Raffel et al., 2020) task under the text-to-text formulation (Raffel et al., 2020). Cross-lingual pretrained models also benefit from translation data. XLM (Conneau and Lample, 2019) jointly learns MLM and the translation language modeling (TLM) task. Unicoder (Huang et al., 2019) presents three cross-lingual tasks to learn mappings among languages. ALM  converts the translation pairs into code-switched sequences as the training examples. Word-aligned BERT models (Cao et al., 2020;Zhao et al., 2020) improves the cross-lingual representations by fine-tuning the mBERT with the objective of minimizing the distance between aligned tokens. AMBER (Hu et al., 2020a) propose to maximize the agreement between the forward and backward attention matrices of the input translation pair. InfoXLM (Chi et al., 2021a) proposes the cross-lingual contrastive learning task that maximizes the InfoNCE (Oord et al., 2018) lower bound of the mutual information between the input translation pair. XLM-Align (Chi et al., 2021b) leverages token-level alignments implied in translation pairs to improve cross-lingual transfer. XNLG (Chi et al., 2020) introduces the cross-lingual transfer for NLG tasks, and achieves zero-shot cross-lingual transfer for question generation and abstractive summarization. VECO (Luo et al., 2020) pretrains a variable cross-lingual pre-training model that learns unified language representations for both NLU and NLG. ERNIE-M (Ouyang et al., 2020) utilizes the back-translation masked language modeling task that generates pseudo parallel sentence pairs for learning TLM.
Encoder-Decoder Pre-training Raffel et al. (2020) use span corruption to pretrain text-to-text Transformer, where both language understanding and generation tasks are formulated as sequenceto-sequence fine-tuning. Song et al. (2019) propose masked sequence-to-sequence pre-training where the model predicts a randomly masked span. BART (Lewis et al., 2020a) design various denoised autoencoding tasks to recover the whole original sentence. PEGASUS  introduces the gap sentence generation task for abstractive summarization pre-training. Chi et al. (2020) use both denoised autoencoding and machine translation for cross-lingual language generation. Another strand of research follows unified language model pre-training (Dong et al., 2019;Bao et al., 2020;Luo et al., 2020), where the encoder and the decoder share parameters. Ma et al. (2020Ma et al. ( , 2021 reuse pretrained multilingual encoder for sequence-to-sequence pre-training.

Conclusion
In this paper, we propose MT6 that improves the multilingual text-to-text transfer Transformer with translation data. We introduce three text-to-text pre-training tasks that are built on parallel corpora, and a training objective for improving text-to-text pre-training. Nonetheless, we present a comprehensive comparison of the text-to-text tasks, and show that our MT6 model outperforms MT5 on both cross-lingual understanding and generation benchmarks. For future work, we would like to pretrain MT6 models at a larger scale, and explore more applications, such as machine translation.

B Hyperparameters for Pre-Training
As shown in Table 9, we present the hyperparameters for pre-training MT6. We extend the vocabulary of the XLM-R (Conneau et al., 2020) with external 100 unique mask tokens as the vocabulary of MT6 and our MT5 re-implementation.

C Hyperparameters for Fine-Tuning
In

D Results on XTREME Cross-Lingual Understanding
We present the detailed results of the MT6 and our re-implemented MT5 models on XTREME in Table 11-16.

E Results on Wikilingua Cross-Lingual Summarization
As shown in Table 17, we present the detailed results of the MT6 and our re-implemented MT5 models on Wikilingua cross-lingual summarization.