The cross-lingual language models are typically pretrained with masked language modeling on multilingual text or parallel sentences. In this paper, we introduce denoising word alignment as a new cross-lingual pre-training task. Specifically, the model first self-labels word alignments for parallel sentences. Then we randomly mask tokens in a bitext pair. Given a masked token, the model uses a pointer network to predict the aligned token in the other language. We alternately perform the above two steps in an expectation-maximization manner. Experimental results show that our method improves cross-lingual transferability on various datasets, especially on the token-level tasks, such as question answering, and structured prediction. Moreover, the model can serve as a pretrained word aligner, which achieves reasonably low error rates on the alignment benchmarks. The code and pretrained parameters are available at github.com/CZWin32768/XLM-Align . Despite the current advances in NLP, most applications and resources are still English-centric, making non-English users hard to access. Therefore, it is essential to build cross-lingual transferable models that can learn from the training data in high-resource languages and generalize on low-resource languages. Recently, pretrained cross-lingual language models have shown their effectiveness for cross-lingual transfer. By pre-training on monolingual text and parallel sentences, the models provide significant improvements on a wide range of cross-lingual end tasks (Conneau and Lample, 2019; Conneau et al., 2020; Liu et al., 2020; Chi et al., 2021b). monolingual and parallel corpora. By simply learning masked language modeling (MLM; Devlin et al. 2019) on monolingual text of multiple languages, the models surprisingly achieve competitive results on cross-lingual tasks (Wu and Dredze, 2019; K et al., 2020). Besides, several pretext tasks are proposed to utilize parallel corpora to learn better sentence-level cross-lingual representations (Con-neau and Lample, 2019; Chi et al., 2021b; Hu et al., 2020a). For example, the translation language modeling (TLM; Conneau and Lample 2019) task performs MLM on the concatenated parallel sentences, which implicitly enhances cross-lingual transferability. However, most pretext tasks either learn alignment at the sentence level or implicitly encourage cross-lingual alignment, leaving explicit fine-grained alignment task not fully explored. In this paper, we introduce a new cross-lingual pre-training task, named as denoising word alignment . Rather than relying on external word aligners trained on parallel corpora (Cao et al., 2020; Zhao et al., 2020; Wu and Dredze, 2020), we utilize self-labeled alignments in our task. During pretraining, we alternately self-label word alignments and conduct the denoising word alignment task in an expectation-maximization manner. Specifically, the model first self-labels word alignments for a translation pair. Then we randomly mask tokens in the bitext sentence, which is used as the perturbed input for denosing word alignment. For each masked token, the model learns a pointer network to predict the self-labeled alignments in the other language. We repeat the above two steps to iteratively boost the bitext alignment knowledge for cross-lingual pre-training. We conduct extensive experiments on a wide range of cross-lingual understanding tasks. Experimental results show that our model outperforms the baseline models on various datasets, particularly on the token-level tasks such as question answering and structured prediction. Moreover, our model can also serve as a multilingual word aligner, which achieves reasonable low error rates on the bitext alignment benchmarks. Our contributions are summarized as follows:  We present a cross-lingual pre-training paradigm that alternately self-labels and predicts word alignments. We introduce a pre-training task, denoising word alignment, which predicts word alignments from perturbed translation pairs. We propose a word alignment algorithm that formulates the word alignment problem as optimal transport. We demonstrate that our explicit alignment objective is effective for cross-lingual transfer. Cross-lingual LM pre-training Pretrained with masked language modeling (MLM; Devlin et al. 2019) on monolingual text, multilingual BERT (mBERT; Devlin et al. 2019) and XLM-R (Con-neau et al., 2020) produce promising results on cross-lingual transfer benchmarks (Hu et al., 2020b). mT5 (Xue et al., 2020) learns a multilingual version of T5 (Raffel et al., 2020) with text-to-text tasks. In addition to monolingual text, several methods utilize parallel corpora to improve cross-lingual transferability. XLM (Conneau and Lample, 2019) presents the translation language modeling (TLM) task that performs MLM on concatenated translation pairs. ALM (Yang et al., 2020) introduces code-switched sequences into cross-lingual LM pre-training. Unicoder (Huang et al., 2019) employs three cross-lingual tasks to learn mappings among languages. From an information-theoretic perspective, InfoXLM (Chi et al., 2021b) proposes the cross-lingual contrastive learning task to align sentence-level representations. Additionally, AM-BER (Hu et al., 2020a) introduces an alignment objective that minimizes the distance between the forward and backward attention matrices. More recently, Ernie-M (Ouyang et al., 2020) presents the back-translation masked language modeling task that generates pseudo parallel sentence pairs for learning TLM, which provides better utilization of monolingual corpus. VECO (Luo et al., 2020) pretrains a unified cross-lingual language model for both NLU and NLG. mT6 (Chi et al., 2021a) improves the multilingual text-to-text transformer with translation pairs. Notably, Word-aligned BERT models (Cao et al., 2020; Zhao et al., 2020) finetune mBERT by an explicit alignment objective that minimizes the distance between aligned tokens. Wu and Dredze (2020) exploit contrastive learning to improve the explicit alignment objectives. However, Wu and Dredze (2020) show that these explicit alignment objectives do not improve cross-lingual representations under a more extensive evaluation. Moreover, these models are restricted to stay close to their original pretrained values, which is not applicable for large-scale pre-training. On the contrary, we demonstrate that employing our explicit alignment objective in large-scale pre-training can provide consistent improvements over baseline models. Word alignment The IBM models (Brown et al., 1993) are statistical models for modeling the translation process that can extract word alignments between sentence pairs. A large number of word alignment models are based on the IBM models (Och and Ney, 2003; Mermer and Saraclar, 2011; Dyer et al., 2013; Ostling and Tiedemann, 2016). Recent studies have shown that word alignments can be extracted from neural machine translation models (Ghader and Monz, 2017; Koehn and Knowles, 2017; Li et al., 2019) or from pretrained cross-lingual LMs (Jalili Sabet et al., 2020; Nagata et al., 2020). Figure 1 illustrates an overview of our method for pre-training our cross-lingual LM, which is called XLM-ALIGN . XLM-ALIGN is pretrained in an expectation-maximization manner with two alternating steps, which are word alignment self-labeling and denoising word alignment. We first formulate word alignment as an optimal transport problem, and self-label word alignments of the input translation pair on-the-fly. Then, we update the model parameters with the denoising word alignment task, where the model uses a pointer network (Vinyals et al., 2015) to predict the aligned tokens from the perturbed translation pair. The goal of word alignment self-labeling is to estimate the word alignments of the input translation pair on-the-fly, given the current XLM-ALIGN model. Given a source sentence S = s 1 . . . s i . . . s n and a target sentence T = t 1 . . . t j . . . t m , we model the word alignment between S and T as a doubly stochastic matrix A  R n  m + such that the rows and the columns all sum to 1 , where A ij stands for the probability of the alignment between s i and t j . The rows and the columns of A represent probability distributions of the forward alignment and the backward alignment, respectively. To measure the similarity between two tokens from S and T , we define a metric function f sim by using cross-lingual representations produced by XLM-ALIGN : f sim ( s i , t j ) =  log max( (cid:15), h (cid:62) i h j ) (1) where (cid:15) is a constant to avoid negative values in the log function, and h i is the hidden vector of the i -th token by encoding the concatenated sequence of S and T with XLM-ALIGN . Empirically, the metric function produces a high similarity score if the two input tokens are semantically similar. We can find that Eq. (2) is identical to the regularized optimal transport problem (Peyre et al., 2019), if we add an entropic regularization to A : max A n (cid:88) i =1 m (cid:88) j =1 A ij f sim ( s i , t j )  A ij log A ij (3) Eq. (3) has a unique solution A  such that A  = diag( u ) K diag( v ) (4) K ij = e f sim ( s i ,t j ) / (5) where u  R n + , v  R m + , K  R n  m + . According to Sinkhorn's algorithm (Peyre et al., 2019), the variables u and v can be calculated by the following iterations: u t +1 = 1 n K v t , v t +1 = 1 m K (cid:62) u t +1 (6) where v t can be initialized by v t =0 = 1 m . Similarly, the backward word alignments  A can be computed by applying argmax over columns. To obtain high-precision alignment labels, we adopt an iterative alignment filtering operation. We initialize the alignment labels A as  . In each iteration, we follow the procedure of Itermax (Jalili Sabet et al., 2020) that first computes A and A by Eq. (7). Then, the alignment labels are updated by: A  A  ( A  A ) (8) Finally, A  is updated by: A  ij    0 , ( i, j )  A A  ij ,  k ( i, k )  A  ( k, j )  A A  ij , others (9) where  is a discount factor. After several iterations, we obtain the final self-labeled word alignments A . After self-labeling word alignments, we update the model parameters with the denoising word alignment (DWA) task. The goal of DWA is to predict the word alignments from the perturbed version of the input translation pair. Consider the perturbed version of the input translation pair ( S  , T  ) constructed by randomly replacing the tokens with masks. We first encode the translation pair into hidden vectors h  with the XLM-ALIGN encoder: h  1 . . . h  n + m = encoder ([ S  , T  ]) (10) where [ S  , T  ] is the concatenated sequence of S  and T  with the length of n + m . Then, we build a pointer network upon the XLM-ALIGN encoder that predicts the word alignments. Specifically, for the i -th source token, we use h  i as the query vector and h  n +1 , . . . , h  n + m as the key vectors. Given the query and key vectors, the forward alignment probability a i is computed by the scaled dot-product attention (Vaswani et al., 2017): a i = softmax( q (cid:62) i K  d h ) (11) q i = linear( h  i ) (12) K = linear([ h  n +1 . . . h  n + m ]) (13) where d h is the dimension of the hidden vectors. Similarly, the backward alignment probability can be computed by above equations if we use target tokens as the query vectors and h  1 . . . h  n as key vectors. Notice that we only consider the self-labeled and masked positions as queries. Formally, we use the following query positions in the pointer network: P = { i | ( i,  )  A  (  , i )  A}  M (14) where M is the set of masked positions. The training objective is to minimize the cross-entropy between the alignment probabilities and the self-labeled word alignments: LDWA = (cid:88) i P CE ( a i , A ( i )) (15) where CE (  ,  ) stands for the cross-entropy loss, and A ( i ) is the self-labeled aligned position of the i -th token. We illustrate the pre-training procedure of XLM-ALIGN in Algorithm 1. In addition to DWA, we also include MLM and TLM for pre-training XLM-ALIGN , which implicitly encourage the cross-lingual alignment. The overall loss function is defined as: LMLM ( X ) + LTLM ( S , T ) + LDWA ( S , T , A ) In each iteration, we first sample monolingual text X , and parallel text ( S , T ) . Then, we self-label word alignments and update the model parameters by learning pretext tasks. Notice that the model parameters are initialized by a cold-start pre-training to avoid producing low-quality alignment labels. The cold-start pre-training can be accomplished by using a pretrained LM as the model initialization. Following previous cross-lingual pretrained models (Conneau and Lample, 2019; Conneau et al., 2020; Chi et al., 2021b), we use raw sentences from the Wikipedia dump and CCNet (Wenzek et al., 2019) for MLM, including 94 languages. For TLM and DWA, we use parallel corpora from MultiUN (Ziemski et al., 2016), IIT Bombay (Kunchukuttan et al., 2018), OPUS (Tiede-mann, 2012), and WikiMatrix (Schwenk et al., 2019), including 14 English-centric language pairs. We pretrain a Transformer with 12 layers and the hidden size of 768 , where the parameters are initialized with XLM-R (Conneau et al., 2020). The model is optimized with the Adam optimizer (Kingma and Ba, 2015) for 150 K steps with batch size of 2 , 048 . Notice that TLM and DWA share the same forward procedure for encoding the perturbed sentence pair. The pre-training of XLM-ALIGN takes about six days with two Nvidia DGX-2 stations. More details of the training data and the hyperparameters are in supplementary document. XTREME is a multilingual benchmark for evaluating cross-lingual generalization. We evaluate our model on 7 cross-lingual downstream tasks included by XTREME, which can be grouped into 3 categories: (1) Structured prediction: part-of-speech tagging on the Universal Dependencies v2.5 (Zeman et al., 2019), and named entity recognition on the WikiAnn (Pan et al., 2017; Rahimi et al., 2019) dataset; (2) Question answering: cross-lingual question answering on MLQA (Lewis et al., 2020) and XQuAD (Artetxe et al., 2020), and gold passage of typologically diverse question answering (TyDiQA-GoldP; Clark et al. 2020); (3) Sentence classification: cross-lingual natural language inference (XNLI; Conneau et al. 2018), and cross-lingual paraphrase adversaries from word scrambling (PAWS-X; Yang et al. 2019). Baselines We use the following pretrained cross-lingual LMs as baselines. (1) Multilingual BERT ( MBERT ; Devlin et al. 2019) is pretrained with masked language modeling (MLM) and next sentence prediction on Wikipedia of 104 languages; (2) XLM (Conneau and Lample, 2019) is jointly pretrained with MLM on 100 languages and translation language modeling (TLM) on 14 language pairs; (3) M T5 (Xue et al., 2020) is the multilingual version of T5 pretrained with text-to-text tasks; (4) XLM-R (Conneau et al., 2020) is pretrained with MLM on large-scale CC-100 dataset with long training steps. Fine-tuning Following Hu et al. (2020b), we adopt the zero-shot transfer setting for evaluation, where the models are only fine-tuned on English training data but evaluated on all target languages. Besides, we only use one model for evaluation on all target languages, rather than selecting different models for each language. The detailed fine-tuning hyperparameters can be found in supplementary document. Results In Table 1, we present the evaluation results on XTREME structured prediction, question answering, and sentence classification tasks. It can be observed that our XLM-ALIGN obtains the best average score over all the baseline models, improving the previous score from 66.4 to 68.9. It demonstrates that our model learns more transferable representations for the cross-lingual tasks, which is beneficial for building more accessible multilingual NLP applications. It is worth mentioning that our method brings noticeable improvements on the question answering and the structured prediction tasks. Compared with XLM-R base , XLM-ALIGN provides 6 . 7% and 1 . 9% F1 improvements on TyDiQA and NER. The improvements show that the Alignment Method Pretrained Alignment Error Rate  Avg Model en-de en-fr en-hi en-ro fast align (Dyer et al., 2013) -32.14 19.46 59.90 -SimAlign Argmax (Jalili Sabet et al., 2020) XLM-R 19. pretrained XLM-ALIGN benefits from the explicit word alignment objective, particularly on the structured prediction and question answering tasks that require token-level cross-lingual transfer. In terms of sentence classification tasks, XLM-ALIGN also consistently outperforms XLM-R base . Word alignment is the task of finding corresponding word pairs in a parallel sentence. We conduct evaluations with golden alignments of four language pairs from EuroParl 1 , WPT2003 2 , and WPT2005 3 , containing 1,244 annotated sentence pairs in total. We use alignment error rate (AER; Och and Ney 1 www-i6.informatik.rwth-aachen.de/ goldAlignment/ 2 web.eecs.umich.edu/mihalcea/wpt/ 3 web.eecs.umich.edu/mihalcea/wpt05/ 2003) as the evaluation metrics. Results We first explore whether our word alignment self-labeling method is effective for generating high-quality alignment labels. Thus, we compare our method with (1) fast align (Dyer et al., 2013), a widely-used implementation of IBM Model 2 (Och and Ney, 2003); (2) SimAlign (Jalili Sabet et al., 2020), state-of-the-art unsupervised word alignment method. For a fair comparison, we use the same pretrained LM and hidden layer as in SimAlign to produce sentence representations. In specific, we take the hidden vectors from the 8 -th layer of XLM-R base or XLM-ALIGN , and obtain the alignments following the procedure as described in Section 3.1. Since the produced alignments are subword-level, we convert the alignments into word-level by the following rule that if two subwords are aligned, the words they belong to are also aligned. As shown in Table 2, we report the AER scores on the four language pairs. It can be observed that our optimal-transport method outperforms fast align and SimAlign , demonstrating that our method can produce high-quality alignment labels, which is helpful for the DWA task. Moreover, our method consistently outperforms SimAlign when using hidden vectors from both XLM-R base and XLM-ALIGN . Then, we compare our XLM-ALIGN with XLM-R base on the word alignment task. Empirically, a lower AER indicates that the model learns better cross-lingual representations. From Table 2, XLM-ALIGN obtains the best AER results over all the four language pairs, reducing the averaged AER from 22 . 64 to 21 . 05 . Besides, un-Models XNLI POS NER MLQA Avg XLM-R* 74.6 75.7 61.6 65.7 69.4 XLM-ALIGN 75.2 75.6 62.6 66.7 70.0  DWA 75.1 75.2 62.0 65.8 69.5  TLM 74.4 76.0 60.4 66.0 69.2 Table 3: Ablation studies on the components of XLM-ALIGN . der both SimAlign and our optimal-transport method, XLM-ALIGN provides consistent reduction of AER, demonstrating the effectiveness of our method for learning fine-grained cross-lingual representations. We also compare XLM-ALIGN with XLM-R base using the hidden vectors from the 3 -th layer to the 12 -th layer. We illustrate the averaged AER scores in Figure 2. Notice that the results on the first two layers are not presented in the figure because of the high AER. It can be observed that XLM-ALIGN consistently improves the results over XLM-R base across these layers. Moreover, it shows a parabolic trend across the layers of XLM-R base , which is consistent with the results in (Jalili Sabet et al., 2020). In contrast to XLM-R base , XLM-ALIGN alleviates this trend and greatly reduces AER in the last few layers. We believe this property of XLM-ALIGN brings better cross-lingual transferability on the end tasks. In this section, we conduct comprehensive ablation studies for a better understanding of our XLM-ALIGN . To reduce the computational cost, we reduce the batch size to 256 , and pretrain models with 50 K steps in the following experiments. We perform ablation studies to understand the components of XLM-ALIGN , by removing the denoising word alignment loss (  DWA), the TLM loss (  TLM), or removing both (XLM-R*), which is identical to continue-training XLM-R base with MLM. We evaluate the models on XNLI, POS, NER, and MLQA, and present the results in Table 3. Comparing  TLM with  DWA, we find that DWA is more effective for POS and MLQA, while TLM performs better on XNLI and NER. Comparing  TLM with XLM-R*, it shows that directly learning DWA slightly harms the perfor-Layer XNLI POS NER MLQA Avg Layer-8 75.1 75.3 61.9 66.7 69.8 Layer-10 75.2 75.6 62.6 66.7 70.0 Layer-12 75.2 75.8 62.3 67.0 70.1 Table 4: Results of XLM-ALIGN with different layers used for word alignment self-labeling during pretraining. mance. However, jointly learning DWA with TLM provides remarkable improvements over  DWA, especially on the question answering and the structure prediction tasks that requires token-level cross-lingual transfer. This indicates that TLM potentially improves the quality of self-labeled word alignments, making DWA more effective for cross-lingual transfer. It has been shown that the word alignment performance has a parabolic trend across the layers of mBERT and XLM-R (Jalili Sabet et al., 2020). It indicates that the middle layers produce higher-quality word alignments than the bottom and the top layers. To explore which layer produces better alignment labels for pre-training, we pretrain three variants of XLM-ALIGN , where we use the hidden vectors from three different layers for word alignment self-labeling. We use the 8 -th, 10 -th, and 12 -th layers for word alignment self-labeling during the pre-training. We present the evaluation results in Table 4. Surprisingly, although Layer-8 produces higher-quality alignment labels at the beginning of the pre-training, using the alignment labels from the 12 -th layer learns a more transferable XLM-ALIGN model for cross-lingual end tasks. Beyond the self-labeling layer, we also investigate which layer is better for learning the denoising word alignment task. Recent studies have shown Filtering XNLI POS NER MLQA Avg Enable 75.2 75.6 62.6 66.7 70.0 Disable 74.2 75.3 61.6 65.3 69.1 Table 6: Effects of alignment filtering in word alignment self-labeling. that it is beneficial to learn sentence-level cross-lingual alignment at a middle layer (Chi et al., 2021b). Therefore, we pretrain XLM-ALIGN models by using three different layers for DWA, that is, using the hidden vectors of middle layers as the input of the pointer network. We compare the evaluation results of the three models in Table 5. It can be found that learning DWA at Layer8 improves XNLI while learning DWA at higher layers produces better performance on the other three tasks. It suggests that, compared with sentence-level pretext tasks that prefers middle layers, the DWA task should be applied at top layers. Although our self-labeling method produces high-quality alignment labels, the alignment filtering operation can potentially make some of the tokens unaligned, which reduces the example efficiency. Thus, we explore whether the alignment filtering is beneficial for pre-training XLM-ALIGN . To this end, we pretrain an XLM-ALIGN model without alignment filtering. In specific, we use the union set of the forward and backward alignments as the self-labeled alignments so that all tokens are aligned at least once. The forward and backward alignments are obtained by applying the argmax function over rows and columns of A  , respectively. Empirically, the alignment filtering operation generates high-precision yet fewer labels, while removing the filtering promises more labels but introduces low-confident labels. In Table 6, we compare the results of the models with or without alignment filtering. It can be observed that the alignment filtering operation improves the performance on the end tasks. This demonstrates that it is necessary to use high-precision labels for learning the denoising word alignment task. On the contrary, using perturbed alignment labels in pre-training harms the performance on the end tasks. as the query vectors in the pointer network. To explore the impact of the DWA query positions, we compare three different query positions in Table 7: (1) masked : only using the masked tokens as queries; (2) unmasked : randomly using 15 % of the unmasked tokens as queries; (3) all-aligned : for each self-labeled aligned pair, randomly using one of the two tokens as a query. Also, we include the no-query baseline that does not use any queries, which is identical to removing DWA. It can be observed that using all the three query positions improves the performance over the no-query baseline. Moreover, using the masked positions as queries achieves better results than the other two positions, demonstrating the effectiveness of the masked query positions. In this paper, we introduce denoising word alignment as a new cross-lingual pre-training task. By alternately self-labeling and predicting word alignments, our XLM-ALIGN model learns transferable cross-lingual representations. Experimental results show that our method improves the cross-lingual transferability on a wide range of tasks, particularly on the token-level tasks such as question answering and structured prediction. Despite the effectiveness for learning cross-lingual transferable representations, our method also has the limitation that requires a cold-start pre-training to prevent the model from producing low-quality alignment labels. In our experiments, we also try to pretrain XLM-ALIGN from scratch, i.e., without cold-start pre-training. However, the DWA task does not work very well due to the low-quality of self-labeled alignments. Thus, we recommend continue-training XLM-ALIGN on the basis of other pretrained cross-lingual language models. For future work, we would like to research on removing this restriction so that the model can learn word alignments from scratch. Despite the current advances in NLP, most NLP research works and applications are English-centric, making none-English users hard to access to NLP-related services. Our method aims to pretrain cross-lingual language models that transfer supervision signals from high-resource languages to low-resource languages, which makes the NLP services and applications more accessible for low-resource-language speakers. Furthermore, our method can build multilingual models that serve on different languages at the same time, reducing the computational resources for building multilingual models separately for each language. Heyan Huang is the corresponding author. The work is supported by National Key R&D Plan (No. 2018YFB1005100), National Natural Science Foundation of China (No. 61751201, 61602197 and 61772076), Natural Science Fund of Beijing (No. Z181100008918002), and the funds of Beijing Advanced Innovation Center for Language Resources (No. TYZ19005).
abstain objective abstain abstain abstain method result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective method method abstain method abstain method method result result objective method objective objective other other other other other other other other other other other other other other other objective other other other method abstain method method abstain abstain method abstain method abstain method method abstain abstain abstain method method method abstain abstain method method abstain abstain method method method other method method method abstain method method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain other abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method result method method abstain method method abstain abstain method other other