In this work, we present an information-theoretic framework that formulates cross-lingual language model pre-training as maximizing mutual information between multilingual-multi-granularity texts. The unified view helps us to better understand the existing methods for learning cross-lingual representations. More importantly, inspired by the framework, we propose a new pretraining task based on contrastive learning. Specifically, we regard a bilingual sentence pair as two views of the same meaning and encourage their encoded representations to be more similar than the negative examples. By leveraging both monolingual and parallel corpora, we jointly train the pretext tasks to improve the cross-lingual transferability of pre-trained models. Experimental results on several benchmarks show that our approach achieves considerably better performance. The code and pre-trained models are available at https://aka.ms/infoxlm . Learning cross-lingual language representations plays an important role in overcoming the language barrier of NLP models. The recent success of cross-lingual language model pre-training (Devlin et al., 2019; Conneau and Lample, 2019; Conneau et al., 2020a; Chi et al., 2020; Liu et al., 2020) significantly improves the cross-lingual transferability in various downstream tasks, such as cross-lingual classification, and question answering. State-of-the-art cross-lingual pre-trained models are typically built upon multilingual masked language modeling (MMLM; Devlin et al. 2019; Conneau et al. 2020a), and translation language modeling (TLM; Conneau and Lample 2019). The goal of both pretext tasks is to predict masked tokens given input context. The difference is that  Contribution during internship at Microsoft Research. MMLM uses monolingual text as input, while TLM feeds bilingual parallel sentences into the model. Even without explicit encouragement of learning universal representations across languages, the derived models have shown promising abilities of cross-lingual transfer. In this work, we formulate cross-lingual pretraining from a unified information-theoretic perspective. Following the mutual information maximization principle (Hjelm et al., 2019; Kong et al., 2020), we show that the existing pretext tasks can be viewed as maximizing the lower bounds of mutual information between various multilingual-multi-granularity views. Specifically, MMLM maximizes mutual information between the masked tokens and the context in the same language while the anchor points across languages encourages the correlation between cross-lingual contexts. Moreover, we present that TLM can maximize mutual information between the masked tokens and the parallel context, which implicitly aligns encoded representations of different languages. The unified information-theoretic framework also inspires us to propose a new cross-lingual pre-training task, named as cross-lingual contrast (XLCO ). The model learns to distinguish the translation of an input sentence from a set of negative examples. In comparison to TLM that maximizes token-sequence mutual information, XLCO maximizes sequence-level mutual information between translation pairs which are regarded as cross-lingual views of the same meaning. We employ the momentum contrast (He et al., 2020) to realize XLCO . We also propose the mixup contrast and conduct the contrast on the universal layer to further facilitate the cross-lingual transferability. Under the presented framework, we develop a cross-lingual pre-trained model (INFOXLM) to leverage both monolingual and parallel corpora. We jointly train INFOXLM with MMLM, TLM and XLCO . We conduct extensive experiments on several cross-lingual understanding tasks, including cross-lingual natural language inference (Con-neau et al., 2018), cross-lingual question answering (Lewis et al., 2020), and cross-lingual sentence retrieval (Artetxe and Schwenk, 2019). Experimental results show that INFOXLM outperforms strong baselines on all the benchmarks. Moreover, the analysis indicates that INFOXLM achieves better cross-lingual transferability. Multilingual BERT (mBERT; Devlin et al. 2019) is pre-trained with the multilingual masked language modeling (MMLM) task on the monolingual text. mBERT produces cross-lingual representations and performs cross-lingual tasks surprisingly well (Wu and Dredze, 2019). XLM (Conneau and Lample, 2019) extends mBERT with the translation language modeling (TLM) task so that the model can learn cross-lingual representations from parallel corpora. Unicoder (Huang et al., 2019) tries several pre-training tasks to utilize parallel corpora. ALM (Yang et al., 2020) extends TLM to code-switched sequences obtained from translation pairs. XLM-R (Conneau et al., 2020a) scales up MMLM pre-training with larger corpus and longer training. LaBSE (Feng et al., 2020) learns cross-lingual sentence embeddings by an additive translation ranking loss. In addition to learning cross-lingual encoders, several pre-trained models focus on generation. MASS (Song et al., 2019) and mBART (Liu et al., 2020) pretrain sequence-to-sequence models to improve machine translation. XNLG (Chi et al., 2020) focuses on the cross-lingual transfer of language generation, such as cross-lingual question generation, and abstractive summarization. Various methods have successfully learned visual or language representations by maximizing mutual information between different views of input. It is difficult to directly maximize mutual information. In practice, the methods resort to a tractable lower bound as the estimator, such as InfoNCE (Oord et al., 2018), and the variational form of the KL divergence (Nguyen et al., 2010). The estimators are also known as contrastive learning (Arora et al., 2019) that measures the representation similarities between the sampled positive and negative pairs. In addition to the estimators, various view pairs are employed in these methods. The view pair can be the local and global features of an image (Hjelm et al., 2019; Bachman et al., 2019), the random data augmentations of the same image (Tian et al., 2019; He et al., 2020; Chen et al., 2020), or different parts of a sequence (Oord et al., 2018; Henaff, 2020; Kong et al., 2020). Kong et al. (2020) show that learning word embeddings or contextual embeddings can also be unified under the framework of mutual information maximization. In representation learning, the learned representations are expected to preserve the information of the original input data. However, it is intractable to directly model the mutual information between the input data and the representations. Alternatively, we can maximize the mutual information between the representations from different views of the input data, e.g., different parts of a sentence, a translation pair of the same meaning. In this section, we start from a unified information-theoretic perspective, and formulate cross-lingual pre-training with the mutual information maximization principle. Then, under the information-theoretic framework, we propose a new cross-lingual pre-training task, named as cross-lingual contrast (XLCO ). Finally, we present the pre-training procedure of our INFOXLM. The goal of multilingual masked language modeling (MMLM; Devlin et al. 2019) is to recover the masked tokens from a randomly masked sequence. For each input sequence of MMLM, we sample a text from the monolingual corpus for pretraining. Let ( c 1 , x 1 ) denote a monolingual text sequence, where x 1 is the masked token, and c 1 is the corresponding context. Intuitively, we need to maximize their dependency (i.e., I ( c 1 ; x 1 ) ), so that the context representations are predictive for masked tokens (Kong et al., 2020). For the example pair ( c 1 , x 1 ) , we construct a set N that contains x 1 and |N |  1 negative samples drawn from a proposal distribution q . According to the InfoNCE (Oord et al., 2018) lower bound, we have: I ( c 1 ; x 1 ) (cid:62) E q ( N ) (cid:20) log f  ( c 1 , x 1 ) (cid:80) x (cid:48) N f  ( c 1 , x (cid:48) ) (cid:21) + log |N | (1) where f  is a function that scores whether the input c 1 and x 1 is a positive pair. Given context c 1 , MMLM learns to minimize the cross-entropy loss of the masked token x 1 : LMMLM =  log exp( g  T ( c 1 ) (cid:62) g  E ( x 1 )) (cid:80) x (cid:48) V exp( g  T ( c 1 ) (cid:62) g  E ( x (cid:48) )) (2) where V is the vocabulary, g  E is a look-up function that returns the token embeddings, g  T is a Transformer that returns the final hidden vectors in position of x 1 . According to Equation (1) and Equation (2), if N = V and f  ( c 1 , x 1 ) = exp( g  T ( c 1 ) (cid:62) g  E ( x 1 )) , we can find that MMLM maximizes a lower bound of I ( c 1 ; x 1 ) . Next, we explain why MMLM can implicitly learn cross-lingual representations. Let ( c 2 , x 2 ) denote a MMLM instance that is in different language as ( c 1 , x 1 ) . Because the vocabulary, the position embedding, and special tokens are shared across languages, it is common to find anchor points (Pires et al., 2019; Dufter and Schutze, 2020) where x 1 = x 2 (such as subword, punctuation, and digit) or I ( x 1 , x 2 ) is positive (i.e., the representations are associated or isomorphic). With the bridge effect of { x 1 , x 2 } , MMLM obtains a v-structure dependency  c 1  { x 1 , x 2 }  c 2 , which leads to a negative co-information (i.e., interaction information) I ( c 1 ; c 2 ; { x 1 , x 2 } ) (Tsujishita, 1995). Specifically, the negative value of I ( c 1 ; c 2 ; { x 1 , x 2 } ) indicates that the variable { x 1 , x 2 } enhances the correlation between c 1 and c 2 (Fano, 1963). In summary, although MMLM learns to maximize I ( c 1 , x 1 ) and I ( c 2 , x 2 ) in each language, we argue that the task encourages the cross-lingual correlation of learned representations. Notice that for the setting without word-piece overlap (Artetxe et al., 2020; Conneau et al., 2020b; K et al., 2020), we hypothesize that the information bottleneck principle (Tishby and Zaslavsky, 2015) tends to transform the cross-lingual structural similarity into isomorphic representations, which has similar bridge effects as the anchor points. Then we can explain how the cross-lingual ability is spread out as above. We leave more discussions about the setting without word-piece overlap for future work. Similar to MMLM, the goal of translation language modeling (TLM; Conneau and Lample 2019) is also to predict masked tokens, but the prediction is conditioned on the concatenation of a translation pair. We try to explain how TLM pre-training enhances cross-lingual transfer from an information-theoretic perspective. Let c 1 and c 2 denote a translation pair of sentences, and x 1 a masked token taken in c 1 . So c 1 and x 1 are in the same language, while c 1 and c 2 are in different ones. Following the derivations of MMLM in Section 3.1, the objective of TLM is maximizing the lower bound of mutual information I ( c 1 , c 2 ; x 1 ) . By re-writing the above mutual information, we have: I ( c 1 , c 2 ; x 1 ) = I ( c 1 ; x 1 ) + I ( c 2 ; x 1 | c 1 ) (3) The first term I ( c 1 ; x 1 ) corresponds to MMLM, which learns to use monolingual context. In contrast, the second term I ( c 2 ; x 1 | c 1 ) indicates cross-lingual mutual information between c 2 and x 1 that is not included by c 1 . In other words, I ( c 2 ; x 1 | c 1 ) encourages the model to predict masked tokens by using the context in a different language. In conclusion, TLM learns to utilize the context in both languages, which implicitly improves the cross-lingual transferability of pre-trained models. Inspired by the unified information-theoretic framework, we propose a new cross-lingual pre-training task, named as cross-lingual contrast (XLCO ). The goal of XLCO is to maximize mutual information between the representations of parallel sentences c 1 and c 2 , i.e., I ( c 1 , c 2 ) . Unlike maximizing token-sequence mutual information in MMLM and TLM, XLCO targets at cross-lingual sequence-level mutual information. where N is a set that contains the positive pair c 2 and |N | 1 negative samples. In order to maximize the lower bound of I ( c 1 ; c 2 ) , we need to design the function f  that measures the similarity between the input sentence and the proposal distribution q ( N ) . where g  is the Transformer encoder that we are pre-training. Following (Devlin et al., 2019), a special token [CLS] is added to the input, whose hidden vector is used as the sequence representation. Additionally, we use a linear projection head after the encoder in g  . Momentum Contrast Another design choice is how to construct N . As shown in Equation (4), a large |N | improves the tightness of the lower bound, which has been proven to be critical for contrastive learning (Chen et al., 2020). In our work, we employ the momentum contrast (He et al., 2020) to construct the set N , where the previously encoded sentences are progressively reused as negative samples. Specifically, we construct two encoders with the same architecture which are the query encoder g  Q and the key encoder g  K . The loss function of XLCO is: LXLCO =  log exp( g  Q ( c 1 ) (cid:62) g  K ( c 2 )) (cid:80) c (cid:48) N exp( g  Q ( c 1 ) (cid:62) g  K ( c (cid:48) )) (6) During training, the query encoder g  Q encodes c 1 and is updated by backpropagation. The key encoder g  K encodes N and is learned with momentum update (He et al., 2020) towards the query encoder. The negative examples in N are organized as a queue, where a newly encoded example is added while the oldest one is popped from the queue. We initialize the query encoder and the key encoder with the same parameters, and pre-fill the queue with a set of encoded examples until it reaches the desired size |N | . Notice that the size of the queue remains constant during training. Mixup Contrast For each pair, we concatenate it with a randomly sampled translation pair from another parallel corpus. For example, consider the pairs (cid:104) c 1 , c 2 (cid:105) and (cid:104) d 1 , d 2 (cid:105) sampled from two different parallel corpora. The two pairs are concatenated in a random order, such as (cid:104) c 1 d 1 , c 2 d 2 (cid:105) , and (cid:104) c 1 d 2 , d 1 c 2 (cid:105) . The data augmentation of mixup encourages pre-trained models to learn sentence boundaries and to distinguish the order of multilingual texts. Contrast on Universal Layer As a pre-training task maximizing the lower bound of sequence-level mutual information, XLCO is usually jointly learned with token-sequence tasks, such as MMLM, and TLM. In order to make XLCO more compatible with the other pretext tasks, we propose to conduct contrastive learning on the most universal (or transferable) layer in terms of MMLM and TLM. In our implementations, we instead use the hidden vectors of [CLS] at layer 8 to perform contrastive learning for base-size (12 layers) models, and layer 12 for large-size (24 layers) models. Because previous analysis (Sabet et al., 2020; Dufter and Schutze, 2020; Conneau et al., 2020b) shows that the specific layers of MMLM learn more universal representations and work better on cross-lingual retrieval tasks than other layers. We choose the layers following the same principle. The intuition behind the method is that MMLM and TLM encourage the last layer to produce language-distinguishable token representations because of the masked token classification. But XLCO tends to learn similar representations across languages. So we do not directly use the hidden states of the last layer in XLCO . We pretrain a cross-lingual model INFOXLM by jointly maximizing the lower bounds of three types of mutual information, including monolingual token-sequence mutual information (MMLM), cross-lingual token-sequence mutual information (TLM), and cross-lingual sequence-level mutual information (XLCO ). Formally, the loss of cross-lingual pre-training in INFOXLM is defined as: L = LMMLM + LTLM + LXLCO (7) where we apply the same weight for the loss terms. Both TLM and XLCO use parallel data. The number of bilingual pairs increases with the square of the number of languages. In our work, we set English as the pivot language following (Conneau and Lample, 2019), i.e., we only use the parallel corpora that contain English. In order to balance the data size between high-resource and low-resource languages, we apply a multilingual sampling strategy (Conneau and Lample, 2019) for both monolingual and parallel data. An example in the language l is sampled with the probability p l  ( n l /n ) 0 . 7 , where n l is the number of instances in the language l , and n refers to the total number of data. Empirically, the sampling algorithm alleviates the bias towards high-resource languages (Conneau et al., 2020a). In this section, we first present the training config-uration of INFOXLM. Then we compare the fine-tuning results of INFOXLM with previous work on three cross-lingual understanding tasks. We also conduct ablation studies to understand the major components of INFOXLM. Corpus We use the same pre-training corpora as previous models (Conneau et al., 2020a; Conneau and Lample, 2019). Specifically, we reconstruct CC-100 (Conneau et al., 2020a) for MMLM, which remains 94 languages by filtering the language code larger than 0.1GB. Following (Con-neau and Lample, 2019), for the TLM and XLCO tasks, we employ 14 language pairs of parallel data that involves English. We collect translation pairs from MultiUN (Ziemski et al., 2016), IIT Bombay (Kunchukuttan et al., 2018), OPUS (Tiede-mann, 2012), and WikiMatrix (Schwenk et al., 2019). The size of parallel corpora is about 42GB. More details about the pre-training data are described in the appendix. Model Size We follow the model configurations of XLM-R (Conneau et al., 2020a). For the Transformer (Vaswani et al., 2017) architecture, we use 12 layers and 768 hidden states for INFOXLM (i.e., base size), and 24 layers and 1,024 hidden states for INFOXLMLARGE (i.e., large size). Hyperparameters We initialize the parameters of INFOXLM with XLM-R. We optimize the model with Adam (Kingma and Ba, 2015) using a batch size of 2048 for a total of 150K steps for INFOXLM, and 200K steps for INFOXLMLARGE . The same number of training examples are fed to three tasks. The learning rate is scheduled with a linear decay with 10K warmup steps, where the peak learning rate is set as 0 . 0002 for INFOXLM, and 0 . 0001 for INFOXLMLARGE . The momentum coefficient is set as 0 . 9999 and 0 . 999 for INFOXLM and INFOXLMLARGE , respectively. The length of the queue is set as 131 , 072 . The training procedure takes about 2 . 3 days  2 Nvidia DGX-2 stations for INFOXLM, and 5 days  16 Nvidia DGX-2 stations for INFOXLMLARGE . Details about the pre-training hyperparameters can be found in the appendix. We conduct experiments over three cross-lingual understanding tasks, i.e., cross-lingual natural language inference, cross-lingual sentence retrieval, and cross-lingual question answering. Cross-Lingual Natural Language Inference The Cross-Lingual Natural Language Inference corpus (XNLI; Conneau et al. 2018) is a widely used cross-lingual classification benchmark. The goal of NLI is to identify the relationship of an input sentence pair. We evaluate the models under the following two settings. (1) Cross-Lingual Transfer: fine-tuning the model with English training set and directly evaluating on multilingual test sets. (2) Translate-Train-All: fine-tuning the model with the English training data and the pseudo data that are translated from English to the other languages. Cross-Lingual Sentence Retrieval The goal of the cross-lingual sentence retrieval task is to extract parallel sentences from bilingual comparable corpora. We use the subset of 36 language pairs of the Tatoeba dataset (Artetxe and Schwenk, 2019) for the task. The dataset is collected from Tatoeba 1 , which is an open collection of multilingual parallel sentences in more than 300 languages. Following (Hu et al., 2020), we use the averaged hidden vectors in the seventh Transformer layer to compute cosine similarity for sentence retrieval. Cross-Lingual Question Answering We use the Multilingual Question Answering (MLQA; Lewis et al. 2020) dataset for the cross-lingual QA task. MLQA provides development and test data in seven languages in the format of SQuAD v1.1 (Rajpurkar et al., 2016). We follow the fine-tuning method introduced in (Devlin et al., 2019) that concatenates the question-passage pair as the input. We compare INFOXLM with the following pretrained Transformer models: (1) Multilingual BERT ( MBERT ; Devlin et al. 2019) is pre-trained with MMLM on Wikipedia in 102 languages; (2) XLM (Conneau and Lample, 2019) pretrains both MMLM and TLM tasks on Wikipedia in 100 languages; (3) XLM-R (Conneau et al., 2020a) scales up MMLM to the large CC-100 corpus in 100 languages with much more training steps; (4) UNICODER (Liang et al., 2020) continues training XLM-R with MMLM and TLM. (5) INFOXLM  XLCO continues training XLM-R with MMLM and TLM, using the same pre-training datasets with INFOXLM. Cross-Lingual Natural Language Inference Table 1 reports the classification accuracy on each test of XNLI under the above evaluation settings. The final scores on test set are averaged over five random seeds. INFOXLM outperforms all baseline models on the two evaluation settings of XNLI. In the cross-lingual transfer setting, INFOXLM achieves 76.5 averaged accuracy, outperforming XLM-R (reimpl) by 1.5. Similar improvements can be observed for large-size models. Moreover, the ablation results   XLCO  show that cross-lingual contrast is helpful for zero-shot transfer in most languages. We also find that INFOXLM improves the results in the translate-train-all setting. 2 and Table 3, we report the top-1 accuracy scores of cross-lingual sentence retrieval with the base-size models. The evaluation results demonstrate that INFOXLM produces better aligned cross-lingual sentence representations. On the 14 language pairs that are covered by parallel data, INFOXLM obtains 77.8 and 80.6 averaged top-1 accuracies in the directions of xx  en and en  xx, outperforming XLM-R by 20.2 and 21.1. Even on the 22 language pairs that are not covered by parallel data, INFOXLM outperforms XLM-R on 16 out of 22 language pairs, providing 8.1% improvement in averaged accuracy. In comparison, the ablation variant   XLCO  (i.e., MMLM + TLM) obtains better results than XLM-R in Table 2, while getting worse performance than XLM-R in Table 3. The results indicate that XLCO encourages the model to learn universal representations even on the language pairs without parallel supervision. Cross-Lingual Question Answering Table 4 compares INFOXLM with baseline models on MLQA, where we report the F1 and the exact match (EM) scores on each test set. Both INFOXLM and INFOXLMLARGE obtain the best results against the four baselines. In addition, the results of the ablation variant   XLCO  indicate that the proposed cross-lingual contrast is benefi-cial on MLQA. Models Direction ar bg zh de el fr hi ru es sw th tr ur vi Avg XLM-R xx  en 36.8 67.6 60.7 89.9 53.7 74.1 54.2 72.5 74.0 18.7 38.3 61.1 36.6 68.4 57.6 INFOXLM xx  en 59.0 78.6 86.3 93.9 62.1 79.4 87.1 83.8 88.2 39.5 84.9 83.3 73.0 89.6 77.8  XLCO xx  en 42.9 65.5 69.5 91.1 55.6 76.4 71.6 74.9 74.8 20.5 68.1 69.8 51.6 81.8 65.3 XLM-R en  xx 38.6 69.9 60.3 89.4 57.3 74.3 49.3 73.0 74.6 14.4 58.4 64.0 36.9 72.5 59.5 INFOXLM en  xx 68.6 78.6 86.4 95.1 72.6 84.0 88.3 85.7 87.2 40.8 91.2 84.7 73.3 92.0 80.6  XLCO en  xx 45.4 64.0 69.3 88.1 56.5 72.3 69.6 73.6 71.5 22.1 79.7 64.3 48.2 79.8 64.6 Table 2: Evaluation results on Tatoeba cross-lingual sentence retrieval. To understand INFOXLM and the cross-lingual contrast task more deeply, we conduct analysis from the perspectives of cross-lingual transfer and cross-lingual representations. Furthermore, we perform comprehensive ablation studies on the major components of INFOXLM, including the cross-lingual pre-training tasks, mixup contrast, the contrast layer, and the momentum contrast. To reduce the computation load, we use INFO XLM15 in our ablation studies, which is trained on 15 languages for 100K steps. Cross-Lingual Transfer Gap Cross-lingual transfer gap (Hu et al., 2020) is the difference between the performance on the English test set and the averaged performance on the test sets of all other languages. A lower cross-lingual transfer gap score indicates more end-task knowledge from the English training set is transferred to other languages. In Table 5, we compare the cross-lingual transfer gap scores of INFOXLM with baseline models on MLQA and XNLI. Note that we do not include the results of XLM because it is pre-trained on 15 languages or using #M=N. The results show that INFOXLM reduces the gap scores on both MLQA and XNLI, providing better cross-lingual transferability than the baselines. cross-lingual transfer, learning good cross-lingual representations is also the goal of cross-lingual pre-1 training. In order to analyze how the cross-lingual contrast task affects the alignment of the learned cross-lingual representations, we evaluate the representations of different middle layers on the Tatoeba test sets of the 14 languages that are covered by parallel data. Figure 1 presents the averaged top-1 accuracy of cross-lingual sentence retrieval in the direction of xx  en. INFOXLM outperforms XLM-R on all of the 12 layers, demonstrating that our proposed task improves the cross-lingual alignment of the learned representations. From the results of XLM-R, we observe that the model suffers from a performance drop in the last few layers. The reason is that MMLM encourages the representations of the last hidden layer to be similar to token embeddings, which is contradictory with the goal of learning cross-lingual representations. In Models en es de ar hi vi zh Avg MBERT * 77.7 / 65.2 64.3 / 46.6 57.9 / 44.3 45.7 / 29.8 43.8 / 29.7 57.1 / 38.6 57.5 / 37.3 57.7 / 41.6 XLM* 74.9 / 62.4 68.0 / 49.8 62.2 / 47.6 54.8 / 36.3 48.8 / 27.3 61.4 / 41.8 61.1 / 39.6 61.6 / 43.5 UNICODER 80.6 / -68.6 / -62.7 / -57.8 / -62.7 / -67.5 / -62.1 / -66.0 / -XLM-R 77.1 / 64.6 67.4 / 49.6 60.9 / 46.7 54.9 / 36.6 59.4 / 42.9 64.5 / 44.7 61.8 / 39.3 63.7 / 46.3 XLM-R (reimpl) 80.2 / 67.0 67.7 / 49.9 62.1 / 47.7 56.1 / 37.2 61.1 / 44.0 67.0 / 46.3 61.4 / 38.5 65.1 / 47.2 INFOXLM 81.6 / 68.3 69.8 / 51.6 64.3 / 49.4 60.6 / 40.9 65.2 / 47.1 70.2 / 49.0 64.8 / 41.3 68.1 / 49.6  XLCO 81.2 / 68.1 69.6 / 51.9 64.0 / 49.3 59.7 / 40.2 64.0 / 46.3 69.3 / 48.0 64.1 / 40.6 67.4 / 49.2 XLM-RLARGE 80.6 / 67.8 74.1 / 56.0 68.5 / 53.6 63.1 / 43.5 69.2 / 51.6 71.3 / 50.9 68.0 / 45.4 70.7 / 52.7 XLM-RLARGE (reimpl) 84.0 / 71.1 74.4 / 56.4 70.2 / 55.0 66.5 / 46.3 71.1 / 53.2 74.4 / 53.5 68.6 / 44.6 72.7 / 54.3 INFOXLMLARGE 84.5 / 71.6 75.1 / 57.3 71.2 / 56.2 67.6 / 47.6 72.5 / 54.2 75.2 / 54.1 69.2 / 45.4 73.6 / 55.2 Table 4: Evaluation results on MLQA cross-lingual question answering. contrast, INFOXLM still provides high retrieval accuracy at the last few layers, which indicates that INFOXLM provides better aligned representations than XLM-R. Moreover, we find that the performance is further improved when removing TLM, demonstrating that XLCO is more effective than TLM for aligning cross-lingual representations, although TLM helps to improve zero-shot cross-lingual transfer. Effect of Cross-Lingual Pre-training Tasks To better understand the effect of the cross-lingual pre-training tasks, we perform ablation studies on the pre-training tasks of INFOXLM, by removing XLCO , TLM, or both. We present the experimental results in Table 7. Comparing the results of  TLM and  XLCO with the results of  TLM  XLCO , we find that both XLCO and TLM effectively improve cross-lingual transferability of the pre-trained INFOXLM model. TLM is more effective for XNLI while XLCO is more effective for MLQA. Moreover, the performance can be further improved by jointly learning XLCO and TLM. on the universal layer improves cross-lingual pretraining. As shown in Table 6, we compare the evaluation results of four variants of INFOXLM, where XLCO is applied on the layer 8 (i.e., universal layer) or on the layer 12 (i.e., the last layer). We find that contrast on the layer 8 provides better results for INFOXLM. However, conducting XLCO on layer 12 performs better when the TLM task is excluded. The results show that maximizing context-sequence (TLM) and sequence-level (XLCO ) mutual information at the last layer tends to interfere with each other. Thus, we suggest applying XLCO on the universal layer for pre-training INFOXLM. pretrain a model that directly uses translation pairs for XLCO without mixup contrast (  TLM  Mixup). As shown in Table 7, we present the evaluation results on XNLI and MLQA. We observe that mixup contrast improves the performance of INFOXLM on both datasets. Effect of Momentum Contrast In order to show whether our pre-trained model benefits from momentum contrast, we pretrain a revised version of INFOXLM without momentum contrast. In other words, the parameters of the key encoder are always the same as the query encoder. As shown in Table 7, we report evaluation results (indicated by   TLM  Momentum) of removing momentum contrast on XNLI and MLQA. We observe a performance descent after removing the momentum contrast from INFOXLM, which indicates that momentum contrast improves the learned language representations of INFOXLM. In this paper, we present a cross-lingual pre-trained model INFOXLM that is trained with both monolingual and parallel corpora. The model is motivated by the unified view of cross-lingual pretraining from an information-theoretic perspective. Specifically, in addition to the masked language modeling and translation language modeling tasks, INFOXLM is jointly pre-trained with a newly introduced cross-lingual contrastive learning task. The cross-lingual contrast leverages bilingual pairs as the two views of the same meaning, and encourages their encoded representations to be more similar than the negative examples. Experimental results on several cross-lingual language understanding tasks show that INFOXLM can considerably improve the performance. Currently, most NLP research works and applications are English-centric, which makes non-English users hard to access to NLP-related services. Our work focuses on cross-lingual language model pretraining. With the pre-trained model, we are able to transfer end-task knowledge from high-resource languages to low-resource languages, which helps to build more accessible NLP applications. Additionally, incorporating parallel corpora into the pretraining procedure improves the training efficiency, which potentially reduces the computational cost for building multilingual NLP applications. We appreciate the helpful discussions with Bo Zheng, Shaohan Huang, Shuming Ma, and Yue Cao. H.H. is the corresponding author. X.M. and H.H. are partly supported by National Key R&D Plan (No. 2018YFB1005100), National Natural Science Foundation of China (No. 61751201, 61602197 and 61772076), Natural Science Fund of Beijing (No. Z181100008918002), and the funds of Beijing Advanced Innovation Center for Language Resources (No. TYZ19005).
method abstain objective method result result other abstain abstain abstain abstain abstain abstain abstain method result abstain method objective abstain abstain abstain objective objective method method abstain abstain other other other other other other other other other other other other other other other other other other other abstain method objective method abstain method abstain abstain method other other abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain method other abstain abstain method abstain method abstain abstain abstain abstain abstain method abstain method abstain abstain method method method abstain abstain method method abstain abstain abstain abstain result abstain method method method method abstain abstain method method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain method abstain abstain other abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method abstain other other other