Existing work in multilingual pretraining has demonstrated the potential of cross-lingual transferability by training a unified Transformer encoder for multiple languages. However, much of this work only relies on the shared vocabulary and bilingual contexts to encourage the correlation across languages, which is loose and implicit for aligning the contextual representations between languages. In this paper, we plug a cross-attention module into the Transformer encoder to explicitly build the interdependence between languages. It can effectively avoid the degeneration of predicting masked words only conditioned on the context in its own language. More importantly, when fine-tuning on downstream tasks, the cross-attention module can be plugged in or out on-demand, thus naturally benefiting a wider range of cross-lingual tasks, from language understanding to generation. As a result, the proposed cross-lingual model delivers new state-of-the-art results on various cross-lingual understanding tasks of the XTREME benchmark, covering text classification, sequence labeling, question answering, and sentence retrieval. For cross-lingual generation tasks, it also outperforms all existing cross-lingual models and state-of-the-art Transformer variants on WMT14 English-to-German and English-to-French translation datasets, with gains of up to 1  2 BLEU. 1 1 Introduction Cross-lingual pre-trained models like mBERT (De-vlin et al., 2019), XLM (Lample and Conneau, 2019) and XLM-R (Conneau et al., 2019) that target providing contextualized representations for the inputs across languages, have shown large poten-* Equal contribution. chrome-extension://mbniclmhobmnbdlbpiphghaielnnpgdp/screenshot.html?id=screenshot_0.0007918474100581108 1/1 Behind the great success, two major factors play the role of aligning the contextual representations between languages: 1) build the shared vocabulary across languages through subword tokenization, which supports the simple extension of masked language modeling (MLM) from English corpus to multilingual corpus; 2) capture the alignment in parallel data via concatenating two sentences as input, called translation language modeling (TLM). However, both of these two mechanisms rely on the self-attention module (query=key/value) of the Transformer encoder to implicitly enhance the interdependence between languages, which may lead to few attention patterns across languages. Taking Figure 1 as an example, even though inputting a pair of parallel sentences, both models only attend to the English context to build the representation of English tokens, while ignoring the se-b) Translation Language Modeling (TLM) mantically related Chinese tokens. That is, the self-attention module captures little communication across languages, which is crucial for learning universal cross-lingual representations. Based on the above observation, we propose to plug a cross-attention module (query!=key/value) into the Transformer encoder and design a cross-attention MLM task to explicitly capture the interdependence between languages. As illustrated in Figure 2 (c), the cross-attention module takes the representation of x as query and y as key/value (purple lines) to build the representations of x in the next layer, thus explicitly aligning the representations across languages (purple attention ma-trices). It can effectively avoid the degeneration of predicting masked words only conditioned on the context in its own language. Moreover, what distinguishes our work from pre-training an encoder-decoder model (Liu et al., 2020b) is that we also keep the good nature (i.e., bidirectional contextual modeling) of the original encoder by unplugging the cross-attention from the model to predicting the masked words (e.g., x 2 and y 3 ). Furthermore, when fine-tuning on various downstream tasks, we can choose either plug-in or plug-out the cross-attention module on-demand, thus making it suitable for both cross-lingual language understanding (NLU) and generation tasks (NLG). For cross-lingual NLU tasks, if plugging the cross-attention module out, we can adopt the same fine-tuning methods as an encoder-only model like XLM. However, we find that plugging the cross-attention module in fine-tuning can better utilize the bilingual context to boost the performance. For cross-lingual NLG like machine translation (MT), the cross attention is already jointly pre-trained with the whole network. Therefore, the parameters of the decoder do not need to be re-adjusted substantially in the following tuning process, thus fundamentally solving the main drawback of utilizing pre-trained encoders like XLM for initializing encoder-decoder models. We call our approach VECO for Variable and Flexible Cross-lingual Pre-training. We validate VECO on a variety of representative cross-lingual understanding and generation benchmarks. Regrading cross-lingual understanding tasks, we conduct experiments on the XTREME benchmark consisting of 9 cross-lingual tasks, including text classification, sequence labeling, question answering, and sentence retrieval. VECO ranks first at the XTREME leaderboard 2 at the submission deadline. Regrading cross-lingual generation tasks, we validate VECO on the widely used WMT14 English-German and English-French machine translation benchmarks. VECO obtains 44.5 and 31.7 BLEU scores, consistently outperforming existing cross-lingual pre-training approaches and state-of-the-art Transformer variants by around 1  2 BLEU. 2 https://sites.research.google/xtreme 2 Pre-training of VECO 2.1 Overview of VECOVECO extends from a multi-layer Transformer encoder and plugs a cross-attention module in each layer. Given a pair of input ( x , y ) and its corrupted version ( x ,  y ) via randomly masking part of its tokens, the model builds two types of contextualized vector representation for each token:  One suit of contextual representations H , denoted as green blocks and yellow blocks in Figure 2 (c), are only build on self-attention module (i.e., unpluging the cross-attention module) in each layer. Another suit of contextual representations S , denoted as mixed color blocks in Figure 2 (c), are build on both the self-attention and cross-attention modules 3 . The model is trained to predict the masked tokens via two corresponding representations, conditioning on both its own context and paired context, respectively. Take predicting the masked words in sequence x as an example, the training objective is the cross-entropy of the gold distribution and predicted distribution P ( x |  x ) and P ( x |  y ,  x ) computed via the above two suits of contextual representations. Thus, the training objective of cross-attention masked language modeling (CA-MLM) can be formulated as L ( x , y ) =  log P ( x |  x ;  s )  log P ( x |  y ,  x ;  s ,  c )  log P ( y |  y ;  s )  log P ( y |  x ,  y ;  s ,  c ) (1) where  s and  c are the parameters of self-attention and cross-attention modules. The backbone network of VECO is composed of a stack of N Transformer layers. Each layer has three modules: a required self-attention module, a plug-and-play cross-attention module, and a required feed-forward linear module. Both self-attention and cross-attention modules are based on the multihead attention (Vaswani et al., 2017). An attention function can be described as mapping a query ( Q ) and a set of key-value ( K V ) pairs to an output. For the self-attention module, all the queries, keys and values are the same representations from the previous layer. Specifically, for the l -th Transformer layer, the output of a self-attention head A sl is computed via: Q = H l  1 W Ql (2) K = H l  1 W Kl (3) V = H l  1 W Vl (4) A sl = softmax( QKT  d k ) V (5) where H l  1 are the previous layer's outputs, W Ql , W Kl , W Vl are the parameter matrices of self-attention modules. For the cross-attention module, the queries come from the previous layer, and the keys and values come from the last layer's representations of paired input. Specifically, for the l -th layer, the output of a cross-attention head A cl is computed via: Q = S l  1 U Ql (6) K = HLU Kl (7) V = HLU Vl (8) A cl = softmax( QKT  d k ) V (9) where S l  1 are the previous layer's outputs, U Ql , U Kl , U Vl are the parameter matrices of cross-attention modules. Finally, the output HL of the last layer is used to recover the masked tokens of x , conditioning on its own context. P ( x |  x ) = softmax( f ( H Lx )) (10) P ( y |  y ) = softmax( f ( H Ly )) (11) where f is the feed-forward network that maps the output vectors into the dictionary. H Lx and H Ly are computed via Eq 2  5 when H 0 x and H 0 y are the word embeddings of x and y , respectively. Meanwhile, SL , conditioning on the context of the paired sequence  x and  y , is used to predict the masked tokens of y . where S Lx and S Ly are computed via Eq 6  9 with the corresponding word embeddings and HL . 12 Figure 3: The overview of VECO . During pre-training, a plug-and-play cross-attention module is jointly pre-trained along with the self-attention module. When fine-tuning on natural language understanding (NLU) tasks, the cross-attention module can be either plug-in or plug-out on demand. When fine-tuning on natural language generation (NLG) tasks, VECO can initialize an encoder-decoder module (the mainstream backbone model of generation tasks) since all those necessary modules in the encoder and decoder are already pre-trained. Note that when optimizing the objectives based on Eq 12 and Eq 13, we apply a stop-gradients operation (Chen and He, 2020) to HL (i.e., HL is treated as a constant in this term). This operation can largely speed up the training by avoiding the backpropagation on a 2 L -layer network. Moreover, it even stabilizes the training of deep post-layernorm Transformer, which requires non-trivial efforts regarding carefully designing learning rate schedulers and cutting-edge optimizers (Liu et al., 2020a; Bachlechner et al., 2020). As Figure 3 illustrated, when fine-tuning on various downstream tasks, one advantage of VECO is its flexibility for initializing both the encoder-only Transformer for understanding tasks and encoder-decoder Transformer for generation tasks. Beyond it, we also explore a fine-tuning approach combined with the characteristics of VECO . Due to the plug-and-play cross-attention module, we explore two fine-tuning approaches: Plug-Out fine-tuning is to unplug the cross-attention module from the pre-trained model. In other words, the architecture of the fine-tuned model is almost the same as mBERT or XLM. Specifically, the contextual representations from the last layer H Lx is used to predict the label of input x . the bilingual or automatically translated training data y is available in the downstream task. Specifically, we concatenated the two representations [ HL x : SL x ] to predict the label of x , [ H Ly : S Ly ] to predict the label of y . 4 . For pre-trained encoders like XLM, it is not a trivial problem to incorporate them into the sequence-to-sequence architecture  the mainstream backbone model of generation tasks (Zhu et al., 2020). One of the drawbacks or challenges could be that the encoder-to-decoder attention is not pre-trained. Therefore, the parameters of the decoder need to be re-adjusted along with the encoder in the following fine-tuning process (Ren et al., 2019). However, under the framework of VECO , the cross-attention is jointly pre-trained along with the whole network, making it easy to provide full initialization for sequence-to-sequence models. Specifically, the self-attention module is used to initialize both the corresponding modules in the encoder and decoder for contextual modeling, while the cross-attention module is used to initialize the encoder-to-decoder attention. It's okay whether you continue to tie the self-attention parameters during fine-tuning. Directly pre-training a sequence-to-sequence model like mBART (Liu et al., 2020b) could be another solution for NLG tasks, but we found mBART is not so effective in cross-lingual NLU tasks. We refer the reader to the Section 7 for detailed experiments and analysis. 4 Plug-In fine-tuning is not suitable for the zero-shot setting (also called cross-lingual transfer ) due to the lack of bilingual or translated pair ( x , y ) Model Architecture #Parameters Enc Layers Dec Layers #Languages #Vocab Training Data mBERT (Devlin et al., 2019) Encoder-only 110M 12 104 110k Wikipedia XLM (Lample and Conneau, 2019) Encoder-only 570M 24 100 200k Wikipedia XLM-R (Conneau et al., 2019) Encoder-only 550M 24 100 250k CommonCrawl mRASP (Lin et al., 2020) Encoder-decoder 375M 6 6 32 64k Translation MMTE (Siddhant et al., 2020) Encoder-decoder 375M 6 6 103 64k Translation mBART (Liu et al., 2020b) Encoder-decoder 680M 12 12 25 250k CommonCrawl VECO Flexible 662M 24* 50 250k CommonCrawl + Translation Table 1: Comparison of large cross-lingual models. Model Configuration We pre-train a 24-layer model with 1024 embedding/hidden size and 4096 feed-forward size. We do not use language embeddings to allow our model to better deal with downstream tasks of unseen languages. We adopt the same 250K vocabulary that is also used by XLM-R (Conneau et al., 2019). Table 1 shows the other details of baselines and VECO . Pre-Training Data We collect monolingual and bilingual corpus covering 50 languages. For monolingual training datasets, we reconstruct CommonCrawl Corpus used in XLM-R (Conneau et al., 2019). We extract 1.36TB data in 50 languages, which contains 6.5G sentences and 0.4G documents. We up/down-sample the monolingual text like XLM from each language with a smoothing parameter  = 0 . 5 . For bilingual data, we collect from the OPUS website 5 like previous works (Lample and Conneau, 2019; Chi et al., 2020b). There are 6.4G parallel sentences, covering 879 language pairs across 50 languages. See more statistics of training data in Appendix A. Optimization Settings For each iteration, we alternately sample a batch of adjacent segments from the monolingual corpus and a batch of parallel sentences from bilingual datasets to conduct a pair of masked input ( x ,  y ) . We adopt the translation language modeling (TLM) when the inputs are parallel bilingual sentences. Thus the overall training objective is the sum of TLM and the proposed CA-MLM objectives. During training, the model parameters except for cross-attention are initialized by XLM-R. We first freeze the parameters of XLM-R and only update the cross-attention parameters for faster convergence. Then, we jointly train the whole model. We pre-train our model with mixed-precision training using 64 Nvidia Telsa V100 32GB GPUs. Appendix A shows additional details. Downstream Tasks We conduct cross-lingual NLU evaluations on XTREME (Hu et al., 2020), a representative massively multilingual benchmark that consists of 9 understanding tasks over 40 languages. XTREME tasks can be classified into four different categories: (1) sentence-pair classification: XNLI (Conneau et al., 2018), PAWS-X (Yang et al., 2019); (2) structured prediction: POS (Nivre et al., 2018), Wikiann NER (Pan et al., 2017); (3) question answering: XQuAD (Artetxe et al., 2020), MLQA (Lewis et al., 2020), TyDiQA (Clark et al., 2020); (4) sentence retrieval: BUCC 2018 (Zweigenbaum et al., 2017), Tatoeba (Artetxe and Schwenk, 2019). Tasks in the first three categories are provided: 1) golden training corpus in English, 2) translated training corpus in other languages, and 3) dev/test set in all languages. For sentence retrieval tasks, no training datasets are provided. We refer the reader to Hu et al. (2020) for additional details about the datasets. Fine-tuning Setting Following previous works (Conneau et al., 2019; Hu et al., 2020), we consider two typical fine-tuning settings: (1) Cross-lingual Transfer which fine-tunes the pre-trained model using English golden data only and directly performs inference on the test data of different target languages; (2) Translate-Train-All fine-tunes a multilingual model on the concatenation of all data (golden training corpus in English and translated training corpus in other languages). Note that for two sequence-labeling tasks (POS, NER), the position of token labels in the translated text generally differs from that in the source text. Following FILTER (Fang et al., 2020), we use the model trained only on the English training dataset as a teacher, to label the translated text. To have a fair comparison with the strong baseline XLM-R (Conneau et al., 2019) Datasets XNLI PAWS-X POS NER XQuAD MLQA TyDiQA BUCC Tatoeba # Languages 15 7 33 40 11 7 9 5 33 Metrics Acc Acc F1 F1 F1/EM F1/EM F1/EM F1 Acc Avg. The detailed test results of nine tasks on the XTREME benchmark are shown in Table 2. It demonstrates that the proposed VECO outperforms previous cross-lingual models on all datasets. Compared to XLM-R, it averagely scores 5.0 and 6.6 points higher under the cross-lingual transfer and translation-train-all settings, respectively. In the cross-lingual transfer setting, VECO delivers a large improvement compared to XLM-R, especially on zero-shot sentence retrieval tasks (BUCC, Tatoeba). This phenomenon reflects that our model can better build the interdependence between languages. Thus it can better mine parallel sentences in a multilingual corpus. Under the translation-train-all setting, it can be observed that VECO with Plug-In fine-tuning (VECO in ) is better than Plug-Out fine-tuning (VECO out ). We conclude the reasons as two-fold. On the input side, the Plug-Out fine-tuning individually takes multilingual instances as input, while the Plug-In fine-tuning considers the bilingual instances 6 at each run. On the model side, the Plug-In fine-tuning can encourage correspondence across language via the cross-attention module. Note that the Plug-In fine-tuning method also outperforms FILTER (Fang et al., 2020), an enhanced cross-lingual fine-tuning method that also takes the 6 English instance with its translated one. bilingual instance as the input of XLM-R. It further demonstrates the effectiveness of VECO and its specialized fine-tuning method. We conclude the reasons for the above performance improvement as two-fold: 1) the introduction of bilingual data during pre-training, which is a direct way to enhance the cross-lingual ability of the model; 2) Stronger ability to enhance the interdependence and fusion among languages via the proposed CA-MLM pre-training tasks. To analyze which plays a leading role, we conduct a set of more fair experiments in Section 7. Datasets We choose the machine translation (MT) task, a typical cross-lingual generation scenario. In order to illustrate the generality of our approach and have a fair comparison with the most recent state-of-the-art Transformer work (Liu et al., 2020a), we choose two most widely used datasets: WMT14 English  German (En-De) and English  French (En-Fr) translation. WMT14 EnDe is a medium-resource dataset that provides 4.5M pairs for training and validation. We adopt standard newstest2014 as the test set. WMT14 En-Fr is a high-resource dataset that contains 36M pairs of parallel sentences. We use new-stest2012+newstest2013 for validation and new-stest2016 for test. We measure case-insensitive tokenized BLEU with multi-bleu.perl and de-Model WMT14 En-Fr WMT14 En-De BLEU SacreBLEU BLEU SacreBLEU Randomly Initialize Baseline 42.9 40.4 28.7 27.8 Liu et al. (2020a) 43.8 41.8 30.1 29.5 Randomly Initialize + More Bilingual Data * Baseline* -30.6 29.5 Cross-lingual Model Initialize mBART 43.2 41.0 30.0 29.1 mRASP 44.3 41.7 30.3 XLM-R 43.8 41.2 30.9 29.9 VECO 44.5 42.0 31.7 30.6 10 15 20 25 30 35 Epochs 25 26 27 28 29 30 s a c r e BLEUVECO Init. tokenized SacreBLEU 7 to avoid the influence of different tokenization and normalization between models (Post, 2018). Fine-tuning Setting We fine-tune our model using fairseq 8 toolkit and adopt comparable training settings with baselines. We run WMT 14 EnDe and En-Fr MT experiments on 16 and 32 V100 GPUs, respectively. The batch size is 64k for EnDe and 256k for En-Fr. The total training updates are set to 100k. The learning rate is 1e-4/2e-4, with linear warm-up over the first 16k steps and linear decay. We average the last 10 checkpoints and use beam search with a beam size of 5. Baselines We consider two types of Transformer baselines: randomly initialized and cross-lingual models initialized. For random initialization, we reproduce a Transformer baseline that adopts the same architecture and fine-tuning hyperparameters as VECO but with random initialization. Besides, we compare to the state-of-the-art Deep Transformer (Liu et al., 2020a). For cross-lingual encoder-decoder models, we include mBART (Liu et al., 2020b) and mRASP (Lin et al., 2020), which show impressive results on MT. Note that since we tied the self-attention weights of each encoder layer with each decoder layer, the whole parameters of mBART and VECO are comparable. We also conduct the WMT experiments for XLM-R, following the totally same fine-tuning settings as VECO , but leaving the encoder-to-decoder attention un-initialized. Table 3 (left) shows the results on the machine translation. We can observe that VECO can largely outperform the randomly initialized same-sized Transformer baseline by 2.3 BLEU points. Moreover, it even beats the (randomly initialized) state-of-the-art Deep-Transformer (Liu et al., 2020a), which is three times deep as VECO . Among the cross-lingual models, VECO can consistently outperform the best models, averaged on two datasets, by 0.8 BLEU points. Table 3 (right) displays the BLEU scores of same-sized models during training. We find that VECO initialized model can get a surprising more than 28 SacreBLEU score just after 10 epochs, which is better than the final score of the randomly initialized model at 35 epochs. It reveals that VECO can provide a fairly good initialization for the machine translation model, which can converge quickly and further boost the results. One might suspect that the main reason for the performance improvement is leveraging parallel corpus during pre-training. To figure it out, we conduct a more comparable experiment. We first train an out-of-domain Transformer model using the whole En-De parallel data (  68M) used in VECO pre-training, and then continue to train the model on the in-domain WMT14 En-De training dataset. Results are shown in Table 3 (left) marked with *. Under this set of a totally fair comparison, VECO still maintains a lead of 1.1 BLEU score. This directly confirms that the improvement in MT is not only due to the use of bilingual data. More importantly, CA-MLM ensures better use of bilingual and large-scale unlabeled multilingual corpus. Online translation applications usually have a restriction of inference time. The most direct way is to reduce the decoder layers since previous MT works (Liu et al., 2020a) have shown that deeper encoders are more worthwhile than deeper decoders. Based on this, we also explore the potential of the VECO to initialize deep encoder and shallow decoder Transformers, which is a blank in the cross-lingual pre-training works. Table 4 contrasts two ways of initializing a Transformer with n decoder layers ( n < 24) via selecting: (1) the first n layers; (2) the last n layers from a 24-layer pre-trained VECO model. We consider n = { 3 , 6 } to conduct experiments. We find that selecting the last n layers exhibits better performance than selecting the first n layers. It reveals that the last several layers play a more important role in making predictions over the whole vocabulary. Moreover, we can find that there is 0.2  0.3 BLEU gain when increasing the decoder layers from 3 to 6. However, we observe that only marginal improvement can be gained when further increasing the decoder layers from 6 to 24, which is also in line with the findings in Liu et al. (2020a). Regardless of the initialization method, the VECO initialized model can gain consistent 1  2 BLEU improvement over the randomly initialized model. We perform an ablation study to investigate where the improvement in cross-lingual NLU and NLG tasks mainly comes from. Specifically, there are three main aspects we have studied: 1. How much performance improvement comes from the parallel translation corpus used in pre-training? 2. How effective of the CA-MLM pre-training Data Models Tasks XNLI IWSLT Mono. task, especially compared to the MLM and TLM pre-training tasks? 3. How about pre-training a sequence-to-sequence model like mBART for NLU and NLG tasks? To figure out these questions, we train XLM, mBART and VECO model from scratch using the same datasets and parameter settings (see Appendix A for more details). All of them is pre-trained via MLM and TLM tasks. Note that the MLM task generally refers to predict the masked words of source language, while the TLM task generally refers to predict the words of the target language. Specifi-cally for mBART that is under the framework of encoder-decoder, the input of encoder is masked sequence  x , and the target of decoder is the masked words of source input x (for MLM task), or the parallel sentence y (for TLM task). Table 5 shows the results of two representative datasets of cross-lingual NLU and NLG. We can observe that, when using monolingual corpus only, VECO can outperform XLM by 0.8 points on the XNLI dataset and 0.3 BLEU scores on the IWSLT14 De-En translation dataset. It suggests that the CA-MLM can still benefit from adjacent sentences in monolingual corpus 9 , to be equipped with a stronger ability of contextual modeling. Moreover, when pre-training both on the monolingual and bilingual corpus, VECO can even achieve a larger improvement compared to XLM, with 3.2 and 2.1 points improvement on two datasets, respectively. It reveals that CA-MLM objective of VECO can better utilize the bilingual corpus, compared to only optimized by TLM and MLM of XLM. Moreover, we find that pre-training a sequence-to-sequence model like mBART (Liu et al., 2020b) 9 As noted in Section 4, we take two adjacent sentences in the monolingual corpus as ( x , y ) . performs worst on NLU tasks like XNLI 10 , almost 6 points worse than VECO and near 2 points worse than XLM. One possible explanation could be that the unidirectional language modeling in the decoder might be sub-optimal for NLU tasks. And even on the machine translation task, mBART still performs worse than VECO when pre-training on the same bilingual datasets. We conclude that it is because that VECO can do better in the contextual modeling of source input x via a explicit masked language modeling objective in Eq 10 applied to x 2 in Figure 2 (c). mBERT (Devlin et al., 2019) is a key step towards building a unified contextual language representation over multiple languages. It simply shares all languages' vocabulary and trains a bidirectional Transformer encoder, achieving promising results in various cross-lingual NLU tasks. There have been several extensions that follow the same encoder-only backbone as mBERT. The main difference is the introduction of more training corpus (e.g., bilingual data) and pre-training tasks. XLM (Lample and Conneau, 2019) utilizes both monolingual and bilingual corpus to perform the masked language modeling. XLM-R (Conneau et al., 2019) extends to be built on RoBERTa (Liu et al., 2019) using larger monolingual training data. Other works (Huang et al., 2019; Yang et al., 2020; Chi et al., 2020b) propose new pre-training tasks to utilize the bilingual data better. However, there are two main drawbacks of these works. First, they mainly rely on the self-attention module in the Transformer encoder to implicitly build the interdependence between languages, leading to few attention patterns across languages due to the lazy network. Second, even though they show impressive performance improvement on cross-lingual understanding tasks like XNLI, only marginal improvement has been gained on cross-lingual generation tasks like machine translation, especially on high-resource languages. A feasible solution for cross-language generation is to pre-train a denoising auto-encoder like mBART (Liu et al., 2020b). It extends BART (Lewis et al., 2019) to the multilingual setting, demonstrating significant gains in low/medium-resource machine translation, but 10 We follow BART (Lewis et al., 2019) by utilizing the final representation from the decoder for classification tasks. with a decrease in high resource languages. Unlike mBART, Chi et al. (2020a) first trains an encoder via MLM and then frozen the encoder to train the decoder only via two generative tasks. A similar approach is also proposed in Liang et al. (2020) and Lin et al. (2020), with the main difference in the joint training of encoder-decoder with code-switch tricks. However, all these cross-lingual models emphasize training a dedicated model for NLG. Thus they may hurt the NLU capabilities of the model. The ablation study in Section 7 also validates that it is sub-optimal to train an encoder-encoder network for NLU tasks. This paper endeavors to build a unified cross-lingual model for NLU and NLG tasks via a plug-and-play cross-attention module. More importantly, the cross-attention module plays a role in the explicit alignment of encoded representations of different languages, thus largely contributing to building a unified cross-lingual model. We present VECO , a variable and flexible cross-lingual pre-training model, targets at explicitly capturing the interdependence between languages via a plug-and-play cross-attention module. Based on the flexible characteristics, VECO can initialize both NLU preferred encoder-only and NLG specialized encoder-decoder Transformer. Moreover, we also introduce a Plug-In fine-tuning approach to encourage the fusion between languages, combining the feature of VECO and cross-language downstream tasks. Taken together, VECO achieves consistent improvements on various language understanding and generation tasks, broadening the way of thinking about pre-trained backbone architecture and fine-tuning methods under the cross-lingual scenario.
abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain method method method result abstain abstain abstain abstain method abstain method abstain other abstain objective objective objective abstain abstain other abstain abstain abstain abstain abstain other abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain other method method method abstain method method method method abstain method abstain method method abstain abstain method method method abstain method other abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method abstain method method abstain method method abstain abstain abstain method method method method method method abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other other other other other other other other abstain other other other other other other method other method abstain method abstain