Multilingual neural machine translation models are trained to maximize the likelihood of a mix of examples drawn from multiple language pairs. The dominant inductive bias applied to these models is a shared vocabulary and a shared set of parameters across languages; the inputs and labels corresponding to examples drawn from different language pairs might still reside in distinct subspaces. In this paper, we introduce multilingual crossover encoder-decoder ( mXEncDec ) to fuse language pairs at an instance level. Our approach interpolates instances from different language pairs into joint crossover examples' in order to encourage sharing input and output spaces across languages. To ensure better fusion of examples in multilingual settings, we propose several techniques to improve example interpolation across dissimilar languages under heavy data imbalance. Experiments on a large-scale WMT multilingual dataset demonstrate that our approach significantly improves quality on English-to-Many, Many-to-English and zero-shot translation tasks (from +0 . 5 BLEU up to +5 . 5 BLEU points). Results on code-switching sets demonstrate the capability of our approach to improve model generalization to out-of-distribution multilingual examples. We also conduct qualitative and quantitative representation comparisons to analyze the advantages of our approach at the representation level. Multilingual modeling has been receiving increasing research attention over the past few years, arising from successful demonstrations of improved quality across a variety of tasks, languages and modalities (Lample and Conneau, 2019; Arivazhagan et al., 2019b; Conneau et al., 2021). The success of these models is typically ascribed to vocabulary sharing, parameter tying and implicit pivoting through dominant languages like English (Conneau et al., 2020). These conventional techniques are effective, but might not be exploiting the full potential of multilingual models to learn the underlying inductive bias: the learning signal from one language should benefit the quality of other languages (Caruana, 1997; Arivazhagan et al., 2019b). Here we study two related issues that exist in the context of multilingual Neural Machine Translation (NMT) training (Dong et al., 2015; Firat et al., 2016a; Johnson et al., 2017). First, NMT models (Bahdanau et al., 2015; Vaswani et al., 2017) are trained with maximum likelihood estimation which has a strong tendency to overfit and even memorize observed training examples, particularly posing challenges for low resource languages (Zhang et al., 2018). Second, training examples from distinct language pairs are separately fed into multilingual NMT models without any explicit instance-level sharing (with the exception of multi-source NMT (Zoph and Knight, 2016; Firat et al., 2016b)); as a consequence, given large enough capacity, the models have the liberty to map representations of different languages into distinct subspaces, limiting the extent of cross-lingual transfer. In this work, we introduce multilingual crossover encoder-decoder ( mXEncDec ) to address these issues following the recent work on XEncDec (Cheng et al., 2021) and mixup (Zhang et al., 2018; Cheng et al., 2020; Guo et al., 2020). Inspired by chromosomal crossovers (Rieger et al., 2012), mXEncDec fuses two multilingual training examples to generate crossover examples inheriting the combinations of traits of different language pairs, which is capable of explicitly capturing cross-lingual signals compared to the standard training which mechanically combines multiple language pairs. mXEncDec has the following advantages: 1. Enhancing the cross-lingual generalization. Thanks to crossover examples generated by fusing different language pairs, the multilingual NMT is encouraged to learn to transfer 4092 explicitly via more languages rather than implicitly via the predominant languages. 2. Improving the model generalization and robustness. As vicinity examples around each example in the multilingual corpus (akin to Vicinal Risk Minimization (Chapelle et al., 2001)), crossover examples produced by mXEncDec can enrich the support of the training distribution and lead to better generalization and robustness respectively on general and noisy inputs (Zhang et al., 2018). 3. Alleviating overfitting to low-resource languages. mXEncDec can increase the diversity of low-resource languages by fusing low-resource examples with others, instead of the simple duplication in the standard training. In mXEncDec , we randomly pick up two training examples drawn from the multilingual training corpus and first interpolate their source sentences where we have to prudently deal with language tags. Then we leverage a mixture decoder to produce a virtual target sentence. To account for heavy data imbalance of each language pair, we propose a pairwise sampling strategy to adjust interpolation ratios between language pairs. We also propose to simplify the target interpolation to cope with noisy attention and fusions of dissimilar language pairs. Different from XEncDec fusing two heterogeneous tasks (Cheng et al., 2021), we attempt to adapt it to deeply fuse different language pairs. Experimental results on a large-scale WMT multilingual dataset show that mXEncDec yields improvements of +1 . 13 and +0 . 47 BLEU points averagely on xx-en and en-xx test sets over a vanilla multilingual Transformer model. We also evaluate our approaches on zero-shot translations and obtain up to +5 . 53 BLEU points over the baseline method, which corroborates the better transfer-abilty of multilingual models with our approaches. The more stable performance on noisy input text demonstrates the capability of our approach to improve the model robustness. To further explain the model behaviors at the representation level, qualitative and quantitative comparisons on representations manifest that our approach learns better multilingual representations, which indirectly explicates the BLEU improvements. optimizes the conditional probability P ( y | x ;  ) of translating a source-language sentence x into a target-language sentence y . The encoder reads the source sentence x = x 1 , ..., x I as a sequence of word embeddings e ( x ) . The decoder acts as a conditional language model over embeddings e ( y ) and the encoder outputs with a cross-attention mechanism (Bahdanau et al., 2015). For clarity, we denote the input and output in the decoder as z and y , i.e. , z = (cid:104) s (cid:105) , y 1 ,    , y J  1 as a shifted copy of y , where (cid:104) s (cid:105) is a sentence start token. Then the decoder generates y as P ( y | x ;  ) = (cid:81) Jj =1 P ( y j | z  j , x ;  ) . The cross-attention matrix is denoted as A  RJ  I . NMT optimizes the parameters  by maximizing the likelihood of a parallel training set D : LD (  ) = E ( x , y ) D [ (cid:96) ( f ( x , y ;  ) , v ( y ))] , (1) where (cid:96) is the cross entropy loss between the model prediction f ( x , y ;  ) and label vectors v ( y ) for y . v ( y ) could be a sequence of one-hot vectors with smoothing in Transformer (Vaswani et al., 2017). Multilingual NMT extends NMT from the bilingual to the multilingual setting, in which it learns a one-to-many, many-to-one or many-to-many mapping from a set of languages to another set of languages (Firat et al., 2016a; Johnson et al., 2017). More specifically, the multilingual NMT model is learned over parallel corpora M = {D l i } Li =1 where L is the number of language pairs: LM (  ) = ED li M E ( x , y ) D li [ (cid:96) ( f ( x , y ;  ) , v ( y ))] , (2) where all the parallel training sets are fed into the NMT model. XEncdec: Crossover Encoder-Decoder . XEncDec aims to fuse two parallel examples (called parents) in the encoder-decoder model (Cheng et al., 2021). The parents' source sentences are shuffled into a sentence (the offspring's source) on the encoder side, and a mixture decoder model predicts a virtual target sentence (the offspring's target). Given a pair of examples ( x , y ) and ( x (cid:48) , y (cid:48) ) where their lengths are different in most cases, padding tokens are appended to the shorter one to align their lengths. The crossover example ( x ,  y ) (offspring) is generated by carrying out XEncDec over ( x , y ) and ( x (cid:48) , y (cid:48) ) (parents). where m = m 1 ,    , m |  x |  { 0 , 1 } |  x | is sampled from a distribution or constructed according to a hyperparameter ratio p ; e.g., p = 0 . 15 means that 15% of elements in m are 0 . |  x | is the length of  x , which is equal to max( | x | , | x (cid:48) | ) . On the crossover decoder side, a mixture conditional language model is employed for the generation of the virtual target sentence. The input embedding e ( z j ) and output label v ( y j ) for the decoder at the j -th position are calculated as: e ( z j ) = e ( y j  1 ) t j  1 + e ( y (cid:48) j  1 )(1  t j  1 ) , (4) v ( y j ) = v ( y j ) t j + v ( y (cid:48) j )(1  t j ) , (5) where t = t 1 , ..., t |  y |  [0 , 1] |  y |  R |  y | . In contrast to a common language model fed with a single word y j  1 for predicting y j at the j -th position, the crossover decoder aims to generate an interpolated vector v ( y j ) by averaging v ( y j ) and v ( y (cid:48) j ) with t j , on condition that the current input embedding is also weighted on embeddings e ( y j  1 ) and e ( y (cid:48) j  1 ) with t j  1 . The weight vector t used for interpolating target inputs and labels is computed as: t j = (cid:80) Ii =1 A ji m i (cid:80) Ii =1 A ji m i + (cid:80) I (cid:48) i =1 A (cid:48) ji (1  m i ) , (6) where A and A (cid:48) are the alignment matrices for ( x , y ) and ( x (cid:48) , y (cid:48) ) . In practice the cross-attention scores in the NMT model are utilized as an alternative noisy alignment matrix (Garg et al., 2019). The cross-entropy is utilized to compute the loss for XEncDec when feeding e ( x ) , e ( z ) and v ( y ) into the encoder-decoder model, denoted as: (cid:96) ( f ( x ,  y ;  ) , v ( y )) = (cid:88) j KL ( v ( y j ) (cid:107) P ( y |  z  j ,  x ;  )) . (7) 3 mXEncDec In this work, we aim to leverage XEncDec to encourage multilingual NMT models to better exploit cross-lingual signals with crossover examples created by explicitly fusing different language pairs. We introduce its variant, called mXEncDec as shown in Figure 1, in which the parent examples could belong to either the same or different language pairs. The subsequent subsections discuss (cid:39)(cid:86)(cid:83)(cid:87)(cid:87)(cid:51)(cid:90)(cid:73)(cid:86) (cid:80)(cid:69)(cid:82)(cid:75)(cid:89)(cid:69)(cid:75)(cid:73)(cid:4)(cid:88)(cid:83)(cid:79)(cid:73)(cid:82)(cid:87) (cid:81)(cid:60)(cid:41)(cid:82)(cid:71)(cid:40)(cid:73)(cid:71) (cid:91)(cid:83)(cid:86)(cid:72)(cid:87) Figure 1: An illustration of multilingual crossover encoder-decoder ( mXEncDec ). Language Interpolation . As multilingual NMT involves a large number of language pairs, several techniques have been adopted to distinguish translation directions among them, such as prepending a language tag to source inputs (Johnson et al., 2017) or both source and target sentences (Wang et al., 2018), training language-specific embeddings for different languages (Lample and Conneau, 2019), and so on (Dabre et al., 2020). When following Lample and Conneau (2019), it is natural to interpolate language-specific embeddings as we do for token embeddings. However, if we want to adopt a language tag in the first word of a source sentence to indicate the target language (Johnson et al., 2017), we need to address how to interpolate them. As Figure 1 shows, to make the sentence  x still carry language-specific information from x and x (cid:48) , we conduct a soft combination over their language tags, that is: e ( x 1 ) = e ( x 1 ) (cid:80) | m | i =2 m i | m |  1 + e ( x (cid:48) 1 ) (cid:80) | m | i =2 (1  m i ) | m |  1 , (8) where | m | is the length of m . e ( x 1 ) captures the proportion of words in  x coming from the translation pairs ( x , y ) and ( x (cid:48) , y (cid:48) ). Simplified Target Interpolation . In comparison to bilingual NMT, attention matrices learned in multilingual NMT models are excessively noisy, which results in an inappropriate design of using the attention-based target interpolation in Eq. (6) for mXEncDec . Instead, we can employ a simple linear interpolation by setting t as a constant vector, here exemplified by the case of using language tags: t j = (cid:80) | m | i =2 m i | m |  1 ,  j  { 1 , ..., |  y |} , (9) 4094 A similar equation can be obtained for using language embeddings. In addition, dispensing with attention can improve the parallel efficiency with 10% speed-up gain. Hard Target Input Interpolation . For multilingual NMT with multiple languages on the target side, i.e., one-to-many and many-to-many models, we need to carefully design combinations of target input word embeddings. As representations from the same language are usually close to each other, it can still augment the representation space by linearly interpolating target embeddings in Eq. (4). But for dissimilar languages, in particular distantly related languages, the interpolation points between them are comparatively unreliable. To tackle this issue, we simply quantize t j to 1 if t j > 0 . 5 , otherwise t j = 0 when interpolating target input embeddings for two different target languages in Eq. (4). A better solution should consider varying the interpolation ratio based on the language similarity or encourage interpolations of similar languages. We leave this for future exploration. Pairwise Sampling . The multilingual corpus is usually heavily imbalanced: most of its data distribution concentrates on high-resource language pairs (Arivazhagan et al., 2019b). When interpolating high-resource and low-resource sentence pairs, we assume the fusion should be encouraged to be in favor of high-resource language pairs because the representation space supported by high-resource sentences is relatively reliable and stable (Kudugunta et al., 2019). This indicates a more frequent small p (e.g. p < 0 . 5 ) to weigh high-resource sentences over low-resource sentences if ( x , y )  D l i is a high-resource sentence and ( x (cid:48) , y (cid:48) )  D l j is a low-resource sentence. To this end, we propose a pairwise sampling method to sample the source shuffle ratio p l i ,l j for interpolating language pair l i and l j : g  Bernoulli (1 / (1 + exp (  d ( l i , l j ))) , (10) p l i ,l j = gp + (1  g )(1  p ) , (11) where  is a temperature hyperparameter to control the tendency of g towards 0 or 1 for the Bernoulli distribution. d ( l i , l j ) can be an arbitrary metric to measure the relationship between language l i and l j . Here we use d ( l i , l j ) = | D l i | / | D l j | where | D l i | denotes the data size of the language pair l i . LX (  ) = ED li M ED lj M E ( x , y ) D li E ( x (cid:48) , y (cid:48) ) D lj [ (cid:96) ( f ( x ,  y ;  ) , v ( y ))] , (12) where the generation of ( x ,  y ) depends on ( x , y ) and ( x (cid:48) , y (cid:48) ) . Algorithm 1 shows how to compute Eq. (12) effectively. We shuffle the min-batch consisting of all the language pairs. Then the shuffled batch and original batch can be used to generate ( x ,  y ) to compute the mXEncDec loss. Instead of using one-hot labels v ( y j ) in Eq. (5), we adopt label co-refinement (Li et al., 2019) by linearly combing the ground-truth one-hot label with the model prediction, that is v ( y j )  + f j ( x , y ;   )(1   ) . Finally, our approach optimizes the model loss involving two training losses, Eq. (2) and Eq. (12):   = argmin {L M (  ) + LX (  ) } . Data and Evaluation . We conduct experiments on the English-centric WMT multilingual dataset composed of 16 languages (including English) and 30 translation directions from past WMT evaluation campaigns before and on WMT' 19 (Barrault et al., 2019). The data distribution is highly skewed, varying from roughly 10k examples in En-Gu to roughly 60M examples in En-Cs. Two non-English test sets, Fr-De and De-Cs, are used to verify zero-shot translations. In addition, we also use multi-4095  = -2 -0.8 -0.4 0 0.4 0.8 2 xx-en 27.22, 27.42, 27.21, 27.41, 27.46, 27.60, 27.41 en-xx 21.76, 21.83, 21.74, 21.87, 21.89, 22.01, 21.87 Table 1: Effect of the temperature  in the pairwise sampling. To mitigate the data imbalance in the WMT multilingual corpus, we follow Arivazhagan et al. (2019b) and adopt a temperature-based data sampling strategy to over-sample the low-resource languages where the temperature is set to 5 . We apply SentencePiece (Kudo and Richardson, 2018) to learn a vocabulary of 64 k sub-words. We perform experiments in three settings: many-to-one, one-to-many and many-to-many translations. The 15 test language pairs are cast into three groups according to their data size: High ( > 10 M , 5 languages), Low ( < 1 M , 7) and Medium ( > 1 M & < 10 M , 3). We report not only the average detokenized BLEU scores for each group as calculated by the SacreBLEU script (Post, 2018) but also winning ratio (WR) indicating the ratio of all the test sets on which our approach beats the baseline method. Models and Hyperparamters . Following Chen et al. (2018), we select the Transformer Big (6 layer, 1024 model dimension, 8192 hidden dimension) as the backbone model and implement them with the open-source Lingvo (Shen et al., 2019). Adafactor (Shazeer and Stern, 2018) is adapted as our training optimizer, in which the learning rate is set to 3 . 0 and adjusted with 40 k warm-up steps. We use a beam size of 4 and a length penalty of 0 . 6 for all the test sets. We apply language-specific embeddings to both many-to-one and one-to-many models while languages in many-to-many models are specified with language tags. Many-to-one and one-to-many models are optimized for 150 k steps while many-to-many models run for 300 k steps. All Transformer models utilize a large batch of around 5600  64 tokens over 64 TPUv4/TPUv3 chips. We average the last 8 checkpoints to report model performance. We tune p over the set: { 0 . 10 , 0 . 15 , 0 . 25 , 0 . 50 } and set it to 0 . 15 except for many-to-one using 0 . 25 . The temperature  used in Eq. (10) to sample the shuffle ratio is selected over the set { 0 ,  0 . 4 ,  0 . 8 ,  2 . 0 } . = 0 . 8 is selected for many-to-many models while  = 0 is for others as Table 1 suggests. The parameter  in label co-refinement is annealed from 0 to 0 . 7 in the first 40 K steps. We find that a non-zero and non-one  can not only better capture informative label but also substantially improve the training stability. Training Efficiency . If we adopt the simplified target interpolation, the loss computations for LM (  ) and LX (  ) in Eq. (13) are totally independent. But we have to halve the batch size to load interpolation examples ( LX (  ) ) into memory. To make the 4096 Method Many-to-Many xx-en en-xx Low Med. baseline models and our models observe the same amount of parallel examples per step, we double the number of TPUs to compensate for it. many-to-one, one-to-many and many-to-many settings:  mXEncDec -A: the target interpolation t is computed by normalizing attention in Eq. (6). mXEncDec -S: the target interpolation t is simplified to a constant vector in Eq. (9). We compare mXEncDec to the baseline methods:  MLE: the vanilla Multilingual NMT is trained with maximum likelihood estimation. mixup : we adapt mixup (Zhang et al., 2018) to multilingual NMT by mixing source and target sequences following the methods proposed in Cheng et al. (2020) and Guo et al. (2020). For a fair comparison, we also mix co-refined labels rather than one-hot labels. The comparisons between the baseline MLE and our approach suggest that mXEncDec can improve the translation performance on both xx-en and en-xx translation settings (up to +1 . 06 BLEU & 93 . 33 WR on xx-en and +0 . 47 BLEU & 86 . 66 WR on en-xx). In particular, using simplified target interpolation to substitute the noisy attention-based interpolation ( mXEncDec -S vs. mXEncDec -A) can achieve better results on xx-en translations (+0.64 BLEU) while slightly performing worse on en-xx translations (-0.16 BLEU). After incorporating quantized target interpolation, it yields an additional improvement for mXEncDec -S on en-xx translations (+0.32 BLEU). The improvement differences between xx-en and en-xx (+1.06 BLEU vs. +0.47 BLEU) to some extent imply that interpolations on the target side are more favourable to similar languages, and interpolations on the encoder side are not sensitive to language types. Table 3 shows results for many-to-many models. Among all the training methods, our approaches still obtain the best results for both xx-en and en-4097 0.0 0.1 0.2 0.3 0.4 0.5 Noise Fraction 14 16 18 20 22 24 26 28 BLEUMLE mixup mXEncDec-A mXEncDec-S * xx translations (up to +1 . 13 BLEU & 100 WR on xx-en and +0 . 46 BLEU & 73.33 WR). We consistently find that mXEncDec -S benefits much more from the quantized target interpolation with +0.68 BLEU on xx-en and +0.21 BLEU on en-xx. Although this technique slightly impairs the performance of mXEncDec -A on both xx-en and en-xx translations, it significantly boosts its zero-shot translations as shown in Table 4. We also observe that removing the pairwise sampling with  = 0 has big negative effects on high-resource language pairs for many-to-many models. Pairwise sampling can not only stabilize the performance on low-resource language pairs and significantly improve high-resource language pairs. Compared to mixup , our approaches still attain better performance except that mXEncDec -A on xx-en performs slightly worse. mixup trains models on linear interpolations of examples and their labels. By contrast, mXEncDec combines training examples in a non-linear way on the source side, and encourages the decoder to decouple the non-linear interpolation with a ratio related to the source end. To further verify cross-lingual transfer of our approaches, we utilize many-to-many models to decode language pairs not pesent in the training data, i.e., zero-shot sets from WMT and FLORES. In Table 4, our approaches achieve notable improvements across all the test sets compared to baseline methods. On average, our best approach ( mXEncDec -A + Hard) can gain up to +4 . 49 BLEU over MLE. Interestingly, this model is not the best on general translations but delivers the best results on zero-shot translations. These substantial improvements demonstrate the strong transferability of our approaches. We construct a noisy test set comprising code-switching noise to test the robustness of multilingual NMT models (Belinkov and Bisk, 2018; Cheng et al., 2019). Following the method proposed in Cheng et al. (2021), we randomly replace a certain ratio of English/non-English source words with non-English/English target words by resorting to an English-centric dictionary. From results in Figure 2, we find our approaches to exhibit higher robustness with larger improvements as the noise fraction increases. mXEncDec -A shows similar robustness to mXEncDec -S  on zero-shot translations and even higher robustness on xx-en translations although its performance on clean test sets falls behind mXEncDec -S  . mXEncDec -S  performs significantly better on en-xx translations compared to other approaches. Moreover, it is noteworthy that our approaches have better stability on xx-en translations where we replace non-English words with English counterparts, which is in complete agreement with the finding in section 4.4 that English representations tend to be fused into non-English representations by virtue of our approaches. To better interpret the advantages of our approaches over baselines, we attempt to delve deep into the representations incurred by models. A common method is to study the encoder representations of multilingual NMT models (Kudugunta et al., 2019), which we follow. We aggregate the sentence representations by averaging the encoder outputs. The data computing representations come from FLORES (Goyal et al., 2021) as it provides a high quality of multi-way translations implying that sentences from each language are semantically equivalent to each other. We use the first 100 sentences in 4098 en cs fr ru zh es fi de et lv lt ro hi kk tr gu We argue that the encoder in a good multilingual NMT model prefers to distribute sentence representations based on their semantic similarities rather than language families. Figure 3 depicts visualisa-tions of representations plotted by t-SNE (Van der Maaten and Hinton, 2008) on xx-en translations. We make the following observations: 1. In each figure, sentences with the same semantics incline to form a single cluster. 2. For MLE in Figure (a), most sentences are dispersed into each cluster based on semantics while extremely low-resource languages (Hi, Gu, Kk) and English possess their own distinct clusters. 3. For mixup , mXEncDec -A and mXEncDec -S  in Figure (b)-(d), sentences from extremely low-resource languages start to be assimilated into their own semantic clusters. 4. For mXEncDec -A and mXEncDec -S  in Figure (c)-(d), English sentences attempt to fuse into representations of other languages. English sentences prefer to become an individual cluster. Because when using the language tag <2en> to compute English encoder representations, it is treated as a copy task instead of translation tasks for computing representations of other languages. However, our approach promotes English sentences to be closer to their semantic equivalents in other languages. This leads to enhanced robustness toward code-switching noise when translating sentences in languages that are mixed with English codes. The evident representation amelioration for extremely low-resource languages corroborates significant BLEU improvements on low-resource translations in Table 2 and Table 3. The encoder learned by our approach performs the best and complies with our argument. We also conduct quantitative analyses to evaluate the clustering effect of each method in Figure 3. In Table 5, we adopt three clustering metrics, SC (Silhouette Co-efficient), CH (Calinski-Harabaz Index), and DB 1 We also have similar findings from visualizations for en-xx translations. (Davies-Bouldin Index). Although these metrics cannot adequately assess multilingual representations as they advocate distinct separations between different clusters and tight closeness within the same cluster, we believe they can still measure the within-cluster variance in part. Among them, mXEncDec -S  performs the best while mixup and mXEncDec -A yield similar performance. Multilingual NMT has made tremendous progress in recent years (Dong et al., 2015; Firat et al., 2016a; Johnson et al., 2017; Arivazhagan et al., 2019b; Fan et al., 2021). Recent research efforts to improve the generalization of multilingual models concentrate on enlarging the model capacity (Huang et al., 2019; Zhang et al., 2020; Lep-ikhin et al., 2020), incorporating hundreds of languages (Fan et al., 2021), pretraining multilingual models (Liu et al., 2020), and introducing additional regularization constraints (Arivazhagan et al., 2019a; Al-Shedivat and Parikh, 2019; Yang et al., 2021). Our work is related to the last three ones in that they try to enable models to better transfer across languages by introducing an alignment loss to learn an interlingua (Arivazhagan et al., 2019a) or imposing an agreement loss on translation equivalents (Al-Shedivat and Parikh, 2019; Yang et al., 2021). However, we propose to utilize mXEncDec to directly combine language pairs for better exploitation of cross-lingual signals. Another related research line is data mixing. Since mixup (Zhang et al., 2018; Yun et al., 2019) was proposed in computer vision, we have observed great success in NLP (Guo et al., 2019; Cheng et al., 2020; Guo et al., 2020; Cheng et al., 2021). mXEncDec shares the commonality of combining example pairs as inspired by XEncDec (Cheng et al., 2021). To the best of our knowledge, we are the first to fuse different language pairs to improve cross-lingual generalization and robustness for multilingual NMT. We have presented mXEncDec to fuse different language pairs at instance level for multilingual NMT, which enables the model to better exploit cross-lingual signals. Experimental results on general, zero-shot and noisy test sets demonstrate that our approach can significantly improve the cross-lingual generalization, zero-shot transfer and robustness of multilingual NMT models. Representation analyses further confirms that our approach is capable of learning better multilingual representations, which coincides with improvements in BLEU. We plan to investigate whether this approach can improve the model generalization in a broader scope like domain generalization. We find that mXEncDec can easily achieve notable improvements for xx-en translations because they share an identical target language. However, there still exits huge headroom for en-xx translations. We plan to explore how to interpolate target languages more effectively, for example, possibly considering language similarity.
abstain abstain abstain method objective objective objective method abstain abstain abstain method abstain other method abstain abstain objective objective objective objective objective objective method objective objective abstain abstain abstain abstain result abstain objective abstain other other other abstain other other other other other other other other other other other other other other other other other other other other objective abstain other abstain other method abstain other abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain other abstain method abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain result result abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain other other abstain objective other abstain other objective method objective method objective result abstain objective