Complete Multi-lingual Neural Machine Translation (C-MNMT) achieves superior performance against the conventional MNMT by constructing multi-way aligned corpus, i.e., aligning bilingual training examples from different language pairs when either their source or target sides are identical. However, since exactly identical sentences from different language pairs are scarce, the power of the multi-way aligned corpus is limited by its scale. To handle this problem, this paper proposes "Extract and Generate" (EAG), a two-step approach to construct large-scale and high-quality multi-way aligned corpus from bilingual data. Specifically, we first extract candidate aligned examples by pairing the bilingual examples from different language pairs with highly similar source or target sentences; and then generate the final aligned examples from the candidates with a well-trained generation model. With this two-step pipeline, EAG can construct a large-scale and multi-way aligned corpus whose diversity is almost identical to the original bilingual corpus. Experiments on two publicly available datasets i.e., WMT-5 and OPUS-100, show that the proposed method achieves significant improvements over strong baselines, with +1.1 and +1.4 BLEU points improvements on the two datasets respectively. Multilingual Neural Machine Translation (MMMT) (Dong et al., 2015; Firat et al., 2017; Johnson et al., 2017; Aharoni et al., 2019) has achieved promising results on serving translations between multiple language pairs with one model. With sharing parameters of the model, MNMT can facilitate information sharing between similar languages and make it possible to translate between low-resource  Equal contribution. and zero-shot language pairs. Since the majority of available MT training data are English-centric, i.e., English either as the source or target language, most non-English language pairs do not see a single training example when training MNMT models (Freitag and Firat, 2020). Therefore, the performance of MNMT models on non-English translation directions still left much to be desired: 1) Lack of training data leads to lower performance for non-English language pairs(Zhang et al., 2021); 2) MNMT models cannot beat the pivot-based baseline systems which translate non-English language pairs by bridging through English (Cheng et al., 2016; Habash and Hu, 2009). Recently, Freitag and Firat (2020) re-kindle the flame by proposing C-MNMT, which trains the model on the constructed multi-way aligned corpus. Specifically, they extract the multi-way aligned examples by aligning training examples from different language pairs when either their source or target sides are identical (i.e., pivoting through English, for German  English and English  French to extract German-French-English examples). Since they directly extract the multi-way aligned examples from the bilingual corpus, we refer to their approach as the extraction-based approach. Despite improving the performance, the scale of multi-way aligned corpus extracted by Freitag and Firat (2020) is always limited compared to English-centric bilingual corpus, e.g., only 0.3M German-Russian-English multi-way aligned corpus extracted from 4.5M German-English and 33.5M English-Russian bilingual corpus. A simple idea for remedying this problem is to add the roughly-aligned corpus by extracting the training examples when either their source or target sides are highly similar. However, our preliminary experiments show that the performance of the model decreases dramatically when we train the model with appending the roughly-aligned corpus. 1 One possible solution, referred 1 Detailed descriptions about the preliminary experiment 8141 to as the generation-based approach, is to generate the multi-way aligned examples by distilling the knowledge of the existing NMT model, e.g., extracting German-English-French synthetic three-way aligned data by feeding the English-side sentences of German-English bilingual corpus into the English-French translation model. Although the generation-based approach can theoretically generate non-English corpus with the same size as original bilingual corpus, its generated corpus has very low diversity as the search space of the beam search used by NMT is too narrow to extract diverse translations (Wu et al., 2020; Sun et al., 2020; Shen et al., 2019), which severely limits the power of the generation-based approach. In order to combine advantages of the two branches of approaches mentioned above, we propose a novel two-step approach, named EAG (Extract and Generate), to construct large-scale and high-quality multi-way aligned corpus for C-MNMT. Specifically, we first extract candidate aligned training examples from different language pairs when either their source or target sides are highly similar; and then we generate the final aligned examples from the pre-extracted candidates with a well-trained generation model. The motivation behind EAG is two-fold: 1) Although identical source or target sentences between bilingual examples from different language pairs are scarce, highly similar sentences in source or target side are more wide-spread; 2) Based on the pre-extracted candidate aligned examples which have highly similar source or target sentences, EAG can generate the final aligned examples by only refining the sentences partly with a few modifications. Therefore, the non-English corpus constructed by EAG has almost identical diversity to the original bilingual corpus. Experiments on the publicly available data sets, i.e., WMT-5 and OPUS-100, show that the proposed method achieves substantial improvements over strong baselines. Bilingual NMT Neural machine translation (Sutskever et al., 2014; Cho et al., 2014; Vaswani et al., 2017) achieves great success in recent years due to its end-to-end learning approach and large-scale bilingual corpus. Given a set of sentence pairs D = { ( x, y )  ( X  Y ) } , the NMT model is trained to learn the parameter  by maximizing can be found in Section 5.1. MNMT Considering training a separate model for each language pair is resource consuming, MNMT (Dong et al., 2015; Johnson et al., 2017; Gu et al., 2020) is introduced to translate between multiple language pairs using a single model (John-son et al., 2017; Ha et al.; Lakew et al., 2018). We mainly focus on the mainstream MNMT model proposed by Johnson et al. (2017), which only introduces an artificial token to the input sequence to indicate which target language to translate. C-MNMT C-MNMT is proposed to build a complete translation graph for MNMT, which contains training examples for each language pair (Freitag and Firat, 2020). A challenging task remaining is how to get direct training data for non-English language pairs. In Freitag and Firat (2020), non-English training examples are constructed by pairing the non-English sides of two training examples with identical English sides. However, this method can't get large-scale training examples since the quantity of exactly identical English sentences from different language pairs is small. Another feasible solution is to generate training examples with pivot-based translation where the source sentence cascades through the pre-trained source  English and English  target systems to generate the target sentence (Cheng et al., 2016). Despite a large quantity of corpus it can generate, its generated corpus has very low diversity (Wu et al., 2020; Sun et al., 2020; Shen et al., 2019). The proposed EAG has a two-step pipeline. The first step is to extract the candidate aligned examples from the English-centric bilingual corpus. The second step is to generate the final aligned examples from the candidates extracted in the first step. Different from Freitag and Firat (2020) who extract non-English training examples by aligning the English-centric bilingual training examples with identical English sentences, we extract the candidate aligned examples by pairing two English-centric training examples with highly similar English sentences. Various metrics have been proposed to measure the superficial similarity of two sentences, such as TF-IDF (Aizawa, 2003; Huang et al., 2011), edit distance (Xiao et al., 2008; Deng 8142 Figure 1: Examples constructed by EAG. et al., 2013), etc. In this paper, we take edit distance as the measurement to decide the superficial similarity of two English sentences. Three main considerations are behind. Firstly, since edit distance measures the similarity of two sentences with the minimum number of operations to transform one into the other, it tends to extract sentences with similar word compositions and sentence structures. Secondly, since edit distance only utilizes three operations, i.e., removal, insertion, or substitution, it is easier to mimic these operations in the process of generating the final aligned examples (we leave the explanation in the next subsection). Finally, unlike TF-IDF which only considers word bags in two sentences, edit distance also considers the word order in each sentence. Formally, given two English-centric bilingual corpora from two different language pairs { X 1 , Y 1 } and { X 2 , Y 2 } , where X 1 and X 2 are English sides, Y 1 and Y 2 belong to language L a and L b respectively. For sentence pair ( x 1 i , y 1 i )  { X 1 , Y 1 } and ( x 2 j , y 2 j )  { X 2 , Y 2 } , we take ( x 1 i , y 1 i , x 2 j , y 2 j ) as a candidate aligned example if the two English sentences x 1 i and x 2 j meets: f ed ( x 1 i , x 2 j )    min ( | x 1 i | , | x 2 j | ) ,   (0 , 1) (1) where f ed refers to the function of edit distance calculation, | x | represents the length of the sentence x ,  is the similarity threshold which can be set by users beforehand to control the similarity of sentences in the candidate aligned examples. With setting  = 0 , we can directly extract the same multi-way aligned examples with Freitag and Firat (2020). With larger  , more candidate aligned examples can be extracted for looser restriction. Accordingly, there are more noises in the extracted candidate aligned examples. In the extracted candidate aligned example ( x 1 i , y 1 i , x 2 j , y 2 j ) , ( x 2 j , y 2 j ) is not well aligned to ( x 1 i , y 1 i ) if f ed ( x 1 i , x 2 j ) does not equal to zero. To construct the final three-way aligned example, we search for one sentence pair ( x 2 j ,  y 2 j ) in the language pair { X 2 , Y 2 } , where  x 2 j has the same meaning to x 1 i (thus ( x 1 i , y 1 i ,  y 2 j ) is a three-way aligned example). Unfortunately, it is very difficult for us to directly find such a sentence pair in the large search space. However, considering  x 2 j and x 1 i are both in English, we can take an extreme case where  x 2 j is identical to x 1 i in the superficial form. Now, the remained question is that we need to search for the sentence  y 2 j in language L b , which has the same meaning to x 1 i . By comparing ( x 1 i ,  y 2 j ) with ( x 2 j , y 2 j ) , as x 1 i can be transformed from x 2 j with the operations performed by edit distance, it is naturally to suppose that we can find such a  y 2 j which can be transformed from y 2 j with these operations similarly. Therefore, we can limit the search space for  y 2 j with two restrictions: Firstly, sentence  y 2 j has the same meaning with x 1 i ; Secondly,  y 2 j is transformed from y 2 j with the operations performed by edit distance. Considering the restrictions mentioned above, we apply an NMT model m to search and generate  y 2 j . There are two main questions left to be resolved: how to train such a model m and how to generate  y 2 j with a well-trained m . Training Motivated by the recent success of self-supervised training (Devlin et al., 2018; Conneau and Lample, 2019; Song et al., 2019; Yang et al., 2020) in natural language processing, we automatically construct the training corpus for m from the candidate aligned examples. Given the candidate aligned example ( x 1 i , y 1 i , x 2 j , y 2 j ) , the training ex-8143 ample for m is built as: ([ x 2 j ;  y 2 j ] , y 2 j ) (2) where y 2 j is the target sentence, the concatenation of x 2 j and  y 2 j is the source-side input. y 2 j is the noisy form of y 2 j which we build by mimicking the operations of edit distance, i.e, performing insertion, removal, or substitution on some pieces of y 2 j randomly. Specifically, with probability  , each position of sentence y 2 j can be noised by either removed directly, inserted or substituted with any other words in the dictionary W b , which is constructed from the corpus Y 2 . With the self-constructed training examples, the model m is trained to generate the target sentence, which is recovered from the right-side of the concatenated input with the operations performed by edit distance, and has the same meaning to the left-side of the input. Generating With a well-trained m , we generate the final aligned examples by running the inference step of m . Formally, for the final aligned example ( x 1 i , y 1 i ,  y 2 j ) , the sentence  y 2 j is calculated by:  y 2 j = m ([ x 1 i ; y 2 j ]) (3) where [  ;  ] represents the operation of concatenation, and m ( x ) refers to running the inference step of m with x fed as input. With this generation process,  y 2 j is not only has the same meaning to x 1 i (thus also aligned to y 1 i ), but also keeps the word composition and sentence structure similar to y 2 j . Therefore, EAG can construct the final aligned corpus for each non-English language pair, and keep the diversity of the constructed corpus almost identical to the original English-centric corpus. For a clear presentation, Algorithm 1 in Appendix A.2 summarizes the process of generating the final aligned examples. We also provide a toy example in Figure 1 to illustrate how the proposed EAG works. For fair comparison, we evaluate our methods on the publicly available dataset WMT-5, which is used by Freitag and Firat (2020). Additionally, we test the scalability of our method by further conducting experiments on Opus-100, which contains English-centric bilingual data from 100 language pairs (Zhang et al., 2020). In the extraction process, we run our extraction code on the CPU with 24 cores and 200G memory. 2 In the generation process, we take transformer-big (Vaswani et al., 2017) as the configuration for m , and m is trained with the self-constructed examples mentioned in Section 3.2 on eight V100 GPU cards. 3 We choose Transformer as the basic structure for our model and conduct experiments on two standard configurations, i.e, transformer-base and transformer-big. All models are implemented based on the open-source toolkit fairseq (Ott et al., 2019) and trained on the machine with eight V100 GPU cards. 4 All bilingual models are trained for 300,000 steps and multi-lingual models are trained for 500,000 steps. We add a language token at the beginning of the input sentence to specify the required target language for all of the multi-lingual models. For the hyper-parameters  and  , we set them as 0.5 and 0.3 by default and also investigate how their values produce effects on the translation performance. Following Freitag and Firat (2020), we take WMT13EnEs, WMT14EnDe, WMT15EnFr, WMT18EnCs and WMT18EnRu as the training data, the multi-way test set released by WMT2013 evaluation campaign (Bojar et al., 2014) as the test set. The size of each bilingual training corpus (the non-English corpus constructed by Freitag and Firat (2020) included) is presented in Table 1. For the bilingual translation task, the source and target languages are jointly tokenized into 32,000 subword units with BPE (Sennrich et al., 2016). The multi-lingual models use a vocabulary of 64,000 sub-word units tokenized from the combination of all the training corpus. Similar to Freitag and Firat (2020), we use a temperature-based data sampling strategy to over-sample low-resource language pairs in standard MNMT models and low-resource target-languages in C-MNMT models (temperature T = 5 for both cases). We use BLEU scores (Papineni et al., 2002) to measure the model performance and all BLEU scores are calculated with sacreBLEU (Post, 2018). 5 2 Intel(R) Xeon(R) Platinum 8255C CPU @ 2.50GHz 3 Detained training process for m can be found in the Appendix A.1. Table 2 shows the training data after constructing non-English examples from English-centric corpus by the proposed EAG. By comparing Table 2 with Table 1, we can find that EAG can construct much more multi-way aligned non-English training examples than Freitag and Firat (2020), e.g., EAG constructs 1.4M bilingual training corpus for the language pair German  Russian which is almost up to 4 times more than the corpus extracted by Freitag and Firat (2020). In all, EAG constructs no less than 1M bilingual training examples for each non-English language pair. In order to properly and thoughtfully evaluate the proposed method, we take the following five kinds of baseline systems for comparison: Bilingual systems (Vaswani et al., 2017) Apart from training bilingual baseline models on the original English-centric WMT data, we also train bilingual models for non-English language pairs on the direct bilingual examples extracted by Freitag and Firat (2020). Bridging (pivoting) systems (Cheng et al., 2016) In the bridging or pivoting system, the source sentence cascades through the pre-trained source  English and English  target systems to generate the target sentence. Extraction-based C-MNMT systems (Freitag and Firat, 2020) Freitag and Firat (2020) construct the multi-way aligned examples by directly extracting and pairing bilingual examples from different language pairs with identical English sentences. Generation-based C-MNMT systems The generation-based C-MNMT baselines construct non-English bilingual examples by distilling the knowledge of the system which cascades the source  English and English  target models. Different from the bridging baselines which just feed the test examples into the cascaded system and then measure the performance on the test examples, the generation-based C-MNMT baselines feed the non-English sides of the bilingual training examples into the cascaded systems and then get the non-English bilingual training examples by pairing the inputs and outputs. The combination of the generated non-English corpus and original English-centric corpus is used to train the C-MNMT model. We first report the results of our implementations and then present the comparisons with previous works. In our implementations, we take the transformer-base as the basic model structure since it takes less time and computing resources for training. To make a fair comparison with previous works, we conduct experiments on transformer-big which is used by baseline models. Results of our implementation Table 3 shows the results of our implemented systems. Apart from the average performance of the translation directions from each language to others, we also report the average performance on the English-centric and non-English language pairs. 6 As shown in Table 3, we can find that the proposed EAG achieves better performance than all of the baseline systems. Compared to the extraction-based C-MNMT, the proposed method achieves an improvement up to 1.1 BLEU points on non-English language pairs. 6 Readers can find the detailed results for each language pair in the Appendix A.3. The generation-based C-MNMT performs worse than the extraction-based one even if it generates much larger corpus. Since there is no any training example for non-English language pairs in standard MNMT, the standard MNMT system achieves inferior performance to the pivot and bilingual systems on non-English translation directions. However, with the constructed non-English training examples, EAG achieves 3.2 and 7.5 BLEU points improvements compared with the pivot and bilingual systems respectively. Results compared with previous works Table 4 shows the results of the proposed EAG. We can find that the proposed EAG surpasses Freitag and Firat (2020) almost on all of the translation directions, and achieves an improvement with up to 2.4 BLEU points on the Russian-to-German direction. sampling from the OPUS collection (Tiedemann, 2012). Opus-100 is an English-centric dataset which contains 100 languages on both sides and up to 1M training pairs for each language pair. To evaluate the performance of non-English language pairs, Zhang et al. (2020) sample 2000 sentence pairs of test data for each of the 15 pairings of Arabic, Chinese, Dutch, French, German, and Russian. Following Zhang et al. (2020), we report the sacreBLEU on the average of the 15 non-English language pairs. 7 The statistics about the non-English corpus constructed by Freitag and Firat (2020) and EAG are presented in Table 5. We can find that EAG is able to construct much more bilingual corpus for non-English language pairs (almost nine times more than Freitag and Firat (2020) for each language pair). We use a vocabulary of 64,000 sub-word units for all of the multi-lingual models, which is tokenized from the combination of all the training corpus with SentencePiece. Results Apart from the baselines mentioned above, we also compare with other two systems proposed by Zhang et al. (2020) and Fan et al. (2020). Zhang et al. (2020) propose the online 7 Signature: BLEU+case.mixed+numrefs.1+smooth.exp+ tok.13a+version.1.4.1 8146 System non-English MNMT system 4.5 pivot system 13.1 generation-based C-MNMT 13.8 extraction-based C-MNMT 16.5 Zhang et al. (2020) 14.1 Fan et al. (2020) 18.4 EAG 17.9  Table 6: Results on Opus-100. Fan et al. (2020) build a C-MNMT model, named m 2 m 100 , which is trained on 7.5B training examples built in house. Following Zhang et al. (2020), we take the transformer-base as the basic model structure for the experiments and results are reported in Table 6. We can find that EAG achieves comparable performance to Fan et al. (2020) which utilizes much more data than ours. This is not a fair comparison as the data used Fan et al. (2020) is 75 times as much as ours. Additionally, our model surpasses all other baseline systems and achieves +1.4 BLEU points improvement compared with the extraction-based C-MNMT model. We analyze the proposed method on Opus-100 and take the transformer-base as the model structure. The similarity threshold  and the noise ratio  are important hyper-parameters in EAG. In this section, we want to test how these two hyper-parameters affect the final translation performance and how they work with each other. We investigate this problem by studying the translation performance with different  and  , where we vary  and  from 0 to 0.7 with the interval 0.2. We report the average BLEU score for the translation directions from Arabic to other five languages on the development sets built in house. Figure 2 shows the experimental results. With  = 0 , it means that the generation process is not applied and we directly train the NMT model with the extracted roughly aligned examples. And this is the setting of our motivated experiments mentioned in Section 1. We can find that, the final performance drops sharply when we directly train the model with the roughly aligned sentence pairs. For each curve in Figure 2, we can find that the model achieves the best performance when the  is around  , and then the performance decreases with  growing. A relatively unexpected result is that the model usually achieves the best performance when  = 0 . 5 rather than when  = 0 . 7 (with a larger  , m is trained to handle more complex noise). We conjecture the main reason is that the noise in the training data when  = 0 . 7 is beyond the capacity of m , which makes m converge poorly during training. Overall, with  set as 0.5 and  set as 0.3, the model achieves the best performance. We test how the ability of m affects the final performance. Apart from the Transformer-base, i.e., the default setting used in EAG, we also test other two settings, namely Transformer-big (Vaswani et al., 2017) and Transformer-deep (20 encoder layers and 4 decoder layers). With different settings, m is expected to perform different abilities in the generation process. The experimental results are presented in Table 7. We can find that if we remove m , the final performance drops dramatically (compar-ing #0 with #1). This shows that the generation step 8147 Figure 3: Examples constructed by EAG. plays a significant role in the proposed EAG. However, by comparing among #0, #2 and #3, we can find that the ability for m shows little effect on the final performance. Taking all of #0, #1, #2 and #3 into consideration, we can reach a conclusion that, the generation step is very important for EAG and a simple generation model, i.e., a baseline NMT model, is enough to achieve strong performance. We are very curious about how EAG works with back-translation (BT). To investigate this problem, we utilize the extraction-based C-MNMT model to decode the non-English monolingual sentences in the candidate aligned examples extracted by EAG, and then get the synthetic non-English sentence pairs by pairing the decoded results with the original sentences. The reversed sentence pairs are appended into the training corpus for the MNMT models. The experimental results are presented in Table 8. We find that BT improves the performances of both the two systems. Additionally, BT can work as a complementary to the proposed EAG. Figure 3 presents some examples constructed by EAG, each of which includes the extracted candidate aligned example and the generated sentence for Arabic  Chinese. The extracted candidate aligned example contains two bilingual examples, which are extracted from Arabic  English and Chinese  English respectively. In Figure 3  the two bilingual examples in case one are extracted as a candidate aligned example as their English sentences have high similarity. And there is a composition gap between x 1 and x 2 since "Bobby Jordan" is mentioned in x 1 , but not in x 2 . By comparing the generated Chinese sentence  y 2 with the original sentence y 2 , we can find that  y 2 is modified from y 2 by inserting the Chinese words     ", which has the same meaning with "Bobby Jordan". Therefore, the generated  y 2 is aligned to x 1 and y 1 . In case 2, the Chinese word    (out)" in y 2 has been replaced with Chinese words "     (justice)" in  y 2 , which makes the  y 2 aligned to x 1 and y 1 . Case 3 in Figure 3 behaves similarly. While achieving promising performance, the proposed approach still has some weaknesses in the real application: 1) The two-step pipeline performed by EAG is somewhat time-consuming compared to Freitag and Firat (2020); 2) The generated multi-way aligned examples by EAG are sometimes not strictly aligned as the generation process does not always perform perfectly. In this paper, we propose a two-step approach, i.e., EAG, to construct large-scale and high-quality multi-way aligned corpus from English-centric bilingual data. To verify the effectiveness of the proposed method, we conduct extensive experiments on two publicly available corpora, WMT-5 and Opus-100. Experimental results show that the proposed method achieves substantial improvements over strong baselines consistently. There are three promising directions for the future work. Firstly, we plan to test whether EAG is applicable to the domain adaptation problem in NMT. Sec-8148 ondly, we are interested in applying EAG to other related tasks which need to align different example pairs. Finally, we want to investigate other model structures for the generation process. The authors would like to thank the anonymous reviewers of this paper, and the anonymous reviewers of the previous version for their valuable comments and suggestions to improve our work.
abstain abstain objective objective abstain abstain abstain abstain abstain abstain other abstain abstain method abstain abstain result abstain other objective objective abstain abstain abstain other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain method abstain abstain method method abstain abstain abstain method method method method method method method abstain method abstain method abstain abstain method other abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result result abstain result abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain objective objective abstain abstain abstain method objective other