In this paper, we show that neural machine translation (NMT) systems trained on large back-translated data overfit some of the characteristics of machine-translated texts. Such NMT systems better translate human-produced translations, i.e., translationese, but may largely worsen the translation quality of original texts. Our analysis reveals that adding a simple tag to back-translations prevents this quality degradation and improves on average the overall translation quality by helping the NMT system to distinguish back-translated data from original parallel data during training. We also show that, in contrast to high-resource configurations, NMT systems trained in low-resource settings are much less vulnerable to overfit back-translations. We conclude that the back-translations in the training data should always be tagged especially when the origin of the text to be translated is unknown. During training, neural machine translation (NMT) can leverage a large amount of monolingual data in the target language. Among existing ways of exploiting monolingual data in NMT, the so-called back-translation of monolingual data (Sennrich et al., 2016a) is undoubtedly the most prevalent one, as it remains widely used in state-of-the-art NMT systems (Barrault et al., 2019). NMT systems trained on back-translated data can generate more fluent translations (Sennrich et al., 2016a) thanks to the use of much larger data in the target language to better train the decoder, especially for low-resource conditions where only a small quantity of parallel training data is available. However, the impact of the noisiness of the synthetic source sentences generated by NMT largely remains unclear and understudied. Edunov et al. (2018) even showed that introducing synthetic noise in back-translations actually improves translation quality and enables the use of a much larger quantity of back-translated data for further improvements in translation quality. More recently, Caswell et al. (2019) empirically demonstrated that adding a unique token at the beginning of each back-translation acts as a tag that helps the system during training to differentiate back-translated data from the original parallel training data and is as effective as introducing synthetic noise for improving translation quality. It is also much simpler since it requires only one editing operation, adding the tag, and non-parametric. However, it is not fully understood why adding a tag has such a significant impact and to what extent it helps to distinguish back-translated data from the original parallel data. In this paper, we report on the impact of tagging back-translations in NMT, focusing on the following research questions (see Section 2 for our motivation). Q1. Do NMT systems trained on large back-translated data capture some of the characteristics of human-produced translations, i.e., translationese ? Q2. Does a tag for back-translations really help differentiate translationese from original texts? Q3. Are NMT systems trained on back-translation for low-resource conditions as sensitive to translationese as in high-resource conditions? During the training with back-translated data (Sen-nrich et al., 2016a), we can expect the NMT system to learn the characteristics of back-translations, i.e., translations generated by NMT, and such characteristics will be consequently exhibited at test time. However, translating translations is a rather artifi-cial task, whereas users usually want to perform translation of original texts. Nonetheless, many of the test sets used by the research community for evaluating MT systems actually contain a large portion of texts that are translations produced by humans, i.e., translationese . Translationese texts are known to be much simpler, with a lower mean sentence length and more standardized than original texts (Laviosa-Braithwaite, 1998). These characteristics overlap with those of translations generated by NMT systems that have been shown simpler, shorter, and to exhibit a less diverse vocabulary than original texts (Burlot and Yvon, 2018). These similarities raise Q1 . Caswell et al. (2019) hypothesized that tagging back-translations helps the NMT system during training to make some distinction between the back-translated data and the original parallel data. Even though the effectiveness of a tag has been empirically demonstrated, the nature of this distinction remains unclear. Thus, we pose Q2 . The initial motivation for back-translation is to improve NMT for low-resource language pairs by augmenting the training data. Therefore, we verify whether our answers to Q1 and Q2 for high-resource conditions are also valid in low-resource conditions, answering Q3 . As parallel data for training our NMT systems, we used all the parallel data provided for the shared translation tasks of WMT19 1 for English German (en-de), excluding the Paracrawl corpus, and WMT15 2 for EnglishFrench (en-fr). 3 As monolingual data for each of English, German, and French to be used for back-translation, we concatenated all the News Crawl corpora provided by WMT, and randomly extracted 25M sentences. For our simulation of low-resource conditions, we randomly sub-sampled 200k sentence pairs from the parallel data to train NMT systems and used these systems to back-translate 1M sentences randomly sub-sampled from the monolingual data. For validation, i.e., selecting the best model after training, we chose newstest2016 for en-de and newstest2013 for en-fr, since they are rather balanced on their source side between translationese and original texts. For 1 http://www.statmt.org/wmt19/ translation-task.html 2 http://www.statmt.org/wmt15/ translation-task.html 3 After pre-processing and cleaning, we obtained 5.2M and 32.8M sentence pairs for en-de and en-fr, respectively. evaluation, since most of the WMT test sets are made of both original and translationese texts, we used all the newstest sets, from WMT10 to WMT19 for en-de, and from WMT08 to WMT15 for en-fr. 4 All our data were pre-processed in the same way: we performed tokenization and truecasing with Moses (Koehn et al., 2007). For NMT, we used the Transformer (Vaswani et al., 2017) implemented in Marian (Junczys-Dowmunt et al., 2018) with standard hyper-parameters for training a Transformer base model. 5 To compress the vocabulary, we learned 32k byte-pair encoding (BPE) operations (Sennrich et al., 2016b) for each side of the parallel training data. The back-translations were generated through decoding with Marian the sampled monolingual sentences using beam search with a beam size of 12 and a length normalization of 1.0. The back-translated data were then concatenated to the original parallel data and a new NMT model was trained from scratch using the same hyper-parameters used to train the model that generated the back-translations. We evaluated all systems with BLEU (Papineni et al., 2002) computed by sacreBLEU (Post, 2018). To evaluate only on the part of the test set that have original text or translationese on the source side, we used the --origlang option of sacreBLEU with the value non-L1 for translationese texts and L1 for original texts, where L1 is the source language, and report on their respective BLEU scores. 6 3.3 Results in Resource-Rich Conditions Our results with back-translations (BT) and tagged back-translations (T-BT) are presented in Table 1. When using BT, we consistently observed a drop of BLEU scores for original texts for all the translations tasks, with the largest drop of 12.1 BLEU points (en  fr, 2014). Conversely, BLEU scores for translationese texts were improved for most tasks, with the largest gain of 10.4 BLEU points 4 For WMT14, we used the full version instead of the default filtered version in sacreBLEU that does not contain information on the origin of the source sentences. 5 The full list of hyper-parameters is provided in the supplementary material (Appendix A). 6 sacreBLEU signatures where L1 and L2 respectively indicates a two-letter identifier for the source and target languages of either de-en, en-de, fr-en, or en-fr, and XXX the name of the test set: BLEU+case.mixed+lang.L1L2+numrefs.1+ { origlang.L1,origlang.non-L2 } +smooth.exp+test.XXX+tok.13a+version.1.4.2 System test set de  en en  de all o n-o all o n-o BT 2010 28.9 (+0.5) 33.2 (-0.9) 27.9 (+0.7) 21.8 (-2.3) 24.6 (-5.7) 21.0 (-1.2) 2011 25.3 (-0.3) 29.9 (-1.0) 24.2 (-0.2) 19.9 (-1.4) 23.8 (-1.9) 19.0 (-1.1) 2012 27.1 (+0.3) 27.9 (-1.6) 27.0 (+0.7) 20.4 (-1.2) 24.5 (-4.6) 19.3 (-0.2) 2013 30.3 (+0.3) 34.7 (-1.6) 29.2 (+0.6) 23.8 (-1.9) 25.1 (-2.8) 23.6 (-1.7) 2014 32.8 (+2.2) 27.4 (-2.5) 36.8 (+7.0) 25.4 (-0.5) 23.2 (-3.3) 27.9 (+2.7) 2015 33.8 (+2.4) 22.5 (-1.9) 39.5 (+5.5) 27.2 (-1.1) 28.1 (-2.9) 24.7 (+1.9) 2017 35.5 (+3.0) 27.2 (-1.1) 42.8 (+7.4) 26.4 (-0.1) 26.3 (-3.6) 25.5 (+3.3) 2018 43.9 (+4.6) 32.0 (-1.0) 53.8 (+10.4) 38.0 (-1.4) 38.9 (-5.9) 35.0 (+3.8) 2019 -33.1 (-1.5) -31.4 (-4.8) -T-BT 2010 29.5 (+1.1) 34.4 (+0.3) 28.4 (+1.2) 25.0 (+0.9) 30.5 (+0.2) 23.4 (+1.2) 2011 26.4 (+0.8) 31.7 (+0.8) 25.2 (+0.8) 22.1 (+0.8) 25.8 (+0.1) 21.0 (+0.9) 2012 28.1 (+1.3) 30.2 (+0.7) 27.7 (+1.4) 22.8 (+1.2) 30.0 (+0.9) 20.9 (+1.4) 2013 30.8 (+0.8) 36.0 (-0.3) 29.6 (+1.0) 26.4 (+0.7) 28.1 (+0.2) 26.1 (+0.8) 2014 32.4 (+1.8) 29.6 (-0.3) 33.8 (+4.0) 27.9 (+2.0) 26.7 (+0.2) 29.4 (+4.2) 2015 33.9 (+2.5) 24.9 (+0.5) 37.7 (+3.7) 29.9 (+1.6) 32.1 (+1.1) 25.6 (+2.8) 2017 35.5 (+3.0) 28.1 (-0.2) 41.2 (+5.8) 28.7 (+2.2) 30.7 (+0.8) 26.0 (+3.8) 2018 43.2 (+3.9) 33.0 (+0.0) 50.4 (+7.0) 41.8 (+2.4) 45.6 (+0.8) 35.5 (+4.3) 2019 -35.0 (+0.4) -37.6 (+1.4) System test set fr  en en  fr all o n-o all o n-o BT 2008 22.9 (-1.7) 27.9 (-2.6) 22.2 (-1.5) 23.2 (-0.2) 21.2 (-3.3) 23.6 (+0.5) 2009 26.5 (-2.3) 41.1 (-5.3) 23.9 (-1.6) 27.7 (+1.1) 22.7 (-2.0) 28.4 (+1.4) 2010 29.3 (-1.4) 27.4 (-7.8) 29.5 (+0.5) 28.2 (-0.5) 22.5 (-11.1) 29.8 (+2.5) 2011 29.4 (-1.9) 29.3 (-4.7) 29.4 (-1.1) 30.9 (+0.0) 36.7 (-8.2) 29.3 (+2.1) 2012 29.7 (-1.4) 34.3 (-4.3) 28.6 (-0.6) 28.4 (+1.1) 26.3 (-4.1) 29.0 (+2.5) 2014 36.6 (+0.6) 31.4 (-4.7) 40.3 (+5.6) 32.9 (-3.1) 26.1 (-12.1) 39.6 (+6.1) 2015 36.2 (+0.0) 40.9 (-3.1) 29.8 (+3.5) 35.7 (+1.7) 25.1 (-4.4) 44.9 (+6.5) T-BT 2008 24.5 (-0.1) 29.5 (-1.0) 23.7 (+0.0) 23.8 (+0.4) 25.1 (+0.6) 23.5 (+0.4) 2009 28.9 (+0.1) 46.4 (+0.0) 25.7 (+0.2) 27.3 (+0.7) 25.1 (+0.4) 27.7 (+0.7) 2010 31.2 (+0.5) 35.1 (-0.1) 29.6 (+0.6) 30.0 (+1.3) 34.1 (+0.5) 28.9 (+1.6) 2011 31.8 (+0.5) 33.3 (-0.7) 31.4 (+0.9) 31.6 (+0.7) 45.3 (+0.4) 28.0 (+0.8) 2012 31.8 (+0.7) 38.3 (-0.3) 30.1 (+0.9) 28.9 (+1.6) 31.9 (+1.5) 28.1 (+1.6) 2014 37.3 (+1.3) 36.1 (+0.0) 37.2 (+2.5) 38.2 (+2.2) 39.7 (+1.5) 36.5 (+3.0) 2015 36.6 (+0.4) 43.2 (-0.8) 27.9 (+1.6) 36.0 (+2.0) 30.7 (+1.2) 41.2 (+2.8) Table 1: BLEU scores for NMT systems trained with back-translations (BT) and tagged back-translations (T-BT) for each origin of the source text: original (o) or translationese (n-o). (de  en, 2018). These results give an answer to Q1 : NMT overfits back-translations, potentially due to their much larger size than the original parallel data used for training. Interestingly, using back-translations does not consistently improve translation quality. We assume that newstest sets may manifest some different characteristics of translationese from one year to another. Prepending a tag (T-BT) had a strong impact on the translation quality for original texts, recovering or even surpassing the quality obtained by the NMT system without back-translated data, always beating BT. The large improvements of BLEU scores over BT show that a tag helps in identifying translationese (answer for Q2 ). In the supplementary material (Appendix B), we present additional results obtained using more back-translations (up to 150M sentences) showing a similar impact of tags. However, while a tag in such a configuration prevents an even larger drop of the BLEU scores, it is not sufficient to attain a BLEU score similar to the configurations that use less back-translations. Interestingly, the best NMT system was not always the same depending on the translation direction and the origin of the test sets. It is thus possible to select either of the models to obtain the best translation quality given the origin of the source sentences, according to the results on the validation set for instance. 7 7 Since this observation is rather secondary, we present results for best model selection in the supplementary material (Appendix C). Note also that these BLEU scores can potentially be further increased by using a validation set whose source side is either original texts or translationese respectively to translate original texts or translationese at test time. In low-resource conditions, as reported in Table 2, the translation quality can be notably improved by adding back-translations. Using BT, we observed improvements of BLEU scores ranging from 0.7 (fr  en, 2011) to 12.4 (de  en, 2010) BLEU points for original texts and from 2.1 (en  de, 2011) to 21.1 (de  en, 2018) BLEU points for translationese texts. These results remain in line with one of the initial motivations for using back-translation: improving translation quality in low-resource conditions. In this setting without back-translated data, the data in the target language is too small for the NMT system to learn reasonably good representations for the target language. Adding 5 times more data in the target language, through back-translation, clearly helps the systems without any negative impact of the noisiness of the back-translations that were generated by the initial system. We assume here that since the quality of the back-translations is very low, their characteristics are quite different from the ones of translationese texts. This is confirmed by our observation that adding the tag has only a negligible impact on the BLEU scores for all the tasks (answer to Q3 ). A tag on back-translations helps identifying translationese during NMT training. Thus, adding the same tag on the test sets should have a very different impact depending on the origin of the source sentences. If we tag original sentences and decode them with a T-BT model, then we enforce the decoding of translationese. Since we mislead the decoder, translation quality should drop. On the other hand, by tagging translationese sentences, we help the decoder that can now rely on the tag to be very confident that the text to decode is translationese. Our results presented in Table 3 confirm these System de  en en  de fr  en en  fr 2017 2018 2017 2018 2012 2015 2012 2015 tagged original -2.0 -2.6 -5.9 -9.6 -7.5 -4.9 -10.1 -11.1 tagged non-original +1.6 +3.4 +0.8 +1.6 -3.1 +1.4 -0.3 +3.6 Table 3: Results with tagged test sets, either original or non-original, decoded with the T-BT model in the high-resource condition. assumptions. We observed a drop of BLEU scores when decoding tagged original texts with the T-BT model, while we saw an improvement of translation quality for 6 out of 8 test sets when decoding tagged translationese texts. The remaining 2 test sets for which we did not observed any improvements are newstest2012 for both translation directions of en-fr. It potentially indicates a mismatch between the characteristics of translationese in newstest2012 and those exhibited by back-translations used for training the T-BT model. We empirically demonstrated that training NMT on back-translated data overfits some of its characteristics that are partly similar to those of translationese. Using back-translation improves translation quality for translationese texts but worsens it for original texts. Previous work (Graham et al., 2019; Zhang and Toral, 2019) showed that state-of-the-art NMT systems are better in translating translationese than original texts. Our results show that this is partly due to the use of back-translations which is also confirmed by concurrent and indepen-dent work (Bogoychev and Sennrich, 2019; Edunov et al., 2019). Adding a tag to back-translations prevents a large drop of translation quality on original texts while improvements of translation quality for translationese texts remain and may be further boosted by tagging test sentences at decoding time. Moreover, in low-resource conditions, we show that the overall tendency is significantly different from the high-resource conditions: back-translation improves translation quality for both translationese and original texts while adding a tag to back-translations has only a little impact. We conclude from this study that training NMT on back-translated data, in high-resource conditions, remains reasonable when the user knows in advance that the system will be used to translate translationese texts. If the user does not know it a priori, a tag should be added to back-translations during training to prevent a possible large drop of translation quality. For future work, following the work on automatic identification of translationese (Rabinovich and Wintner, 2015; Rubino et al., 2016), we plan to investigate the impact of tagging translationese texts inside parallel training data, such as parallel sentences collected from the Web. We would like to thank the reviewers for their useful comments and suggestions. A part of this work was conducted under the program Re-search and Development of Enhanced Multilingual and Multipurpose Speech Translation System of the Ministry of Internal Affairs and Communications (MIC), Japan. Benjamin Marie was partly supported by JSPS KAKENHI Grant Number 20K19879 and the tenure-track researcher start-up fund in NICT. Atsushi Fujita was partly supported by JSPS KAKENHI Grant Number 19H05660.
result abstain result objective abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective objective objective objective objective objective method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain result result result abstain other abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result result abstain abstain abstain abstain result abstain result method abstain abstain other other other other