Neural Machine Translation (NMT) systems are known to degrade when confronted with noisy data, especially when the system is trained only on clean data. In this paper, we show that augmenting training data with sentences containing artificially-introduced grammatical errors can make the system more robust to such errors. In combination with an automatic grammar error correction system, we can recover 1 . 0 BLEU out of 2 . 4 BLEU lost due to grammatical errors. We also present a set of Spanish translations of the JFLEG grammar error correction corpus, which allows for testing NMT robustness to real grammatical errors. Neural Machine Translation (NMT) is undeniably a success story: public benchmarks (Bojar et al., 2016) are dominated by neural systems, and neural approaches are the de facto option for industrial systems (Wu et al., 2016; Hassan Awadalla et al., 2018; Crego et al., 2016; Hieber et al., 2018). Even under low-resource conditions, neural models were recently shown to outperform traditional statistical approaches (Nguyen and Chiang, 2018). However, there are still several shortcomings of NMT that need to be addressed: a (non-exhaustive) list of six challenges is discussed by Koehn and Knowles (2017), including out-of-domain testing, rare word handling, the wide-beam problem, and the large amount of data needed for learning. An additional challenge is robustness to noise, both during training and at inference time. In this paper, we study the e  ect of a specific type of noise in NMT: grammatical errors. We primarily focus on errors that are made by non-native  Equal contribution. Work performed at the University of Notre Dame. source-language speakers (as opposed to dialectal language, SMS or Twitter language). Not only is this linguistically important, but we believe that it would potentially have great social impact. Our contributions are three-fold. First, we con-firm that NMT is vulnerable to source-side noise when trained on clean data, losing up to 3 . 6 BLEU on our test set. This is consistent with previous work, yet orthogonal to it, since we use more realistic noise for our experiments. Second, we explore training methods that can deal with noise, and show that including noisy synthetic data in the training data makes NMT more robust to handling similar types of errors in test data. Combining this simple method with an automatic grammar correction system, we find that we can recover 1 . 5 BLEU. Third, we release Spanish translations of the JFLEG corpus, 1 a standard benchmark for English Grammar Error Correction (GEC) systems. We also release all other data and code used in this paper. Our additional annotations on both the JFLEG corpus and the English WMT data will enable the evaluation of the robustness of NMT systems on realistic, natural noise: a robust system would ideally produce the same output when presented with either the original or the noisy source sentence. We hope that our datasets will become a benchmark for noise-robust NMT, because we believe that deployed systems should also be able to handle source-side noise. We focus on NMT from English to Spanish. We choose English to be our source-side language because there exist English corpora annotated with grammar corrections, which we can use as a 1 Freely available at https://bitbucket.com/ antonis/nmt-grammar-noise source of natural noise. Moreover, since English is probably the most commonly spoken non-native language (Lewis et al., 2009), our work could be directly applicable to several translation applications. Our choice of Spanish as a target language enables us to have access to existing parallel data and easily create new parallel corpora (see below,  2.3). For all experiments, we use the Europarl English-Spanish dataset (Koehn, 2005) as our training set. In the synthetic experiments of Section  2.2, we use the newstest2012 and new-stest2013 as dev and test sets, respectively. Furthermore, to test our translation methods on real grammatical errors, we introduce a new collection of Spanish translations of the JFLEG corpus (  2.3). To our knowledge, there are five publicly available corpora of non-native English that are annotated with corrections, which have been widely used for research in Grammar Error Correction (GEC). The NUS Corpus of Learner English (NUCLE) contains essays written by students at the National University of Singapore, corrected by two annotators using 27 error codes (Dahlmeier et al., 2013). It has become the main benchmark for GEC, as it was used in the CoNLL GEC Shared Tasks (Ng et al., 2013, 2014). Other corpora include the Cambridge Learner Corpus First Certificate in English FCE corpus (Yannakoudakis et al., 2011), which is only partially public, the L ang -8 corpus (Tajiri et al., 2012), which was harvested from online corrections, and the AESW 2016 Shared Task corpus, which contains corrections on texts from scientific journals. The last corpus is the JHU FLuency-Extended GUG corpus (JFLEG) (Napoles et al., 2017). This corpus covers a wider range of English proficiency levels on the source side, and its correction annotations include extended fluency edits rather than just minimal grammatical ones. That way, the corrected sentence is not just grammatical, but also guaranteed to be fluent. Ideally, we would train a translation model to translate grammatically noisy language by training it on parallel data with grammatically noisy language. Since, to our knowledge, no such data exist in the quantities that would be needed, an al-Error Type Confusion Set art { a, an, the, } prep { on, in, at, from, for, under, over, with, into, during, until, against, among, throughout,of, to, by, about, like, before, after, since, across, behind, but, out, up, down, o  , } nn { SG, PL } sva { 3SG, not 3SG, 2SG-Past, not 2SG-Past } Table 1: Confusion sets for each grammar error type. ternative is to add synthetic grammatical noise to clean data. An advantage of this approach is that controlled introduction of errors allows for fine-grained analysis. This is a two-step process, similar to the methods used in the GEC literature for creating synthetic data based on confusion matrices (Ro-zovskaya et al., 2014; Rozovskaya and Roth, 2010; Xie et al., 2016; Sperber et al., 2017). First, we mimic the distribution of errors found in real data, and then introduce errors by applying rule-based transformations on automatic parse trees. The first step involves collecting error statistics on real data. Conveniently, the NUCLE corpus has all corrections annotated with 27 error codes. We focus on five types of errors, with the last four being the most common in the NUCLE corpus:  drop : randomly deleting one character from the sentence. 2  art : article / determiner errors  prep : preposition errors  nn : noun number errors  sva : subject-verb agreement errors Using the annotated training set of the NUCLE corpus, we compute error distribution statistics, resulting in confusion matrices for the cases outlined in Table 1. For art and prep errors, we obtain probability distributions that an article, determiner, or preposition is deleted, substituted with another member of the confusion set, or inserted in the beginning of a noun phrase. For nn errors, 2 This error is not part of the NUCLE error list. we obtain the probability of a noun being replaced with its singular or plural form. For sva errors, the probability that a present tense verb is replaced with its third-person-singular (3SG) or not-3SG form. An additional sva error that we included is the confusion between the appropriate form for the verb to be' in the past tense (was' and were'). The second step involves applying the noise-inducing transformations using our collected statistics as a prior. We obtained parses for each sentence using the Berkeley parser (Petrov et al., 2006). The parse tree allows us to identify candidate error positions in each sentence (for example, the beginning of a noun phrase without a determiner, were one could be inserted). For each error type we introduced exactly one error per sentence, wherever possible, which we believe matches more realistic scenarios than previous work. It also allows for controlled analysis of the behaviour of the NMT system (see Section 4). For each error and each sentence, we first identify candidate positions (based on the error type and the parse tree) and sample one of them based on the specific error distribution statistics. Then, we sample and introduce a specific error using the corresponding probability distribution from the confusion matrix. (In the case of drop , nn , and sva errors, we only need to sample the position and only insert / substitute the corresponding error.) If no candidate positions are found (for example, a sentence doesn't have a verb that can be substituted to produce a sva error) then the sentence remains unchanged. Following the above procedure, we added errors in our training, dev, and test set (henceforth referred to as [ error ]). Basic statistics on our produced datasets can be found in Table 2, while example sentences are shown in Table 3. Furthermore, we created training and dev sets that mix clean and noisy data. The clean+ [ error ] training sets are the concatenation of each [ error ] with the clean data, e  ectively including a clean and a noisy version of each sentence pair. We also created a training and dev dataset with mixed error types, in our attempt to study the effect of including all noise types during training. The mix all dataset includes each training pair six times: once with the original (clean) sentence as the source, and once for every possible error. We experimented with a mixed dataset that included each training sentence once, with the number of noisy sentences being proportional to the real error distributions of the NUCLE dataset, but obtained results similar to the [ error ] datasets. The JFLEG corpus consists of a dev and test set (no training set), with 747 and 754 English sentences, respectively, collected from non-native English speakers. Each sentence is annotated with four di  erent corrections, resulting in four (fluent and grammatical) reference sentences. About 14% of the sentences do not include any type of error, with the source and references being equivalent. We created translations of the JFLEG corpus that allow us to evaluate how well NMT fares compared to a human translator, when presented with noisy input. We will refer to the augmented JFLEG corpus as JFLEGes . Two professional translators were tasked with producing translations for the dev and the test set, respectively. The translators were presented only with the original erroneous sentences; they did not have access to the correction annotations. They were asked to produce fluent, grammatical translations in European Spanish (to match the Error Type Example art In October , Tymoshenko was sentenced to seven years in prison for entering into what was reported to be a / *  disadvantageous gas deal with Russia. Its ratification would require  / *the 226 votes. It is a / *the good result, which nevertheless involves a certain risk. prep [. . . ] the motion to revoke an article based on / *in which the opposition leader , Yulia Tymoshenko , was sentenced. Its ratification would require  / *for 226 votes. nn Its ratification would require 226 votes / *vote . The verdict / *verdicts is not yet final ; the court will hear Tymoshenko 's appeal in December. sva As a rule, Islamists win / *wins in the country; the question is whether they are the moderate or the radical ones. This cultural signature accompanies / *accompany the development of Moleskine; Table 3: Example grammatical errors that were introduced in the En-Es WMT test set. Spanish used in the Europarl corpus). There exist cases where a translator might choose to preserve a source-side error when producing the translation, such as translation of literary works where it's possible that grammar or fluency errors are intentional; however, our translators were explicitly asked not to do that. The exact instructions were as follows: Please translate the following sentences. Note that some sentences will have grammatical errors or typos in English. Don't try to translate the sentences word for word (e.g. replicate the error in Spanish). Instead, try to translate it as if it was a grammatical sentence, and produce a fluent grammatical Spanish sentence that captures its meaning. In this section, we provide implementation details and the results of our NMT experiments. For convenience, we will refer to each model with the same name as the dataset it was trained on; e.g. the mix all model will refer to the model trained on the mix all dataset. All data are tokenized, truecased, and split into subwords using Byte Pair Encoding (BPE) with 32 , 000 operations (Sennrich et al., 2016). We filter the training set to only contain sentences up to 80 words. Our LSTM models are implemented using DyNet (Neubig et al., 2017), and our transformer models using PyTorch (Paszke et al., 2017). The transformer model uses 6 layers, 8 attention heads, the dimension for embeddings and positional feed-forward are 512 and 2048 respectively . The sublayer computation sequence follows the guidelines from Chen et al. (2018). Dropout probability is set to 0.2 (also in the source embeddings, following Sperber et al. (2017)). We use the learning rate schedule in Vaswani et al. (2017) with warm-up steps of 24000 but only decay the learning rate until it reaches 10  5 as inspired by Chen et al. (2018). For testing, we select the model with the best performance on the dev set corresponding to the test set. At inference time, we use a beam size of 4 with length normalization (Wu et al., 2016) with a weight of 0 . 6. We report the results obtained with the transformer model, as they were consistently better than the LSTM one. All the result tables for the LSTM models can be found in the Appendix. The performance of our systems on the synthetic WMT test sets, as measured by detokenized BLEU (Papineni et al., 2002), is summarized in Table 4. When the system is trained only on clean data (first row) and tested on noisy data, it unsurprisingly exhibits degraded performance. We observe significant drops in the range of 1 . 03 . 6 BLEU. The largest drop (more than 3 . 5 BLEU) is observed with nn errors in the source sentence. This is not unreasonable: nouns almost always carry content significant for translation. Especially when translating into Spanish, a noun number change can, and apparently does, also a  ect the rest of the sentence significantly, for example, by influencing the conjugation of a subsequent verb. The second-largest drop (more than 3 . 0 BLEU points) is observed in the case of drop errors. This is also to be expected; typos produce out-of-vocabulary (OOV) words, which in the case of BPE are usually segmented to a most likely rarer subword sequence than the original correct word. We find that a training regime that includes both clean and noisy sentences ([ clean+error ) results in better systems across the board. Importantly, these models manage to perform en par with the clean model on the clean test set. Since the original training set is part of the [ clean+error training sets, this behavior is expected. We conclude, thus, that including the full clean dataset during training is important for performance on clean data  one cannot just train on noisy data. The [ clean+error ] systems exhibit a notable pattern: their BLEU scores are generally similar to the clean system on all test sets, except for the test set that matches their training set errors (high-lighted in Table 4), where they generally obtain the best performance. The mix all model is our best system on all test sets (except drop ) and on average. Unlike the [ clean+error ] systems, it outperforms the clean model on all noisy test sets and not only on a specific one. On average, using the mix all training set leads to an improvement of 0 . 4 BLEU over the clean model and 0 . 1  0 . 7 BLEU over the [ clean+error ] models. Furthermore, the mix all model exchibits the smallest performance standard deviation of all models, averaging over all test sets. This is another indication that our system is more robust to multiple source-side variations. We further explore this intuition in Section 4. On the more realistic JFLEGes dev and test sets, we observe same trends but at a smaller scale, as shown in Table 5. Our mix all model generally achieves comparable results when presented with each of the four reference corrections of the test set ( cor X columns). However, when we use the noisy source sentence as input (N o corr column) our mix all model obtains 1 . 4 BLEU improvements over the clean model. The di  erence between the performance of the models when presented with clean and noisy input is another indicator for robustness. On the JFLEGes test set, the noisy source results in a  3 . 1 BLEU point drop for the clean model, while the drop for our mix all model is smaller, at  1 . 7 BLEU points. In addition, we experimented with using an automatic error-corrected source as input to our sys-JFLEGes Dev Training Manual correction No Auto cor 0 cor 1 cor 2 cor 3 avg. tem (column A uto corr of Table 5). We used the publicly available JFLEG outputs of the (al-most) state-of-the-art model of Junczys-Dowmunt and Grundkiewicz (2016) as inputs to our NMT system. 3 This experiment envisions a pipeline where the noisy source is first automatically corrected and then translated. As expected, this helps the clean model (by + 1 . 1 BLEU), but our mix all training helps even further (by another + 0 . 8 BLEU). Interestingly, the automatic GEC system only helps in the test set, while there are no improvements in the dev set. Naturally, since automatic GEC systems are imperfect, the performance of this pipeline still lags behind translating on clean data. We attempt an in-depth analysis of the impact of the di  erent source-side error types on the behavior of our NMT system, when trained on clean data and tested on the artificial noisy data that we created. A rt Errors Table 6 shows the di  erence of the BLEU scores obtained on the sentences, broken down by the type of article error that was introduced. The first observation is that in all cases the di  erence is negative, meaning that we get higher BLEU scores when testing on clean data. Encouragingly, there is practically no di  erence when we substitute a' with an' or an' with a'; the model 3 This model has been recently surpassed by other systems, e.g. (Junczys-Dowmunt et al., 2018), but their outputs are not available online. seems to have learned very similar representations for the two indefinite articles, and as a result such an error has no impact on the produced output. However, we observe larger performance drops when substituting indefinite articles with the definite one and vice versa; since the target language makes the same article distinction as the source language, any article source error is propagated to the produced translation. P rep Errors Due to the large number of prepositions, we cannot present a full analysis of preposition errors, but highlights are shown in Table 7. Deleting a correct preposition or inserting a wrong one leads to performance drops of 1 . 2 and 0 . 8 BLEU points for the clean model, but drops of 0 . 4 and 0 . 7 for the mix all model. N n and S va Errors We found no significant performance di  erence between the di  erent nn errors. Incorrectly pluralizing a noun has the same adverse e  ect as singularizing it, leading to performance reductions of over 4 . 0 and 3 . 5 BLEU points respectively. We observe a similar behavior with sva errors: each error type leads to roughly the same performance degradation. The e  ect of noise in NMT was recently studied by Khayrallah and Koehn (2018), who explored noisy situations during training due to web-crawled data. This type of noise includes misaligned, mistranslated, or untranslated sentences which, when used during training, significantly degrades the performance of NMT. Unlike our Correct Substituted article article a an the  all a  0  2 . work, they primarily focus on a setting where the training set is noisy but the test set is clean. In addition, Heigold et al. (2018) evaluated the robustness of word embeddings against word scrambling noise, and showed that performance in downstream tasks like POS-tagging and MT is especially hurt. Sakaguchi et al. (2017a) studied word scrambling and the Cmabrigde Uin-ervtisy (Cambridge University) e  ect , where humans are able to understand the meaning of sentences with scrambled words, performing word recognition (word level spelling correction) with a semi-character RNN system. Focusing only on character-level NMT models, Belinkov and Bisk (2018) showed that they exhibit degraded performance when presented with noisy test examples (both artificial and natural occurring noise). In line with our findings, they also showed that slightly better performance can be achieved by training on data artificially induced with the same kind of noise as the test set. Sperber et al. (2017) proposed a noise-introduction system reminiscent of WER, based on insertions, deletions, and substitutions. An NMT system tested on correct transcriptions achieves a BLEU score of 55 (4 references), but tested on the ASR transcriptions it only achieves a BLEU score of 35 . 7. By introducing similar noise in the training data, they were able to make the NMT system slightly more robust. Interestingly, they found that the optimal amount of noise on the training data is smaller than the amount of noise on the test data. The notion of linguistically plausible corruption is also explored by Li et al. (2017), who created adversarial examples with syntactic and semantic noise (reordering and word substitutions respec-Substitution model BLEU di  erence clean mix all in  with  6 . tively). When training with these noisy datasets, they obtained better performance on several text classification tasks. Furthermore, in accordance with our results, their best system is the one that combines di  erent types of noise. We present a summary of relevant previous work in Table 8. Synthetic errors refer to noise introduced according an artificially created distribution, and natural errors refer to actual errorful text produced by humans. As for semi-natural , it refers to either noise introduced according to a distribution learned from data (as in our work), or to errors that are learned from data but introduced according to an artificial distribution (as is part of the work of Belinkov and Bisk (2018)). We consider our work to be complementary to the works of Heigold et al. (2018); Belinkov and Bisk (2018), and Sperber et al. (2017). However, there are several important di  erences: 1. Belinkov and Bisk (2018) and Sperber et al. (2017) train their NMT systems on fairly small datasets: 235K (Fr-En), 210K (De-En), 122K (Cz-En), and 138K sentences (Es-En) respectively. Even though they use systems like Nematus (Sennrich et al., 2017) or XNMT (Neubig et al., 2018) which generally achieve nearly SOTA results, it is unclear whether their results generalize to larger training data. In contrast, we train our system on almost 2M sentences. 2. All three systems introduce somewhat unrealistic amounts of noise in the data. The natural noise of Belinkov and Bisk (2018) consists of word substitutions based on Wikipedia errors or corrected essays (in the Work Errors Noise Types NMT level Languages (Heigold et al., 2018) synthetic character swaps, character flips, word scrambling char, BPE De  En (Sperber et al., 2017) synthetic ASR errors word Es  En (Belinkov and Bisk, 2018) synthetic character swap, middle scramble, full scramble, keyboard typo char, BPE Fr,De,Cz  En semi-natural word substitutions this work semi-natural grammar errors: article, preposition, noun number, verb agreement BPE En  Es natural JFLEG corpus Table 8: Previous work on evaluating the e  ect of noise in NMT systems. Czech case) but they substitute all possible correct words with their erroneous version, ending up with datasets with more than 40% of the tokens being noisy. For that reason, we refer to it as semi-natural noise in Table 8. Meanwhile, Sperber et al. (2017) test on the outputs of an ASR system that has a WER of 41 . 3%. For comparison, in the JFLEG datasets, we calculated that only about 3 . 5%5% of the tokens are noisy  the average Levenshtein distance of a corrected reference and its noisy source is 13 characters. 3. The word scrambling noise, albeit interesting, could not be claimed to be applicable to realistic scenarios, especially when applied to all words in a sentence. The solution Belinkov and Bisk (2018) suggested and Sperber et al. (2017) discussed is a characteror spelling-aware model for producing wordor subword-level embeddings. We suspect that such a solution would indeed be appropriate for dealing with typos and other character-level noise, but not for more general grammatical noise. Our method could potentially be combined with GloVe (Pennington et al., 2014) or fastText (Bojanowski et al., 2017) embeddings that can deal with slight spelling variations, but we leave this for future work. On the other side, Grammar Error Correction has been extensively studied, with significant incremental advances made recently by treating GEC as an MT task: among others, Junczys-Dowmunt and Grundkiewicz (2016) used phrased-based MT, Ji et al. (2017) used hybrid character-word neural sequence-to-sequence systems, Sakaguchi et al. (2017b) used reinforcement learning, and Junczys-Dowmunt et al. (2018) combined several techniques with NMT to achieve the current state-of-the-art. Synthetic errors for training GEC systems have also been studied and applied with mixed success (Rozovskaya and Roth, 2010; Rozovskaya et al., 2014; Xie et al., 2016), while more recently Xie et al. (2018) used backtranslation techniques to add synthetic noise for GEC. In this work, we studied the e  ect of grammatical errors in NMT. We not only confirmed previous findings, but also expanded on them, showing that realistic human-like noise in the form of specific grammatical errors also leads to degraded performance. We added synthetic errors on the English WMT training, dev, and test data (including dev and test sets for all WMT 18 evaluation pairs), and have released them along with the scripts necessary for reproducing them. We also produced Spanish translations of the JFLEG corpus, so that future NMT systems can be properly evaluated on real noisy data. This material is based upon work generously supported by the National Science Foundation under grants 1464553 and 1761548. We are grateful to the anonymous reviewers for their useful comments.
abstain objective abstain abstain abstain method abstain abstain abstain abstain method objective abstain abstain method objective objective abstain method objective result abstain abstain method method method method other abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain method other abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain method method abstain abstain method method abstain method abstain method abstain method abstain abstain abstain abstain abstain method abstain abstain abstain other method abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other abstain other other other other abstain other other other other other other other other abstain method other abstain method other other other objective other method other abstain other other abstain other other other other abstain method other other method result method method other other