Regularization methods applying input perturbation have drawn considerable attention and have been frequently explored for NMT tasks in recent years. Despite their simplicity and effectiveness, we argue that these methods are limited by the under-fitting of training data. In this paper, we utilize prediction difference for ground-truth tokens to analyze the fitting of token-level samples and find that under-fitting is almost as common as over-fitting. We introduce prediction difference regularization (PD-R), a simple and effective method that can reduce over-fitting and under-fitting at the same time. For all token-level samples, PDR minimizes the prediction difference between the original pass and the input-perturbed pass, making the model less sensitive to small input changes, thus more robust to both perturbations and under-fitted training data. Experiments on three widely used WMT translation tasks show that our approach can significantly improve over existing perturbation regularization methods. On WMT16 En-De task, our model achieves 1.80 SacreBLEU improvement over vanilla transformer. Neural machine translation models have achieved great success in recent years (Sutskever et al., 2014; Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017). Despite their efficiency and superb performance, NMT models are prone to over-fitting that universal regularization techniques such as dropout (Hinton et al., 2012) and label smoothing (Szegedy et al., 2016) have been indispensable. However, over-fitting is still a significant problem for NMT, especially for small and medium tasks, which motivates researchers to constantly explore more specialized and sophisticated regularization techniques. Particularly, regularization methods applying input perturbation have been frequently explored for NMT models in recent years (Bengio et al., 2015; Wu et al., 2019; Sato et al., 2019; Takase and Kiyono, 2021). In these methods, neural models are trained to maximize the likelihood of perturbed samples that perturbed by a certain type of perturbations, with a primary intention to enhance model's robustness to perturbations, since neural models have been discovered fragile to small input noises (Szegedy et al., 2014; Liang et al., 2018; Belinkov and Bisk, 2018). In the past few years, many types of perturbations have been proposed to machine translation and been shown effective, including word-dropout (Gal and Ghahramani, 2016), word-replacement (Bengio et al., 2015; Wu et al., 2019) and adversarial perturbation (Miyato et al., 2017; Sato et al., 2019), etc. In this paper, unlike previous works which are devoted to finding stronger perturbations and more appropriate perturbation schedules, we rethink the existing perturb-and-fit mechanism and prove that indiscriminate fitting of perturbed samples ignores and aggravates under-fitting, which dramatically limits the effectiveness of perturbation regularization. We further propose prediction difference regularization (PD-R), a simple and effective method that can alleviate over-fitting and under-fitting at the same time and significantly enhance the effectiveness of perturbation regularization. Specifically, we use the prediction difference for ground-truth labels before and after input perturbation as an indicator of over-fitting and under-fitting for token-level samples. Quantitative analysis shows that a considerable part of token-level predictions get improved after input perturbation, indicating that the model is less fitted to those original samples compared to the perturbed samples, which has been ignored by previous works. We then divide labels in a batch into relatively under-fitted and over-fitted subsets according to real-time 7665 prediction difference and train only one subset to fit the perturbed inputs and the other subset to fit the original inputs. Experiments show that training only the relatively under-fitted subset to further fit the perturbed inputs dramatically degrade the model performance, while the opposite gets better results than the existing indiscriminate way. This indicates that existing methods are hindered by the excessive fitting of perturbed data. We further propose to use prediction difference as a regularization term, where the prediction difference is the divergence of prediction distribution caused by input perturbation. Since the value of prediction difference reflects the severity of over-fitting or under-fitting, both of which are cases we want to avoid for training models, regularizing prediction difference has been a natural solution to avoid above fitting problems. By combining cross-entropy loss and the prediction difference term, a model can be trained to fit training data with control of over-fitting and under-fitting. We apply PD-R on simplest word dropout regularization and conduct experiments on three widely used WMT translation tasks covering small-scale, medium-scale, and large-scale data sets. Our method significantly improves over existing perturbation regularization methods. On WMT16 En-De translation task, our method achieves 1.80 SacreBLEU improvement over vanilla transformer model and 1.12 SacreBLEU over traditional word dropout regularization. In this section, we introduce basic principles of neural machine translation, representative types of perturbations as well as their training objectives. For NMT, the probability of a target sentence Y = y 1: J conditioned on its parallel source sentence X = x 1: I is established based on chain rule: where  represents the parameters of the model, y 0 and y J +1 are special tokens representing the beginning and end of a sentence respectively. On this basis, NMT models are trained with the cross-entropy loss to minimize the negative log-likelihood of all samples in the training set D : L = L ( D ,  ) =  1 D (cid:88) ( X,Y ) D (cid:96) ( X, Y,  ) , (2) where D = { ( X n , Y n ) } | V | n =1 , | V | is the size of the data set, (cid:96) ( X, Y,  ) = log p ( Y | X,  ) . Word Dropout and Replacement The simplest way to apply perturbation is to mask or replace one or more tokens of the original input sequence. The resulting sequence  x is sampled from the original sequence and the perturbation sequence:  x i = (cid:40) x i , with probability 1  , x pi , with probability  , (3) where 0 <  < 1 is the hyper-parameter of bernoulli sampling and x pi is the i-th word of the perturbation sequence. Note that the perturbation sequence x p consists of zero vectors for word dropout (Gal and Ghahramani, 2016), and consists of random words sampled from the vocabulary with uniform or a particular distribution for word replacement (Bengio et al., 2015; Wu et al., 2019). Adversarial Perturbation Adversarial Training (AdvT) tries to make perturbation that maximize the loss function, which is believed more effective for regularization. As described in Miyato et al. (2017) and Sato et al. (2019), the perturbed input embedding for x i can be computed as follows:  e i = e i +   r i , (4) where e i is original embedding of i-th source word,  is a scalar hyper-parameter that controls the norm of the perturbation, and  r i is the worst case unit perturbation vector approximated by gradient back-propagation(Goodfellow et al., 2015):  r i = g i || g i || 2 , g i =  e i L ( D ,  ) , (5) where g i is the gradient of a model's loss function with respect to its input embedding e i . In most cases, the inputs of the decoder side can also be perturbed in the same way as the encoder side. For scheduled sampling (Bengio et al., 2015) however, perturbation is limited at the decoder side. Regularization For word dropout and word replacement, the model is trained to fit the perturbed samples  X : where  D is the perturbed data set. For adversarial training, two forward passes and two backward passes are required for computing perturbation vectors and then training with them. The model is trained to fit both the original samples and adversarial samples with loss function: L = L ( D ,  ) + L (  D ,  ) , (7) where  is a hyper-parameter. It is usually believed that a model's prediction for ground-truth target tokens will be hindered when the input is perturbed, which has been the initial motivation for perturbation regularization. However, this conclusion is not necessarily true in experiment for many reasons: Firstly, neural networks are complex and may not behave in an ideally logical way. Secondly, perturbations are randomly produced and may have complex properties. For example, word-replacement may induce synonyms or heteronyms, and high-dimension embedding perturbation is hard to interpret. Thirdly, the model is uncertain during training due to parameter dropout, which further brings uncertainty to the model's reaction to perturbations. In this section, we analyze the influence of perturbations leveraging token-level prediction difference. Here the prediction difference is defined as the change of model's prediction probabilities for ground-truth target tokens (Gu and Tresp, 2019; Li et al., 2019a):  p ( y j ) = p ( y j | y 0: j  1 , X,  )  p ( y j |  y 0: j  1 ,  X,  ) . We apply random perturbations to samples in the test set and divide all target labels into two subsets according to their prediction change: positively influenced subset S p containing labels whose prediction probabilities get bigger after input perturbation and negatively influenced subset S n containing labels whose prediction probabilities get smaller. We compute the quantitative proportion and the average value of  p for these two subsets to evaluate the influence of different perturbations. Since the prediction difference is also under the influence of parameter dropout during training, we also conduct experiments both with and without parameter dropout difference. The original pass and the perturbed pass are carried out by two different sub-models if their parameter dropout mask is different. In experiment, We use a transformer base model trained on WMT16 En-De data set and conduct our experiments on a test set which is a combination of 5 test sets from WMT16 to WMT20. Our analysis covers different kinds of perturbations, including word-dropout, word-replacement, and adversarial perturbation. The word-dropout and word-replacement probabilities are set as 0.05, and all perturbations are applied on both sides of the model. As illustrated in figure 1, for any certain type of perturbation, the negative impact is principal, especially for adversarial perturbation. However, the positive influence is non-negligible since the proportion of positively influenced tokens could reach 30%-40% for word-dropout and word-replacement. With parameter dropout difference, the positive influence could get further bigger and become more crucial. We attribute prediction difference to relatively over-fitting and under-fitting of token-level samples. Since perturbations are very small, perturbed samples can be approximately viewed as good samples. For one target label, if its prediction probability gets smaller after a small input perturbation, it indicates the model is relatively over-fitted to the original sample, while the contrary case is the reflection of relatively under-fitting. With parameter dropout, predictions are carried out by sub-models, and these fitting problems also reflect the relative fitting bias of sub-models, which is also what we want to avoid. As mentioned above, existing perturbation regularization methods are based on the motivation to enhance the model's performance against input perturbation and avoid over-fitting. However, experiments show that a model could be better fitted to the perturbed data rather than the original data, which is regarded by us as a sign of relatively under-fitting. This indicates that training a model 7667 We further carry out selective training for word-dropout regularization, where one subset is trained to fit the perturbed inputs and the other subset is trained to fit the original inputs. As presented in table 1, training only S n gets better results than existing indiscriminate training, while training only S p gets worse results than vanilla transformer. This implies that the existing method suffers from degeneration caused by aggravated under-fitting. Since both positive and negative prediction difference is a sign of improper fitting of samples, we therefore propose prediction difference regularization (PD-R), to regularize the model directly with the prediction difference: where R [  ] is the distance of two distributions, ( X, Y ) is a sample from data set D ,    represents all prediction steps, P ( | X, Y < ,  (cid:48) ) is the prediction distributions for all steps conditioned on original source input X , target teacher forcing target input Y < and sub-model with parameters  (cid:48) , and P ( |  X,  Y < ,  (cid:48)(cid:48) ) is the prediction distributions for all steps conditioned on perturbed source input  X , perturbed target teacher forcing target input  Y < and sub-model with parameters  (cid:48)(cid:48) . The total regularization loss is averaged over all samples in the data set: LPD  R ( D ,  ) = 1 D (cid:88) ( X,Y ) D (cid:96) PD  R ( X, Y,  ) . In experiment, we apply PD-R on simplest word-dropout perturbation with  = 0 . 05 in Eq.(3) and  = 1 . 0 in Eq.(11) without further hyper-parameter search. R [  ] in Eq.(9) is implemented as L1 distance, which performs slightly better than KL-divergence in our experiments. We evaluate PD-R on three public WMT machine translation tasks and compare it with representative related works. To fully verify the effectiveness of our method on NMT, we conduct experiments on three machine translation tasks, including small-scale WMT16 English-Romanian(En-Ro), medium-scale WMT16 English-German (En-De), and large-scale WMT17 Chinese-English (Zh-En). English-Romanian This data set contains about 0.6M processed parallel sentence pairs tokenized by Moses toolkit (Koehn et al., 2007) and segmented with 40K merge operations using BPE (Sennrich et al., 2016). We use news-dev 2016 and news-test 2016 as the validation set and test set respectively. English-German The WMT16 En-De data set consists of about 4.5M parallel sentences pairs coded with 30K BPE merge-operations. For evaluation, we average the last 5 epochs and report results on all test sets from WMT2016 to WMT2020. Chinese-English Our data set consists of over 20M parallel sentence pairs. The English and Chinese sentences are tokenized with Moses toolkit and Stanford Segmenter respectively, which are further applied 32K BPE segmentation. We use newsdev2016 for validation and newstest2017 for testing. To fairly compare each method, we reproduce all compared methods with transformer model (Vaswani et al., 2017) using open-source toolkit Fairseq (Ott et al., 2019), with the same model configuration and hardware facilities. We use transformer base configuration for all experiments, with 6 encoder and decoder layers, 512 hidden dimensions, 8 attention heads and 2048 FFN dimensions. We train all models with 4000 warm-up steps, initial learning rate of 7 e  4 , label smoothing factor of 0.1, and Adam optimizer with  1 = 0 . 9 ,  2 = 0 . 98 and (cid:15) = 1 e  9 as Vaswani et al. (2017). We set the dropout rate to 0.2 for small-scale En-Ro task and 0.1 for En-De and Zh-En tasks. All experiments are conducted on 4 GeForce RTX 3090 GPUs with a distributional batch-size of 4096 tokens each GPU and an overall accumulated batch-size of 4096  8 tokens. During inference, we use beam size of 4 and length penalty of 0.6 for all tasks. For En-Ro and En-De translation tasks, we share the vocabulary for source and target and apply three-way weight tying(TWWT) (Press and Wolf, 2017) for training, the vocabulary sizes of both tasks are limited to 32768 tokens. We train models for 50 epochs for both tasks. For Zh-En translation task, the Chinese and English vocabulary sizes are 44K and 33K respectively, and models are trained for 300K steps. We reproduce four representative perturbation regularization methods and recently proposed R-Drop for comparison. Word-Drop We implement word-dropout (Gal and Ghahramani, 2016) by randomly replace word embeddings with zero vectors with  = 0 . 05 in Eq.(3). SSE-SE The SSE-SE is a word-replacement method that randomly replaces input tokens with other tokens in vocabulary. As in Wu et al. (2019), we set  = 0 . 01 in Eq.(3) and sample perturbation sequence with uniform distribution. Scheduled Sampling Scheduled sampling (Ben-gio et al., 2015) is a word-replacement method that randomly replace target-side input tokens with model predictions. Each model prediction token is sampled using model's output distribution. The replacement rate  follows a curriculum learning strategy:  i = k k + exp ( i/k ) , (12) where i represents training steps, and k is a hyper-parameter depending on the speed of convergence. Our implementation of scheduled sampling for transformer is parallel as in Mihaylova and Martins (2019) and Duckworth et al. (2019). We set k = (4590 , 29350 , 36150) for En-Ro, En-De and Zh-En tasks respectively. The hyper-parameter k is set to make sure that  i is decayed to 0.9 at the end of training. AdvT For adversarial training, we set  = 1 in Eq.(4) and  = 1 in Eq.(7) as Sato et al. (2019). R-Drop R-Drop (Liang et al., 2021) is a very recent work whose implementation is similar to PD-R. However, its motivation is to restrict the freedom of parameters by reducing sub-model divergence, while ours is to avoid token-level sample fitting problems reflected by prediction difference. Since predictions are carried out by sub-models during training, the fitting bias of sub-models is also included in the prediction difference. From this point, R-Drop can be viewed as a sub-component of PD-R. SacreBLEU (Post, 2018) of compared methods and PD-R on three translation tasks are illustrated in table 2 and table 3. We apply PD-R on encoder-side word-dropout, decoder-side word-dropout, and both-side word-dropout. For all compared methods involving input perturbation, perturbation is applied on both sides of the model except scheduled sampling. Note that selective training of word-dropout regularization (only S n , referred as 'ST') is also presented for comparison with Word-Drop and PD-R. Experiments show that existing perturbation regularization methods are similarly effective compared to each other, which is consistent with Takase and Kiyono (2021). R-Drop and selective train-ing(ST) of word-dropout regularization are consistently better than existing perturbation regularization. Our PD-R against word-dropout significantly improves over word-dropout and other perturbation regularization methods on all three tasks, and also performs better than R-Drop on small-scale and medium-scale tasks. On WMT16 En-De, PDR achieves 1.80 SacreBLEU improvement over vanilla transformer, 1.12 SacreBLEU improvement over existing word-dropout perturbation regularization, and 0.73 SacreBLEU improvement over R-Drop. On large scale WMT17 Zh-En task though, the improvement of perturbation regularization gets smaller compared to small and medium tasks, and 7670 R-Drop performs better than PD-R. We attribute it to the fact that large-scale tasks are sufficient in data, regularization in data level has become a burden rather than help while regularizing sub-model bias is still beneficial. Longer sentences contain more complex word combinations that are unseen or seldom seen in the training set and suffer more from exposure bias (Ranzato et al., 2016; Zhang et al., 2019). Performance on long sentences reflects the model's robustness to unexpected inputs. In experiment, we evaluate the performance of different models on WMT16 En-De task. We combine 5 test sets from WMT16 to WMT20 and divide samples into 7 subsets according to sentence length. As shown in figure 2a, PD-R achieves better results in all subsets, and the improvement tends to become larger as the sentence length grows, which implies that PD-R can better handle unexpected inputs of long sentences. To better evaluate model's robustness to perturbations, we conduct perturbation attack for all models, similar as Michel and Neubig (2018) and Moradi and Samwald (2021). In experiment, we apply word-dropout on source sentences and generate target sentences based on perturbed source sentences. Experiment results in figure 2b show that PD-R against word-dropout and existing word-dropout regularization are consistently better than the base model, and the gap becomes larger as the proportion of perturbation grows, which confirms that our approach does improve the model's robustness to perturbation. Note that our experiments on other types of perturbation attack conclude that a model is robust to a certain type of perturbation only if the model is trained on this kind of perturbation, so comparison of different perturbation regularization methods under one certain type of perturbation attack is not the focus of our discussion in this subsection. Ablation study in table 4 shows that training only the positively influenced subset S p using PD-R is also effective, even more effective than training only S n . This indicates that PD-R can properly handle both under-fitting and over-fitting. As mentioned in section 3, sub-model bias is also a source of improper fitting problems. tinguish the contribution of parameter dropout and word-dropout, we conduct experiments where the difference of two passes is restricted to only parameter dropout or only word-dropout. We also conduct experiments on the encoder side and decoder side separately. Experiment results show that parameter dropout is an important source of improvement, word-dropout is nearly as important as parameter dropout for PD-R, while using both of them gets the best results. As for the difference between the encoder side and decoder side, the decoder-side word-dropout contributes more on the En-Ro task, while on the En-De task the contribution of the encoder side is much bigger, this is also true when the two passes have no parameter dropout difference. The encoder side gets more important on larger data set, which is consistent with the main results. Works involving Input Perturbation Apart from the works mentioned above, some works introduce subword uncertainty at the subword segmentation stage, including sampling multiple subword candidates (Kudo, 2018), applying subword dropout (Park et al., 2020) or producing adversarial subword segmentation (Provilkov et al., 2020). For character-level tasks, there are also works using character-level perturbation including character-level random deletion, insertion, substitution and swap (Belinkov and Bisk, 2018; Karpukhin et al., 2019) and adversarial substitution (Ebrahimi et al., 2018). The mixup technique for NLP tasks can also be seen as a form of perturbation where samples are perturbed (mixed) with other samples for data augmentation or generation diversity (Guo et al., 2020; Li et al., 2021; Fang et al., 2022). Our work can be regarded as one example of perturbation regularization. However, unlike previous perturbation regularization works which are focused on finding better perturbation, our work improves the training mechanism and can be applied to any type of perturbations. Influence of Perturbation Perturbation is commonly considered as a negative factor for neural models by previous works (Szegedy et al., 2014; Liang et al., 2018; Belinkov and Bisk, 2018), which is generally correct with the fact that perturbation does degrade the training and inference accuracy of a model. Belinkov and Bisk (2018) demonstrates that the performance of NMT systems degrades monotonously as input modification increases, which is consistent with our observations. Based on the above facts, perturbation regularization is frequently studied to enhance models' robustness to unexpected inputs at the inference stage. From a data selection perspective, Khayrallah and Koehn (2018) and Briakou and Carpuat (2021) demonstrate that noisy or semantically divergent data is harmful to the training of NMT models. In this paper, we find that the interaction between perturbation and model is complicated and positive influence of perturbation is very common, which is further regarded by us as a sign of relatively under-fitting and a variable that needs to be restricted. Prediction Difference Prediction difference is usually considered as a reflection of the relationship between input and output and is often used to analyze model behavior. Zintgraf et al. (2017) utilizes prediction difference to visualize the importance of a specific input image area to model decision. Li et al. (2019b) uses the prediction difference of a target word when a source word is removed to induce word alignment and find it more accurate than attention weights. Guo et al. (2019) finds that adversarial examples can be accurately and efficiently detected via prediction difference. Liang et al. (2021) proposes R-Drop and take prediction difference as a regularization term to regularize sub-model divergence. In this work, prediction difference is used as an analytical tool to detect improper fitting problems and also a regularization term to regularize the model's fitting bias to 7672 token-level samples. In this paper, we propose to use probability difference for ground-truth tokens before and after input perturbation as an indicator to analyze the influence of different types of perturbations and attribute probability difference to improper fitting of token-level samples. We find that under-fitting is almost as common as over-fitting, which is totally ignored and further aggravated by existing perturbation regularization methods. To regularize both under-fitting and over-fitting, we use prediction difference as a regularization term (PD-R) and apply it on word-dropout regularization. Our method achieves significant improvement over existing methods on three WMT translation tasks and is proved more robust to input perturbation. We thank all the anonymous reviewers for their insightful and valuable comments. This work was supported by National Key R&D Program of China (NO. 2017YFE0192900).
abstain method result method abstain result result abstain abstain abstain abstain abstain abstain objective objective method abstain method abstain abstain objective method abstain method result result objective other other other other other other other other other other other other other other abstain abstain abstain abstain abstain method abstain method abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain other abstain method abstain abstain abstain abstain abstain method abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain method abstain abstain abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain method method abstain method method abstain method abstain abstain abstain abstain method abstain abstain abstain other other other abstain abstain other abstain other other abstain other other other other other method objective result method result other other