Current predominant neural machine translation (NMT) models often have a deep structure with large amounts of parameters, making these models hard to train and easily suffering from over-fitting. A common practice is to utilize a validation set to evaluate the training process and select the best checkpoint. Average and ensemble techniques on checkpoints can lead to further performance improvement. However, as these methods do not affect the training process, the system performance is restricted to the checkpoints generated in the original training procedure. In contrast, we propose an online knowledge distillation method. Our method on-the-fly generates a teacher model from checkpoints, guiding the training process to obtain better performance. Experiments on several datasets and language pairs show steady improvement over a strong self-attention-based baseline system. We also provide analysis on data-limited setting against over-fitting. Furthermore, our method leads to an improvement on a machine reading experiment as well. Neural Machine Translation (NMT) (Cho et al., 2014; Sutskever et al., 2014) has been rapidly developed during the past several years. For further performance improvement, deeper and more expressive structures (Johnson et al., 2017; Barone et al., 2017b; Gehring et al., 2017; Vaswani et al., 2017) have been exploited. However, all of these models have more than hundreds of millions of parameters, which makes the training process more challenging. During the training of NMT models, we notice the following two problematic phenomena: First, the training process is unstable. This is evidenced by the decreasing of training loss with  Corresponding Author. fluctuate performance on the validation set. Second, the performance on validation set usually begins to worsen after several epochs, while the training loss keeps decreasing, which suggests the model being at risk of over-fitting. In order to alleviate these issues, the common practice is to periodically evaluate models on a held-out set (with each evaluated model saved as a checkpoint ). Training is terminated when m consecutive checkpoints show no improvement and select the checkpoint with best evaluation score as the final model. Further improvement can be achieved by utilizing more checkpoints, by smoothing , which averages these checkpoints' parameters to generate more desirable parameters (Sennrich et al., 2016a); or by ensemble , which averages these checkpoints' output probabilities at every step during inference (Chen et al., 2017). However, we notice that all of these methods have a limitation. Once the training process gets parameters with poor performance, selecting, smoothing or ensemble from the checkpoints in this process may have limited generalization performance as well. We impute the limitation to the offline property of these methods. In other words, only employing checkpoints after training cannot affect the original training process. In this paper, we propose to utilize checkpoints to lead the training process. Our method is carried out in a knowledge distillation manner. At each training step, because being evaluated on the held-out validation data, the best checkpoint up to the current training step can be seen as a model with the best generalization ability so far. Therefore, we employ this checkpoint as the teacher model, and let the current training model, as the student, learn from the output probability distributions of the teacher model, as well as truth translations in the training data. Such kind of knowledge distillation is performed on-the-fly because the teacher model could always be updated once any latest better checkpoint is generated. We call our method O nline D istillation from C heckpoints (ODC). We conduct experiments on four translation tasks (including two low-resource tasks), and one machine reading comprehension task. All the results demonstrate that our ODC method can achieve improvement upon strong baseline systems. ODC also outperforms checkpoint smoothing and ensemble methods, without extra cost during inference. We can achieve further improvement by combining ODC with those methods. Major contributions of our work include: 1. In contrast to checkpoint smoothing and ensemble which do not affect the training process, we explore the way to distill knowledge from checkpoints to lead the training process in an on-the-fly manner(  3.1,  3.2). We obtain better performance by replacing the best checkpoint with moving average parameters at that step. (  3.3) 2. We conduct experiments on four translation tasks, including two low resource tasks. In all the tasks our method outperforms strong baseline systems (  4.2,  4.3). We also conduct an experiment on machine reading comprehension task and the result shows that our method can be applied to other tasks too (  4.4). 3. We conduct comprehensive analysis and show that our method can significantly alleviate over-fitting issue in low-resource condition (  5.1), and help to find a wider minimum which brings better generation (  5.2). Neural Machine Translation (NMT) systems learn a conditional probability P ( Y | X ) for translating a source sentence X = ( x 1 , ..., x M ) to a target sentence Y = ( y 1 , ..., y N ) , in which x i and y j are the i -th word and j -th word in sentence X and Y , respectively. An NMT model usually consists of an encoder (parameterized by  enc ) and a decoder (parameterized by  dec ). The encoder transforms a sequence of source tokens into a sequence of hidden states: H ( X ) = ( h 1 , ..., h M ) = f enc ( X ;  enc ) . The decoder of NMT is usually a network computing the conditional probability of each target words y j based on its previous words and the source sentence: where s j is the hidden state of decoder at time step j , p is the distribution of NMT model and  is all the parameters of NMT model. The standard way to train an NMT model is to minimize the cross-entropy between the one-hot distribution of the target sentence and the NMT model's output distribution: L (  ) =  N (cid:88) j =1 |V| (cid:88) k =1 1 { y j = k } (3)  log p ( y j = k | y <j , X ;  ) ,   = arg min  L (  ) , (4) where 1 (  ) is the indicator function and V is the target vocabulary. Knowledge distillation is a class of methods which transfers knowledge from a pre-trained teacher model T , to a student model S . The teacher model can be a model with large capacity (Bucila et al., 2006) or an ensemble of several models (Hinton et al., 2015). In knowledge distillation, the student model learns to match the predictions of the teacher model. Concretely, assuming that we learn a classification model (parameterized by  ) on a set of training samples in the form of ( x, y ) with |V| classes. Instead of minimizing the cross-entropy loss between one-hot label y and model's output probability p ( y | x ;  ) , knowledge distillation uses the teacher model's distribution q ( | x ) as soft tar-gets and optimizes the loss: LKD (  ) =  |V| (cid:88) k =1 q ( y = k | x ;  T ) (5) log p ( y = k | x ;  ) , where  T parameterizes the teacher model and p ( | x ) is the distribution of the student model. Kim and Rush (2016) proposed that, as the loss of NMT model (Equation 4) can be factored into minimizing cross-entropy loss between the target Checkpoints: Teacher Models: training steps direction update teacher models knowledge distillation validation score (darker means better)    +1    +2          +1 Figure 1: Illustration of online distillation from checkpoints(ODC). words and word-level probabilities of the NMT model for every position at target side, knowledge distillation on multi-class classification can be naturally applied. They defined word-level knowledge distillation (W-KD) on a sentence as: LW-KD (  ) =  N (cid:88) i = j |V| (cid:88) k =1 q ( y j = k | y <j , H ( x );  T ) (6)  log p ( y j = k | y <j , H ( x );  ) , where V is the target vocabulary. They further proposed sequence-level knowledge distillation (S-KD), which optimizes the student model by matching the predictions of the teacher model in the probability distribution over the space of all possible target sequences: LS-KD (  ) = (cid:88) Y   q ( Y | X ;  T ) log p ( Y | X ;  ) , (7) where  is the space of target side sentences. As summing over exponential numbers of samples here is intractable, they proposed to train student model on samples generated by teacher model as an approximation. which not only requires one pre-training process to obtain the teacher model, but also limits the power of leading the training process. In contrast, we are aiming at a more integrated process where the teacher model does not come from a separate training process, but from the current training routine itself. More specifically, we update the teacher along with the training process, so the distilled knowledge could be updated when a stronger model comes out. Figure 1 illustrates the paradigm of our method. In generation tasks, the knowledge distillation could be performed at the word-level or sequence-level. In this paper, we focus on the word-level distillation because this distillation only needs forced teaching, which could be performed efficiently together with the training of the student model compared to generating translations from teacher model. It is more computational-friendly, especially when the NMT models are built with parallelizable convolution (Gehring et al., 2017) or self-attention structures (Vaswani et al., 2017). Observed from the training process of NMT models, performance on the validation set does not improve monotonically. When the performance of the training model on the validation set declines, we could always select the best checkpoint so far as the teacher, because it has the best generalization performance. Specially, when the best checkpoint is generated at the current time step, we only update the teacher model but perform no distillation. The online distillation process is summarized in Algorithm 1. We use t to denote the training step and  t to denote the parameters at time step t . We denote T k as the time step the k -th time when the model is evaluated on validation and  T as the validation interval, for which T k +1 = T k +  T . Let  T k (  T k  T k ) be the time step when the best checkpoint is obtained up to T k , and  T as the teacher's parameters to lead the following training process. If the current checkpoint is the best checkpoint so far, i.e.  T k = T k , we update the teacher to be this new checkpoint  T =   T k (in Line 16 and 20). The loss for the training process at time step t ( T k < t < T k +1 ) is defined as follows: L t (  ) = (cid:40) L (  )  T k = T k L (  ) + LW-KD (  ) otherwise , (8) where L (  ) and L (  ) W-KD is defined in Equation 4 and 7, respectively (in Line 5-8). Knowledge distillation usually works better when teacher models have better performance. As Tarvainen and Valpola (2017) proposed in their work, averaging model parameters over training steps tends to produce a more accurate model that using final parameters directly. They called this method as Mean Teacher . Following Tarvainen and Valpola (2017), besides updating parameters, we maintain the exponential moving average (EMA) of the model parameters as:  (cid:48) t =  (cid:48) t  1 + (1   )  t , (9) where t is the update step,  is the parameters of the training model and  (cid:48) the parameters of EMA. is the decay weight which is close to 1.0, and typically in multiple-nines range, i.e., 0.999, 0.9999. By doing so, at each timestep t , parameters of NMT model  t has their corresponding EMA parameters  (cid:48) t . Whenever we update teacher model  T with the current best checkpoint, we can use its EMA parameters instead (in Line 17-18). It can further improve the generalization ability of the teacher model, and bring a better performance of knowledge distillation. We will show in  4.2 that using meaning teacher indeed achieves better performance. Algorithm 1: Online Distillation from Checkpoints 1 Input: validation interval  T ; validation count k ;EMA decay weight  ; initial model parameters  0 2 Initialization: k = 0 ; t = 0 ; T 0 =  1 ,  T 0 =  1 ;  T =  ;  (cid:48) 0 =  0 ; L 0 (  ) = L (  ) 3 while not reach stopping criteria do 4 repeat 5 if  T k = T k then 6 L t (  ) = L (  ) 7 else 8 L t (  ) = L (  ) + LW-KD (  ) 9 minimize L t (  ) and update  t ; 10  (cid:48) t =  (cid:48) t  1 + (1   )  t ; 11 t = t + 1 ; 12 until t mod  T == 0 ; 13 T k +1 = t ; 14 evaluate on validation set; 15 if get better checkpoint then 16  T k +1 = t ; 17 if use EMA as teacher then 18  T =  (cid:48) t ; 19 else 20  T =  t ; 21 else 22  T k +1 =  T k ; 23 k = k + 1 ; 4 Experiments 4.1 Setups To evaluate the effectiveness of our method, we conduct experiments on four machine translation tasks: NIST Chinese-English, WMT17 Chinese-English, IWSLT15 English-Vietnamese, and WMT17 English-Turkish. We conduct experiments based on an open source implementation of Transformer (Vaswani et al., 2017) model in NJUNMT-pytorch 1 . For all the translation experiments, we use SacreBLEU 2 to report reproducible BLEU scores. We also present an experiment on machine reading comprehension, showing our method could also be applied to other tasks. Datasets For NIST Chinese-English translation task, training data consists of 1.34M LDC sentence pairs 3 , with 40.8M Chinese words and 45.8M English words, respectively. We use NIST2003 dataset set as the validation set and NIST 2004, 1 https://github.com/whr94621/NJUNMT-pytorch 2 https://github.com/awslabs/sockeye/tree/master /sockeye contrib/sacrebleu 3 The corpora includes LDC2002E18, LDC2003E07, LDC2003E14, Hansards portion of LDC2004T07, LDC2004T08 and LDC2005T06 SYSTEMS NIST Chinese-English NIST03 NIST04 NIST05 NIST06 Average  RNNSearch (Zhang et al., 2018b) 36.59 39.57 35.56 35.29 -Transformer-base(Yang et al., 2018) 42.23 42.17 41.02 -baseline 43.78 44.26 40.97 38.93 41.39 baseline + LKS 44.12 44.87 41.59 39.22 41.89 +0.50 baseline + BKS 44.23 44.98 41.62 39.74 42.11 +0.73 baseline + BKE 44.30 45.01 41.86 40.05 42.31 +0.92 ODC 45.33 45.18 42.60 39.67 42.48 +1.10 ODC + LKS 45.05 45.49 42.99 40.48 42.99 +1.60 ODC + BKS 45.35 45.49 43.21 39.96 42.89 +1.50 ODC + BKE 45.34 45.92 43.35 40.30 43.19 +1.80 ODC-EMA 45.52 45.72 43.01 40.65 43.13 +1.74 Table 1: Case-insensitive BLEU scores of Chinese-English translation on NIST datasets. 2005, 2006 as test sets. We filter out sentence pairs whose source or target side contain more than 50 words. We use BPE (Sennrich et al., 2016b) with 30K merge operations on both sides. For WMT17 Chinese-English translation task, we use the pre-processed version released by WMT 4 . We only use CWMT part of WMT Corpus. We use newsdev2017 as the validation set and newstest2017 s the test set. We learn a BPE model with 32K merge operations and keep all the BPE tokens in the vocabulary. We limit the maximal sentence length as 100 after BPE segmentation. For IWSLT15 English-Vietnamese translation task, we directly use the pre-processed data used in Luong and Manning (2015) 5 , which has 133K sentence pairs, with 2.70M English words and 3.31M Vietnamese words. We use the released validation and test set, which has 1553 and 1268 sentences respectively. Following the settings in Huang et al. (2017), the Vietnamese and English vocabulary size are 7,709 and 17,191, respectively. For WMT17 English-Turkish translation task, We use the pre-processed data released by WMT17 6 . It has 207K sentence pairs, with 5.21M English words and 4.63 Turkish words. We use newstest2016 as our validation set and newstest2017 as the test set. We use joint BPE segmentation (Sennrich et al., 2017) to process the whole training data. The merge operations are 16K. 4 http://data.statmt.org/wmt18/translation-task/preprocessed/zh-en/ 5 https://github.com/tefan-it/nmt-en-vi 6 http://data.statmt.org/wmt17/translation-task/preprocessed/tr-en/ Implementation Details Without specific statement, we follow the transformer base v1 hyper-parameters settings 7 , with 6 layers in both encoder and decoder, 512 hidden units and 8 attention heads in multi-head attention mechanism and 2048 hidden units in feed-forward layers. Parameters are optimized using Adam(Kingma and Ba, 2014). The initial learning rate is set as 0.1 and scheduled according to the method proposed in Vaswani et al. (2017), with warm-up steps as 4000. We periodically evaluate the training model on the validation set by doing translation and compute the BLEU scores. We stop training when 50 subsequent of BLEU scores on validation set do not get improvement. We use beam search with beam size as 5. We first evaluate the capability of our method for improving performance when there are plenty of training data. We conduct experiments on both NIST and WMT17 Chinese-English Translation tasks. Results on NIST Dataset We compare our method with several ways to utilize checkpoints 8 :  last-k-smoothing : After training the baseline model, we average the parameters of the last k checkpoints as the final model. as the final model. In this case, checkpoints may have better performance but higher variance which could be harmful to parameters averaging. best-k-ensemble : Do ensemble inference (av-erage the output probabilities) with the best k checkpoints (Chen et al., 2017). As shown in Table 1, our baseline is comparable to the other two recent published results (Zhang et al. (2018b), Yang et al. (2018)). In consistent with Chen et al. (2017), using checkpoints for smoothing or ensemble does improve the baseline system. Using EMA parameters also improve the baseline system as well, which is in consist with (Tarvainen and Valpola, 2017). Compared to the baseline, our approach ODC brings translation improvement across different test sets and achieves 42.48 BLEU scores on aver-age(+1.09 BLEU v.s. baseline). This result con-firms that using best checkpoint as teacher indeed helps improving the performance of the translation model. Besides, ODC is comparable to the best results among smoothing and ensemble on baseline's checkpoints (achieved by best-k-ensemble). Considering that best-k-ensemble needs to decode with k models, while ODC decodes only one, our model enjoys a better efficiency. Furthermore, we can achieve further improvement by combining these methods on checkpoints generated by ODC. Results also show that ODC-EMA (  3.3) could achieve additional improvement from ODC itself (43.13 v.s. 42.48 BLEU), demonstrating that using EMA of the best checkpoint instead can bring better knowledge distillation performance, as it generates a better teacher model. Results on WMT17 Dataset We present the results on WMT17 Chinese-English translation task in Table 2. We report the results of the baseline, ODC and a recent result published by Zhang et al. (2018c). To make a fair comparison, we follow the experiment setting in Zhang et al. (2018c). The experiment results show similar trends with those on the NIST datasets. Applying ODC leads to the result of 24.22 BLEU, which is 0.85 BLEU higher compared with baseline. We also apply our method to two low resource translation tasks, i.e., IWSLT2015 English-Vietnamese (EN2VI) and WMT17 English-Turkish (EN2TR). Due to the limited amount of training data, models are more likely to suffer from over-fitting. Therefore, we use a higher dropout rate of 0.2 and weight decay, another common technique against over-fitting, with decay weight set as 10  3 as the default setting. We implement weight decay as AdamW (Loshchilov and Hutter, 2017) does. Besides, we further experiment with grid search on the validation set for optimal hyper-parameters of dropout rate and weight decay, which may lead to better results. We adopt a simple heuristic, which first searches an optimal dropout rate, and then further searches weight decay coefficients based on this dropout. We experiment with dropout as 0.2, 0.3, 0.4, and weight decay as 10  1 , 10  2 and 10  3 . As in Table 3, our baseline is comparable to two recent published results, respectively: EN2TR from Zhang et al. (2018c) and EN2VI from offi-cial release tensor2tensor problem 9 . Grid hyper-parameter search does improve the baseline system. ODC leads to better results compared to the baseline, as well as the baseline with grid parameter search. ODC can achieve further improvement after searching for optimal hyper-parameters of dropout and weight decay. Although our main research is focused for the task of machine translation, the idea of ODC could be applied to other tasks as well. We experiments on the Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al., 2016), a machine reading comprehension task. SQuAD contains 107,785 human-generated reading comprehension questions, with 536 Wikipedia articles. Each question is associated with a paragraph extracted from an article, and the corresponding answer is a span from this article. A machine reading comprehension model is designed to predict the start and end positions in the article of the answer. The state-of-the-art machine reading comprehension system also employs a deep neural network structure, which is similar to NMT. We apply our ODC method on BiDAF++ (Choi et al., 2018), a multi-layer SQuAD model that augments BiDAF (Seo et al., 2016) with self-attention and contextualized embeddings. We evaluate the model after each epoch and implement the knowledge distillation by teaching the student with the output distribution of answer start and end positions predicted by the best checkpoint. For the results, ODC improves a base BiDAF++ from 76.83 to 77.40, in EM scores, showing that our method can be applied to a broader range of tasks. We conduct further analysis to probe into the reasons for the advantages of ODC. We first show that our method can significantly alleviate the over-fitting issue in data-limited condition. After that, we show that parameters gained from our method tend to be wider minimums, which represents better generalization. Taking IWSLT15 English-Vietnamese as a test-bed, we analyze whether our method could help handle the over-fitting issue. We first plot the curve of the loss on the validation set at each training step for the different models (in Figure 2, the top curve with rounds). It is easy to see that the loss curve of the baseline increases as the training goes after 50K steps, indicating a severe over-fitting. With better dropout rate and weight decay, the over-fitting is Figure 2: Loss curves (top) and final BLEU scores (bot-tom) on the validation set of baseline, baseline with grid-search and ODC, respectively. less severe; while with ODC the loss curve shows a more steady trend of decrease, and is almost always under the other two's. The final BLEU score on the validation set (Figure 2, bottom) shows corresponding result. The grid search of hyper-parameters improves the BLEU from 26.06 to 26.42 in BLEU, while ODC achieves 26.99. Both results indicate that our method is more effective at handling the over-fitting problem. We hold that minimizing the cross-entropy between the teacher model and the student model serves as regularization to the training of the student model, which avoids the model getting into over-fitting. In the training process in Chinese-English tasks, we do not observe obvious over-fitting issue as shown in low resource translation tasks. In this section, we analyze how ODC helps the model generalization. Keskar et al. (2016) proposed that the width Figure 3: The upper plot shows the validation losses curve along the line segment decided by parameters of baseline and ODC. of the minimum in a loss surface is related to its generalization ability. Therefore, we compare the generalization capability between baseline system and our ODC method by exploring around the parameters. We make use of the visualization technique employed in (Goodfellow and Vinyals, 2014) and analyze the results on the NIST data set. Let  base and  ODC denote the final parameters obtained from baseline and ODC. Consider the line:  (  ) =    ODC + (1 . 0   )   base , (10) which connects  base (  = 0 . 0 ) and  ODC (  = 1 . 0 ). We plot the value of Equation 4 as a function of  (normalized by count of words per sentence) with  =  (  ) . We draw  from  1 . 0 to 2 . 0 at an interval of 0.02. In this way, the width of  base and  ODC can be represented as the steepness of the curve nearby. To further quantitatively represent the steepness, we compute the standard deviation of values on this curve within different distances to the two parameters, respectively. We plot them in Figure From Figure 3 we can see that the loss curve behaves steeper around the parameters of baseline than of ODC. Besides, the standard deviations of losses around the baseline model are consistently higher than ODC within all the distances. It is evident that the parameters of ODC act as a wider minimum c and explains why ODC can lead to a more generalized model. Regularization has broad applications in training NMT models to improve performance and avoid over-fitting. There are some common regularization techniques, such as L 2 normalization and dropout (Srivastava et al., 2014). These methods are simple and easy to implement but need carefully tuning on the validation set. These methods are also orthogonal to our method. There are also some works to exploit regularization techniques in fine tuning of NMT model. Barone et al. (2017a) proposed a tuneout method which randomly replaces columns of weight matrices of out-of-domain parameter matrices. Khayral-lah et al. (2018) shared similar training object with us, as they computed the KL divergence between out-of-domain and in-domain model. Both of their works request a pre-trained teacher model, while we are work on a more general training problem which does not require such kind of model. While traditional knowledge distillation requires a static, pre-trained teacher model, online knowledge distillation tends to overcome this problem by selecting or generating a teacher dynamically from scratch. To the best of our knowledge, Zhang et al. (2017) is the first trial to replace the offline teacher model. They trained peer models to teach each other simultaneously. Compared to their work, our method uses the best checkpoint as the teacher, which avoids introducing extra parameters. Furlanello et al. (2018) tends to update teacher model during the training procedure iteratively, but their method needs to train the teacher model until convergence in each iteration. Instead, our method only needs one phase of training, whose overhead is relatively small. Lan et al. (2018) using an ensemble of several branches of the model as teacher for computer vision tasks, which only needs one-phase training as well. However, their method relies heavily on the multi-branch structures of the tasks, which are not widely applicable in neural machine translation. In this paper, we propose an online knowledge distillation method with the teacher model generated from checkpoints during the training procedure. Experiments on four machine translation tasks and a machine reading task show that our method outperforms strong baseline systems. Further analysis shows that our method can effectively alleviate the over-fitting issue, and tend to find a wider minimum. We would like to thank the anonymous reviewers for their insightful comments. We also thank Boxing Chen from Alibaba Group for his helpful comments. This work is supported by the National Science Foundation of China (No. 61772261, 61672277) and the Jiangsu Province Research Foundation for Basic Research (No. BK20170074). Part of this work is supported by 13th Five-Yea All-Army Common Information System Equipment Pre-Research Project (No. 31510040201).
abstain abstain abstain abstain objective result abstain method abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain method abstain abstain method objective abstain result objective objective result objective method result result result other other other other other other other other other method other other other other other other abstain abstain method abstain abstain method abstain abstain abstain method abstain method method abstain abstain other abstain abstain abstain method abstain abstain method abstain abstain method method method abstain abstain other abstain method method method method method abstain method method method abstain method abstain method method abstain other abstain abstain method method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain method abstain abstain method method abstain abstain method abstain abstain abstain abstain method method abstain method abstain abstain abstain method abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain other other other abstain other other method method other abstain other method other abstain other other objective result result other other other other