Most dominant neural machine translation (NMT) models are restricted to make predictions only according to the local context of preceding words in a left-to-right manner. Although many previous studies try to incorporate global information into NMT models, there still exist limitations on how to effectively exploit bidirectional global context. In this paper, we propose a C onfidence B ased B idirectional G lobal C ontext A ware (CB-BGCA) training framework for NMT, where the NMT model is jointly trained with an auxiliary conditional masked language model (CMLM). The training consists of two stages: (1) multi-task joint training; (2) confidence based knowledge distillation. At the first stage, by sharing encoder parameters, the NMT model is additionally supervised by the signal from the CMLM decoder that contains bidirectional global contexts. Moreover, at the second stage, using the CMLM as teacher, we further pertinently incorporate bidirectional global context to the NMT model on its unconfidently-predicted target words via knowledge distillation. Experimental results show that our proposed CBBGCA training framework significantly improves the NMT model by +1.02, +1.30 and +0.57 BLEU scores on three large-scale translation datasets, namely WMT'14 English-to-German, WMT'19 Chinese-to-English and WMT'14 English-to-French, respectively. In recent years, Neural Machine Translation (NMT) has achieved great progress and attracted more attention. Most dominant NMT models mainly adopt an encoder-decoder framework (Sutskever et al., 2014; Bahdanau et al., 2015; Vaswani et al., 2017; Meng and Zhang, 2019; Song et al., 2019;  This work is done when Chulun Zhou was interning at Pattern Recognition Center, WeChat AI, Tencent Inc, China. * Corresponding author Miao et al., 2021) with the teacher-forcing strategy (Goodfellow et al., 2016) for training. Despite its success, the unidirectional property of teacher-forcing strategy restricts NMT models to only focus on the local context, i.e., the preceding words of the to-be-predicted target word at each decoder step. Apparently, this strategy tends to be limited because word dependencies are always bidirectional involving both preceding and succeeding words on the target side. To address this issue, many previous researches attempt to exploit global information on the target side (Liu et al., 2016; Zhang et al., 2016; Serdyuk et al., 2018; Zhang et al., 2018; Su et al., 2018; Zhang et al., 2019a,b; Su et al., 2019; Zhou et al., 2019; Zhang et al., 2020). Typically, they introduce the modelling of target-side global context in the reverse direction by pairing the conventional left-to-right (L2R) NMT model with a right-to-left (R2L) auxiliary model. However, in these methods, the modelling of reverse global context is separate from the local context of preceding words. Thus, they cannot sufficiently encourage the NMT model to exploit bidirectional global context (Devlin et al., 2019). Meanwhile, some of them adopt bidirectional decoding, which often relies on multi-pass decoding or specially customized decoding algorithms (Liu et al., 2016; Zhang et al., 2018; Zhou et al., 2019; Zhang et al., 2020). Another series of studies (Conneau and Lample, 2019; Edunov et al., 2019; Weng et al., 2020; Baziotis et al., 2020; Yang et al., 2020; Chen et al., 2020) resort to leveraging target-side bidirectional global context contained in large-scale pre-trained language models (PLM), such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019). These PLMs are normally not bilingual-aware for translation and trained independently of the NMT model. As a special case, Chen et al. (2020) design a conditional masked language modelling objective to make BERT aware of source input during the fine-2878 0% 5% 10% 15% 20% 25% 30% P r o p o r t i o n o f t o t a l t a r g e t w o r d s 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Predicted probabilities to the target ground-truths 5.07 6.20 8.77 22.31 12.00 25.67 4.82 4.44 4.54 6.19 Figure 1: The distribution of the NMT-predicted probabilities to the corresponding ground-truth words on the training set of WMT'14 English-to-German translation task, which is output by a fully-trained Transformer model using teacher-forcing strategy. tuning stage. Nevertheless, in these approaches, the pre-trainings of PLMs are independent of NMT models, limiting the potential of model performance. As for how to effectively incorporate global information into NMT models, another notable de-ficiency of previous work is that they do not pertinently enhance the NMT model according to its word-level prediction confidence. Ideally, under the teacher-forcing strategy, a well-trained NMT model should assign high probabilities to the target ground-truth words based on correct previous words, which, however, is not the case. Figure 1 depicts the predicted word-level probabilistic distribution of a fully-trained Transformer model. We find that, even based on totally correct preceding words, there is a considerable portion of target ground-truth words that the model predicts with relatively low probabilities. The reasonable cause of this phenomenon is that the NMT model cannot confi-dently predict these target words according to only the local context of preceding words (Watanabe and Sumita, 2002; Hoang et al., 2017). Hence, we should especially refine the NMT model on these unconfidently-predicted target words. In this paper, we propose a C onfidence B ased B idirectional G lobal C ontext A ware (CBBGCA) training framework for NMT. Under our framework, the NMT model is jointly trained with a conditional masked language model (CMLM) which is essentially bilingual-aware and contains bidirectional global context on the target side. Specifically, the CBBGCA training consists of two stages. At the first stage, we jointly train the NMT model and CMLM in a multi-task learning manner by sharing the encoders of the two models. This preliminarily enhances the NMT model because the encoder is additionally supervised by the signal from the CMLM decoder that contains bidirectional global context. At the second stage, we employ the CMLM to pertinently refine the training of the NMT model on those unconfidently-predicted target words via confidence based knowledge distillation. By doing so, our model can be further encouraged to effectively leverage the bilingual-aware bidirectional global context contained in the CMLM. To sum up, the major contributions of our paper are as follows:  We introduce multi-task learning to benefit the NMT model by sharing its encoder with an auxiliary CMLM, which preliminarily enhances the NMT model to capture bidirectional global context. We further propose confidence based knowledge distillation using the CMLM as teacher to especially refine the NMT model on unconfidently-predicted target words, more effectively exploiting the bidirectional global contextual information. Extensive experiments on large-scale WMT'14 English-to-German, WMT'19 Chinese-to-English and WMT'14 English-to-French translation tasks show that our CBBGCA training framework respectively improves the state-of-the-art Transformer model by +1.02, +1.30 and +0.57 BLEU points, which demonstrate the effectiveness and generalizability of our approach. In this section, we will introduce our proposed CBBGCA training framework that employs a CMLM to enhance the NMT model according to its prediction confidence. In the following subsections, we first describe the basic architectures of our NMT model and CMLM. Then, we introduce the training procedures of our CBBGCA framework, involving two stages. Both the NMT model and CMLM are based on Transformer (Vaswani et al., 2017), which is essen-2879 tially an attentional encoder-decoder framework. 1 2.1.1 Encoder The encoders of the NMT model and the CMLM are identical, which are mainly used to learn the semantic representations of the source sentence. Generally, the encoder consists of L e identical layers, each of which contains two sub-layers: a self-attention (SelfAtt) sub-layer and a position-wise feed-forward network (FFN) sub-layer. The SelfAtt sub-layer takes the hidden states of the previous layer as inputs and conducts multi-head scaled dot-product attention. Let h ( l ) denote the hidden states of the l -th encoder layer, the SelfAtt sub-layer can be formulated as c ( l ) = AN(SelfAtt( h ( l  1) , h ( l  1) , h ( l  1) )) , (1) where AN(  ) denotes the AddNorm , i.e., layer normalization with residual connection. Afterwards, the FFN sub-layer is applied, h ( l ) = AN(FFN( c ( l ) )) . Note that h (0) is initialized as the embedding sequence of the source sentence and the hidden states of the L e -th layer h ( L e ) are used as the final word-level representations of the source sentence. The decoders of the NMT model and the CMLM are similar except their self-attention mechanisms and prediction manners. The NMT Decoder. It is comprised of L d identical layers with each having three sub-layers: a masked self-attention (MaskSelfAtt) sub-layer, a cross-attention (CrossAtt) sub-layer and an FFN sub-layer. Particularly, to preserve the autoregressive property at each time step, the MaskSelfAtt sub-layer performs self-attention with an attention mask that prevents the decoder from seeing succeeding words. To generate the hidden states s ( l ) of the l -th decoder layer, the MaskSelfAtt sub-layer can be formulated as a ( l ) = AN(MaskSelfAtt( s ( l  1) , s ( l  1) , s ( l  1) )) . 1 Please note that our framework can also be adapted to other NMT models. Next, the FFN sub-layer maps z ( l ) into s ( l ) : s ( l ) = AN(FFN( z ( l ) )) . Finally, with the source sentence x , the target translation y <t and the learned top-layer hidden states s , the decoder models the probability distribution over the target vocabulary at the t -th time step as follows: The CMLM Decoder. Typically, it predicts a set of masked target words y m given the source sentence x and the set of observable target words y o . The CMLM decoder also contains L d identical layers, each of which also includes a SelfAtt sub-layer, a CrossAtt sub-layer, and an FFN sublayer. Unlike the MaskSelfAtt sub-layer of the NMT decoder, the attention mask is removed in the SelfAtt sub-layer of the CMLM decoder. Finally, with the learned top-layer hidden states s (cid:48) of the CMLM decoder, the predicted probability distribution for every masked target word y t  y m can be formalized as p ( y t | y o , x ) = softmax( W (cid:48) s (cid:48) t ) , (7) where W (cid:48) is the learnable parameter matrix of the linear transformation. Note that since the CMLM decoder takes y o rather than y <t as input, which includes both preceding and succeeding words with respect to every masked target word, it should contain bidirectional global contextual information. The training of CBBGCA framework involves two stages. At the first stage, we jointly train the NMT model and CMLM by multi-task learning. At the second stage, according to the word-level prediction confidence, we employ the CMLM to refine the training of the NMT model through knowledge distillation. In the first training stage, given a batch of training instances, we jointly train the NMT model and CMLM by simultaneously optimizing their respective objectives: L 1 (  e ,  nd ,  cd ) = L nmt +(1   ) L cmlm , (8) where  is a balancing hyper-parameter and  e ,  nd and  cd denote the parameters of the shared encoder, the NMT decoder and the CMLM decoder, respectively. Very importantly, during this procedure of joint training, we share the same encoder for the NMT model and CMLM. In this manner, the NMT model benefits from the multi-task learning because the encoder is additionally supervised by the signal from the CMLM decoder which contains bidirectional global context. Besides, we adopt the strategy used in (Ghazvinine-jad et al., 2019) to optimize the CMLM. Concretely, we randomly select n words, where n  uniform (1 , | y | ) , and replace each word with a special token [M], splitting y into y o and y m . Formally, we optimize the CMLM by minimizing the following objective for every word in y m : L cmlm (  e ,  cd ) =  (cid:88) y t  y m log p ( y t | y o , x ) . At the second stage, once we obtain the two fully-trained models, we use the CMLM to further refine the training of the NMT model through knowledge distillation (KD). The reason why we introduce such a KD-based model training is that the conventional NMT model predicts a considerable portion of target ground-truth words with relatively low probabilities, as shown in Figure 1. This phenomenon indicates that the NMT model cannot con-fidently predict these target words based on only local context of preceding words. Therefore, we aim to pertinently distill the knowledge of CMLM into the NMT model on these unconfidently-predicted target words because the CMLM contains bilingual-aware bidirectional global context. Figure 2 depicts the training procedure of this stage with an illustrative example. Given the source sentence x and the preceding ground-truth words y <t at each time step t , we first let the NMT model make predictions for every target word using Equation 6, producing word-level probability distributions  p 1 ,  p 2 , ...,  p | y | . Then, we determine the word set y m where the predicted probabilities  p  t to the corresponding ground-truth words are lower than a threshold value (cid:15) , y m = { y t |  p  t  (cid:15), 1  t  | y |} . Next, we obtain the set y o of partially observable target words by replacing those selected ground-truth words with a special token [M]. Subsequently, we feed y o to the CMLM and obtain its predicted probability distribution  q t for every word in y m using Equation 7. To pertinently refine the NMT model on the set y m of its unconfidently-predicted target words, we use the CMLM with fixed parameters as teacher and transfer its knowledge to the NMT model. Along with the supervision from the corresponding ground-truth words, we optimize the NMT model with a balancing factor  through the following 2881 objective: L kd (  ne , nd )= (cid:88) y t  y m {  KL( q t ||  p t )  (1   )log  p  t } , (12) where  ne ,  nd and KL(  ) represent the NMT encoder parameters, the NMT decoder parameters and the KullbackLeibler divergence (Sohn et al., 2015), respectively. Here, we follow (Clark et al., 2019) to linearly decrease the factor  from 1 to 0 throughout training. This guides the NMT model to absorb more knowledge from the CMLM at the early period of the stage 2 and gradually re-focus on the ground-truth words to learn better. Finally, the total training objective of this stage is as follows: L 2 (  ne ,  nd ) = L kd (  ne , nd )  (cid:88) y t  y o \ [M] log  p  t , (13) where y o \ [M] represents y o excluding all special tokens [M]. By doing so, we can fully strengthen the ability of the NMT model to leverage the bilingual-aware bidirectional global context contained in the CMLM. Note that the CMLM is not involved at the inference time. We carry out experiments on three large-scale translation tasks, WMT'14 English-to-German (En  De), WMT'19 Chinese-to-English (Zh  En) and WMT'14 English-to-French (En  Fr). The data are preprocessed using Byte-Pair-Encoding 2 (Sennrich et al., 2016) (BPE). More dataset statistics and the detailed preprocessing procedures are described in Appendix A. 3.2 Implementation Details We follow the settings used in (Vaswani et al., 2017) to build the NMT model under Transformer-base configuration. Concretely, the Transformer-base architecture is comprised of 6 encoder and decoder layers, each with 512 as hidden size, the FFN sublayers of 2,048 dimension and 8 heads in multihead attentions. For more details about the training and inference, please refer to Appendix B. 3.3 Hyper-parameters Apart from all the hyper-parameters we empirically set based on previous experience, the balancing 2 https://github.com/rsennrich/subword-nmt 26.2 26.4 26.6 26.8 27.0 The balancing factor  BLEU s c o re 0.5 0.6 0.8 0.9 1.0 0.7 factor  in Equation 8 and the confidence threshold (cid:15) for determining y m in Equation 13 are the hyper-parameters we need to manually tune on the validation set. To balance the training of the NMT model and CMLM, we select the minimum  that can bring steady improvements to the NMT model within 200,000 steps. As shown in Figure 3(a), we gradually vary  from 0 . 5 to 1 . 0 with an increment of 0 . 1 and evaluate the performance on the validation set. We find that the NMT model achieves its peak when  = 0 . 7 . Hence,  is set to 0 . 7 for the joint training of the two models at the first stage. Given the selected  , at the second training stage, we also investigate the impact of (cid:15) on the validation set. We adjust its value from 0 . 0 to 0 . 3 with an interval of 0 . 05 . As shown in Figure 3(b), the NMT model performs the best when the (cid:15) is 0 . 2 . Therefore, we set (cid:15) = 0 . 2 as the confidence threshold for the second training stage. In our experiments, CBBGCA is the system under our proposed training framework, as described in Section 2.2. Multi-300k denotes the baseline system that jointly optimizes the NMT model and CMLM by sharing their encoders throughout the whole 300k training steps, which is used to make comparison with conducting CBKD at the second training stage. For evaluation, in addition to the widely used BLEU (Papineni et al., 2002), we also adopt the Comet (Rei et al., 2020) which is recently a more welcomed metric. Results on WMT'14 En  De. Table 1 lists several existing competitive NMT systems and ours. First, we can see that Multi-300k surpasses Transformer by +0.58 BLEU and +0.0101 Comet scores. Moreover, the BLEU and Comet scores 2882 System BLEU Comet Existing Systems Transformer (Vaswani et al., 2017)  27.30 -Rerank-NMT (Liu et al., 2016)  27.81 -ABD-NMT (Zhang et al., 2018)  28.22 -FKD-NMT (Zhang et al., 2019a)  27.84 -SB-NMT (Zhou et al., 2019)  29.21 -DBERT-NMT (Chen et al., 2020)  27.53 Our Systems Transformer 27.30 0.2602 Multi-300k 27.88  0.2703  CBBGCA 28.32 (cid:5) 0.2828 (cid:5) Table 1: BLEU (%) and Comet scores on WMT'14 En  De. of CBBGCA are respectively +1.02 and +0.0226 higher than Transformer, verifying that CBKD at the second training stage brings further improvement. Next, our proposed framework outperforms most recent competitive models. Specifically, CBBGCA yields a better result than ABD-NMT (Zhang et al., 2018) and FKD-NMT (Zhang et al., 2019a) that only provide the NMT model with unidirectional future contexts. This proves the power of incorporating target-side bidirectional global context into the NMT model. Note that ABD-NMT needs two-pass decoding, which first obtains reverse state sequence by a backward decoder and then uses the forward decoder to generate final translations. The only exception is SB-NMT (Zhou et al., 2019) that designs elaborately cum-stomized bidirectional decoding algorithms, which is actually not fairly comparable to ours because of its decoding manners and the involvement of synthetic training data. 3 These results demonstrate the effectiveness of our proposed training framework. Results on WMT'19 Zh  En and WMT'14 En  Fr. From Table 2, the Multi-300k preliminarily gains +0.35 and +0.29 BLEU scores over Transformer on Zh  En and En  Fr, respectively. Moreover, our CBBGCA further strongly outperforms Multi-300k and achieves a total of +1.30 and +0.57 BLEU score improve-3 The result of SB-NMT is the reported performance with checkpoint averaging technique from (Zhou et al., 2019). ments over Transformer on the two datasets, respectively. In term of Comet, comparing different models, we can see similar results. Note that the sizes of WMT'19 Zh  En (20M) and WMT'14 En  Fr (36M) datasets are much larger than that of WMT'14 En  De (4.5M) dataset, demonstrating the effectiveness of our proposed framework on various language pairs. To fully investigate each part of our proposed training framework, we conduct ablation studies on WMT'14 En  De translation task. Table 3 reports the ablation results on the test set. We first validate the necessity of our two-stage strategy by only training the model using the multitask joint training. w/ . Dynamic means the weights in Equation 8 are dynamically adjusted. Specifically, we linearly increase  from 0 . 5 to 1 . 0 throughout the whole training process. We can see that its performance is just slightly higher than the fixed-weight Multi-300k and still significantly inferior to the two-stage CBBGCA. For the feasible options in two-stage strategy,  w/o . ShareEnc represents not sharing encoders at the first training stage and its performance decreases by 0 . 56 BLEU score. This shows that the NMT encoder is enhanced by the joint training 2883 with the CMLM. As for  w/o . CBKD 4 , which means not performing KD on any target words at the second training stage (i.e.,  =0 ), its performance also decreases by 0 . 40 BLEU score. This demonstrates the effect of pertinently incorporating bidirectional global context into the NMT model on its unconfidently-predicted target words. In our training framework, for each sentence pair, we adopt KD to transfer the knowledge of the CMLM into the NMT model only on the word set y m . The set is determined by masking k target words whose NMT-predicted probabilities to the corresponding ground-truths are lower than a threshold (cid:15) . Obviously, there are alternative strategies for the above process. Therefore, we further investigate the following variants:  Random : Regardless of confidence, we randomly select k words of a target sentence to be masked for the CMLM. NMT-High : As a contrast, we mask the target words whose NMT-predicted probabilities to the ground-truths are higher than 1  (cid:15) . NMT-Wrong : We mask the target words where the predictions of the NMT model do not coincide with the corresponding ground-truths. All-at-Once : In this variant, to validate the necessity of selectively distilling knowledge on a portion rather than all of target words, we generate CMLM-predicted probability distributions for all target words. As an extreme case, we mask all target words at once with only source sentences as input to the CMLM. Part-to-All : Instead of masking all target words at once, we generate the CMLM-predicted probability distributions in a part-to-all way. Concretely, we first generate the NMT-predicted probability distributions for all words. Then, all target words are divided into several non-overlapping subsets, each corresponding to a certain probability interval. For each time, we mask a subset of target words whose probabilities to the ground-truths are located within the corresponding interval. 4 It differs from Multi-300k in that the CMLM is not optimized at the second stage (200k  300k steps) in  w/o . CBKD. In Multi-300k, between 200k  300k steps, the CMLM and NMT model continue to be jointly optimized by sharing their encoders. Particularly, because the hyper-parameter (cid:15) is 0 . 2 in CBBGCA, we have a total of 5 iterations and the intervals are set to [0 . 0 , 0 . 2] , [0 . 2 , 0 . 4] , [0 . 4 , 0 . 6] , [0 . 6 , 0 . 8] , [0 . 8 , 1 . 0] . Table 4 lists the results with different KD strategies. We can observe that all these variants are inferior to our CBBGCA method. Particularly, the results of Random and NMT-High indicate that conducting knowledge distillation on either randomly selected or confidently-predicted target words is less effective than on those unconfidently-predicted ones. Next, the result of NMT-Wrong is lower than CBBGCA. It may be due to the fact that the NMT model assigns low probabilities to some correctly-predicted target words. Thus, the NMT model fails to absorb the beneficial knowledge from CMLM on these words. Lastly, All-at-Once and Part-to-All represent two approaches to generate CMLM-predicted probability distributions for all target words. It is reasonable for All-at-Once to obtain a a worse performance since the CMLM cannot predict well without any observable word on the target side. For Part-to-All, we can see it improves the NMT model over Multi-300k but is still worse and takes more computational cost than CBBGCA. This also echoes the finding in NMT-High that applying KD on confidently-predicted target words is not optimal. All these results demonstrate that it is crucial for the NMT model to pertinently exploit bidirectional global contexts on its unconfidently-predicted target words. We also investigate the change of model confidence with respect to target ground-truth words on the training set. Table 5 lists the percentage of tokens within each interval, in terms of NMT-predicted probability. Because the probability higher than 0 . 5 must be the maximum across the vocabulary, we group 0 . 5  1 . 0 as a whole high-confidence interval while the others are low-confidence intervals. From the table, we can observe that the number of tokens in low-confidence intervals drops. For instance, the number of tokens locating in [0 . 0 , 0 . 2] becomes 0 . 69% fewer, which is a notable change considering that the WMT'14 En  De training set contains roughly 4.5 million sentences with a total of approximately 140 million tokens. This indicates that the NMT model becomes more confident about the target ground-truth words. There are also some researches (Edunov et al., 2019; Conneau and Lample, 2019; Yang et al., 2020; Weng et al., 2020; Chen et al., 2020) that focus on incorporating large-scale PLMs into the NMT model. Different from these approaches that require pre-training on massive external data, in this work, the integration of CMLM into the training procedure is to directly provide the NMT model with target-side bidirectional global context without external data. To show that our proposed training framework is compatible and orthogonal to existing approaches involving large-scale PLMs, we conduct experiments with an external Roberta model (Liu et al., 2019) in Appendix C. 4.5 Comparison over Stronger Systems To further validate our proposed training framework, we conduct experiments over stronger systems. Particularly, we first compare our proposed training framework with (Baziotis et al., 2020) that used a bidirectional LM as prior to regularize the NMT model, which is similar to our method to some extent. Following their setting, we train a target-side language model using a 6-layer Transformer decoder. Then, it is used as the teacher model to impose soft constraints on the output of the NMT model. The upper rows of Table 6 System BLEU  Transformer 25.54 ref. give the comparison results. We can see that our CBBGCA still outperforms Transformer + LM prior. In addition, back-translated data are often used to boost NMT models. We also involve additional back-translated data during training. Specifically, we use an in-house English-to-Chinese Transformer-base model to translate the English sentences in the WMT'19 Zh  En training set. Then, we add these back-translated data to the WMT'19 Zh  En training set. The lower rows of Table 6 lists the performance of our models under this setting. Similar to previous results, it shows that both of Multi-300k and CBBGCA consistently improve the NMT model on the BT-augmented WMT'19 Zh  En, demonstrating the the effectiveness of our proposed training framework. In Appendix D, we give an illustrative example on the WMT'19 Zh  En test set to show the improvements of our model. This line of research aims at modelling target-side global context in the reverse direction with an auxiliary model. Liu et al. (2016) first adopt L2R and R2L NMT models to independently generate translations through beam search, then re-rank the candidate list via their agreement. Zhang et al. (2018) employ a backward decoder to capture reverse target-side context, which is then exploited by the forward decoder. Serdyuk et al. (2018) pro-2885 pose twin networks, where the forward network is encouraged to generate hidden states similar to those of the backward network. Zhang et al. (2019a) present a future-aware knowledge distillation framework enabling the unidirectional decoder to explore the future context for word predictions. Zhou et al. (2019) propose a synchronous bidirectional NMT model with revised beam search algorithm that involves interactive L2R and R2L decodings. Zhang et al. (2019b) also combine the L2R and R2L NMT models by considering their agreement, helping the model to generate sentences with better prefixes and suffixes. Although these methods indeed gain some improvements, the modelling of reverse global context is independent of the local context of preceding words. Meanwhile, they usually rely on elaborately designed mechanisms for burdensome multi-pass decoding. Another line of research is to exploit global contextual information contained in large-scale pre-trained language models (PLM) via knowledge distillation (KD). For example, Edunov et al. (2019) and Conneau and Lample (2019) feed the top-layer representations of ELMo or BERT to NMT encoders. Yang et al. (2020) explore three techniques to apply BERT on NMT models, namely asymptotic distillation, dynamic switch for knowledge fusion, and rate-scheduled updating. Weng et al. (2020) propose a training framework consisting of a dynamic fusion mechanism and a continuous KD paradigm to leverage the knowledge of various PLMs. Baziotis et al. (2020) incorporate a language model prior for low-resource NMT. Chen et al. (2020) fine-tune the BERT on the parallel corpus to make it aware of source input, and then utilize it to improve the NMT model via KD over all target words. Compared to this, CBBGCA jointly optimizes the NMT model and auxiliary model, leading to better performance. Moreover, our method just selectively conducts KD on a portion of target words, giving higher distillation efficiency, which is similar to (Wang et al., 2021). Even though these PLM-based approaches have gained remarkable improvements, they unavoidably have some inherent limitations: (1) the monolingual PLMs lacks crucial bilingual information for translation; (2) the independence between PLM pre-trainings and NMT model training. In contrast, our proposed model is able to overcome these limitations. In this paper, we propose a CBBGCA training framework for the NMT model to effectively exploit target-side bidirectional global context with an auxiliary CMLM. The training consists of two stages. At the first stage, we introduce multi-task learning to benefit the NMT model by sharing its encoder with an auxiliary CMLM. Then, at the second stage, through confidence based knowledge distillation, we use the CMLM as teacher to especially refine the NMT model on unconfidently-predicted target words. Experimental results show that our framework can significantly improve the NMT model. Compared with previous work, neither external nor synthetic data are needed and only the NMT model is involved during inference. The project was supported by National Natural Science Foundation of China (No. 62036004, No. 61672440), Natural Science Foundation of Fujian Province of China (No. 2020J06001), and Youth Innovation Fund of Xiamen (No. 3502Z20206059). We also thank the reviewers for their insightful comments.
abstain abstain objective abstain abstain method objective abstain method abstain abstain other abstain abstain abstain abstain other abstain other abstain abstain abstain abstain abstain result abstain method objective method abstain objective abstain method method objective objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain method abstain abstain other method abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain method method abstain abstain abstain other other other other other other other other other other other other other other other other method other objective objective abstain objective method result abstain other other