Knowledge distillation (KD) is the preliminary step for training non-autoregressive translation (NAT) models, which eases the training of NAT models at the cost of losing important information for translating low-frequency words. In this work, we provide an appealing alternative for NAT  monolingual KD , which trains NAT student on external monolingual data with AT teacher trained on the original bilingual data. Monolingual KD is able to transfer both the knowledge of the original bilingual data (implicitly encoded in the trained AT teacher model) and that of the new monolingual data to the NAT student model. Extensive experiments on eight WMT benchmarks over two advanced NAT models show that monolingual KD consistently outperforms the standard KD by improving low-frequency word translation, without introducing any computational cost. Monolingual KD enjoys desirable expandability, which can be further enhanced (when given more computational budget) by combining with the standard KD, a reverse monolingual KD, or enlarging the scale of monolingual data. Extensive analyses demonstrate that these techniques can be used together profitably to further recall the useful information lost in the standard KD. Encouragingly, combining with standard KD, our approach achieves 30.4 and 34.1 BLEU points on the WMT14 English-German and German-English datasets, respectively. Our code and trained models are freely available at https://github.com/ alphadl/RLFW-NAT.mono . Non-autoregressive translation (NAT, Gu et al. 2018) has been proposed to improve the decoding efficiency by predicting all tokens independently and simultaneously. However, the independence assumption prevents a model from properly capturing the highly multimodal distribution of target translations. In response to this problem, a sequence-level knowledge distillation (KD, Kim and Rush 2016) becomes the preliminary step for training NAT models, which produces more deterministic knowledge by reducing the translation modes of the bilingual data (Zhou et al., 2020). Although the standard KD on original bilingual data eases the training of NAT models, distillation may lose some important information in the raw training data, leading to more errors on predicting low-frequency words (Ding et al., 2021c,b). To remedy this problem, Ding et al. (2021c) augmented NAT models the ability to learn lost knowledge from the raw bilingual data with an additional objective, and Ding et al. (2021b) first pre-trained NAT models on the raw training data and then fine-tuned them on the distilled training data. While previous studies mainly focus on recalling the lost information during the distillation of the original bilingual data, in this work we propose to improve the prediction of low-frequency words by redistributing them in the external monolingual data, which has the great potential to complement the original bilingual data on the word distribution. Specifically, we leverage the monolingual data to perform KD ( monolingual KD , 2.2), and train the NAT student model on the distilled monolingual data (Figure 1b). Monolingual KD provides appealing benefits. Firstly, the monolingual data and bilingual data in machine translation are generally complementary to each other (Zhang and Zong, 2016; Wu et al., 2019; Zhou and Keung, 2020; Sid-dhant et al., 2020; Jiao et al., 2021). Accordingly, monolingual KD is able to transfer both the knowledge of the bilingual data (implicitly encoded in the trained teacher model) and that of the monolingual data to the NAT student, without introducing additional computational cost. Secondly, the amount 2417 of available monolingual data is several orders of magnitude larger than that of bilingual data, which offers monolingual KD the potential to further improve translation performance by exploiting more monolingual data. Furthermore, we analyze the bilingual links in the bilingual and monolingual distilled data from two alignment directions (i.e. source-to-target and target-to-source). We found that the monolingual KD makes low-frequency source words aligned with targets more deterministically compared to bilingual KD, but both of them fail to align low-frequency words from target to source due to information loss. Starting from this finding, we propose reverse monolingual KD to recall more alignments for low-frequency target words. We then concatenate two kinds of monolingual distilled data ( bidirectional monolingual KD , 2.3) to maintain advantages of deterministic knowledge and low-frequency information. We validated our approach on several translation benchmarks across scales (WMT14 En  De, WMT16 Ro  En, WMT17 Zh  En, and WMT19 En  De) over two advanced NAT models: Mask Predict (Ghazvininejad et al., 2019) and Levenshtein (Gu et al., 2019). Experiments demonstrate the effectiveness and universality of our approach. Specifically, we have the following findings:  Monolingual KD achieves better performance than the standard KD in all cases, and the proposed bidirectional monolingual KD can further improve performance by a large margin. Monolingual KD enjoys appealing expandability: enlarging the scale of monolingual data consistently improves performance until reaching the bottleneck of model capacity. Monolingual KD is complementary to the standard KD, and combining them obtains further improvement by alleviating two key issues of NAT, i.e., the multimodality problem and the low-frequency word translation problem. The paper is an early step in exploring monolingual KD for NAT, which can narrow the performance gap between NAT models and the SOTA AT models. We hope the promising effect of monolingual KD on NAT can draw more interest and can make NAT a common translation framework. Non-Autoregressive Translation Recent years have seen a surge of interest in NAT (Gu et al., 2018), which can improve the decoding efficiency by predicting all tokens independently and simultaneously. Specifically, the probability of generating a target sentence y by given the source sentence x is computed as p ( y | x ) = p L ( T | x ;  ) (cid:81) Tt =1 p ( y t | x ;  ) , where T is the length of y , which is predicted by a separate conditional distribution p L (  ) . The parameters  are trained to maximize the likelihood of a set of training examples according to L (  ) = arg max  log p ( y | x ;  ) . The conditional independence assumption prevents an NAT model from properly capturing the highly multimodal distribution of target translations ( multimodality problem , Gu et al., 2018). As a result, the translation quality of NAT models often lags behind that of AT models (Vaswani et al., 2017). Standard Knowledge Distillation Knowledge distillation is the preliminary step for training NAT models by reducing the modes in the original bilingual data, which makes NAT easily acquire more deterministic knowledge and achieve significant improvement (Zhou et al., 2020). Typically, a sequence-level KD (Kim and Rush, 2016) is employed for NAT training, as shown in Figure 1a. Different Distributions of Source Words To empirically reveal the difference on word distribution between bilingual and monolingual data, we visualize the overall word distributions, as plotted in Figure 2. We can observe the significant difference between bilingual and monolingual data in the low-frequency part, which indicates that the words that occur less in the bilingual data are not necessarily low-frequent in the external monolingual data. Starting from the observation, we propose to exploit external monolingual data to offer more useful information for predicting low-frequent words in bilingual data, which are generally lost in the standard knowledge distillation. Our Approach Researches and competitions have shown that fully exploiting the monolingual data is at the core of achieving better generalization and accuracy for MT systems (Sennrich et al., 2016a; Zhang and Zong, 2016; Barrault et al., 2418 src tgt NAT Student Parallel Data Synthetic Data src tgt AT Teacher train src distill train NAT Student AT Teacher train Monolingual Data reuse src tgt Parallel Data train src tgt Synthetic Data src distill 2020). In this work we want to transfer the distribution of lost information, e.g. low-frequency words, from monolingual data to the NAT training. Figure 1b shows the pipeline of our proposed Monolingual KD for NAT, which differs from the Standard KD at how to construct the distilled data. Instead of reusing the source side of the original bilingual data, monolingual KD performs distillation on newly monolingual data, which eliminates the dependency on the original training data. Intuitively, the monolingual KD can embed both the knowledge of the original bilingual data (im-plicitly encoded in the trained teacher model) and that of the newly introduced monolingual data. The comprehensive experiments in the following section provide empirical support for our hypothesis. In addition, the complementarity between the bilingual and monolingual data makes explicitly combining Standard KD and Monlingual KD can further improve model performance. Recalling Low-Frequency Target Words KD simplifies the training data by replacing low-frequency target words with high-frequency ones (Zhou et al., 2020; Ding et al., 2021c). This is able to facilitate easier aligning source words to target ones, resulting in high bilingual coverage (Jiao et al., 2020). Inspired by the low-frequency word (LFW) links analysis (Ding et al., 2021b), we borrow this LFW analysis to show the necessity of leveraging both the sourceand target-side monolingual data. Concretely, we follow (Ding et al., 2021b) to evaluate the links of low-frequency words aligning from source to target (s (cid:55) t) with three metrics: Recall (R) represents how many low-frequency source words can be aligned to targets; Precision (P) means how many aligned low-frequency links are correct ac-2419 cording to human evaluation. F1 is the harmonic mean between precision and recall. Similarly, we can analyze in an opposite direction (t (cid:55) s) by considering the links of low-frequency target words. Table 1 lists the results. Comparing with the standard  KDB , the forward monolingual KD (  KDM in Section 2.2) achieves better alignment quality of s (cid:55) t LFW links (F1: 80.9 vs. 80.5) by aligning more low-frequency source words (R: 75.1 vs. 73.4). The backward monolingual KD (  KDM ) can complementarily produce better alignment of low-frequency target words (t (cid:55) s LFW links). As we expected, combining the two types of distilled data (  KDM ) can produce better alignments for both low-frequency source (F1: 82.1 vs. 80.5) and target words (F1: 79.9 vs. 74.2). Our Approach ( Bid. Monolingual KD ) Based on the above observations, we propose to train NAT models on bidirectional monolingual data by concatenating two kinds of distilled data. Like back-translation (Edunov et al., 2018), the reverse monolingual distillation  KDM is to synthesize the source sentences by a backward AT teacher, which is trained in the reverse direction of the original bilingual data. The mixture of the source-original and target-original synthetic datasets (i.e.  KDM ) is used to train the final NAT model. We expect that the better alignments of LFW links can lead to overall improvement of translation performance. Bilingual Data We conducted experiments on two widely-used NAT benchmarks: WMT14 English-German and WMT16 English-Romanian tasks, which consist of 4.5M and 0.6M sentence pairs respectively. To prove the universality of our approach on large-scale data, we also validated on WMT17 English-Chinese and WMT19 English-German tasks, which consist of 20.6M and 36.8M sentence pairs respectively. We shared the source and target vocabularies, except for En  Zh data. We split the training data into subword units using byte pair encoding (BPE) (Sennrich et al., 2016b) with 32K merge operations, forming a vocabulary of 37k, 32k, 33k/48k and 44k for WMT14 En  De, WMT16 En  Ro, WMT17 En  Zh and WMT19 En  De respectively. We used case-sensitive token-BLEU (Papineni et al., 2002) to measure the translation quality (except for En-Zh, we used sacre-T a s k Lang. Monolingual Data We closely followed previous works to randomly sample monolingual data from publicly available News Crawl corpus 1 for the WMT tasks (Sennrich et al., 2016a; Wu et al., 2019). We randomly sampled English and German data from News Crawl 2007  2020, and randomly sampled Romanian data from News Crawl 2015. For Chinese monolingual data, we used News Crawl 2008  2020, News Commendary v16 and XMU data. For fair comparison, the monolingual data generally has the same size as corresponding bilingual data, as listed in Table 2. two state-of-the-art NAT models: MaskPredict [MaskT, Ghazvininejad et al. 2019] that uses the conditional masked language model (Devlin et al., 2019) to iteratively generate the target sequence from the masked input. We followed its optimal settings to keep the iteration number be 10 and length beam be 5. Levenshtein Transformer [LevT, Gu et al. 2019] that introduces three steps: deletion, placeholder prediction and token prediction, and the decoding iterations adaptively depends on certain conditions. We followed their setting and reproduced their reported results. We trained both BASE and BIG Transformer (Vaswani et al., 2017) as the AT teachers for both standard and monolingual KD. For BIG models, we adopted large-batch training (i.e. 458K to-1 http://data.statmt.org/news-crawl 2420 Data MaskT LevT BLEU (cid:52) BLEU (cid:52)  KDB 25.4  25.6   KDM 25.8 +0.4 26.2 +0.6  KDM 24.9 -0.5 24.5 -1.1  KDM 26.6 +1.2 26.7 +1.1  KDM +  KDB 26.7 +1.3 26.8 +1.2  KDM +  KDB 26.6 +1.2 26.5 +0.9  KDM +  KDB 27.1 +1.7 27.3 +1.7 Table 3: BLEU scores of different monolingual distillation strategies. kens/batch) to optimize the performance (Ott et al., 2018). The En  Ro tasks employed Transformer-B ASE as the teacher, and the other tasks used Transformer-B IG as the teacher. We also used large-batch (i.e. 480K tokens/batch) to train NAT models with Adam optimizer (Kingma and Ba, 2015). The learning rate warms up to 1  10  7 for 10K steps, and then decays for 60k steps with the cosine schedule (Ro  En models only need 4K and 21K steps, respectively). Following the common practices (Ghazvininejad et al., 2019; Kasai et al., 2020), we evaluate the performance on an ensemble of 5 best checkpoints (ranked by validation BLEU) to avoid stochasticity. In this section, we evaluated the impact of different components of the monolingual KD on WMT14 En-De validation sets. Impact of Distillation Strategy Table 3 lists the results of different distillation strategies. The forward monolingual KD (  KDM ) consistently outperforms its standard counterpart (  KDB ) (i.e. 25.8 vs. 25.4, and 26.2 vs. 25.6), which we attribute to the advantage of monolingual KD on exploiting both the original bilingual data knowledge (implicitly encoded in the trained AT teacher model) and the new monolingual data knowledge. Concatenating forwardand reverse-KD (  KDM ) can further improve the NAT performance, which is consistent with the findings in Table 1. is complementary to standard KD (i.e. +  KDB  column). As seen, standard KD consistently improves translation performance across monolingual KD variants. Another interesting finding is that although reverse monolingual KD (  KDM ) significantly underperforms its forward counterpart (  KDM ) when used alone, they achieve comparable performance when using together with standard KD. We discuss in details how the two KD models complement each other in Section 3.4. Impact of Monolingual Data Sampling Some researchers may doubt that our approach heavily depends on the sampled monolingual data. To dispel the doubt, we investigated whether our model is robust to the selected monolingual data by varying the sampling strategies. Specifically, we conducted experiments on the full set of monolingual data from News Crawl 2007  2020, which consist of 243M English and 351M German sentences. We compared with two representative approaches that sampled data with different priors: (1) LOWFREQ samples difficult examples containing low-frequency words (Fadaee and Monz, 2018); (2) LM-SEL selects high quality examples with language model (Moore and Lewis, 2010). As listed in Table 4, the difference of three sampling strategies w.r.t BLEU is not significant under the significance test p < 0 . 05 (Collins et al., 2005), demonstrating that our approach is robust to the monolingual data sampling . For the simplicity and robust applicability of our approach across different scenarios, we used RANDOM sampling as the default strategy in the following experiments. NAT Benchmarks Table 5 lists the results on the WMT14 En  De and WMT16 En  Ro benchmarks. Encouragingly, the conclusions in Section 3.2 hold across language pairs, demonstrating the effectiveness and universality of our approach. We also compared the performance against several 2421 Model Iter. previous competitive NAT models. Although the results are not directly comparable since we used additional monolingual data, our approach improves previous SOTA BLEU on the NAT benchmarks. Notably, our data-level approaches neither modify model architecture nor add extra training loss, thus does not increase any latency (Speed), maintaining the intrinsic advantages of NAT models. The main side-effect of our approach is the increased training time for training an additional AT teacher model to build distilled data in the reverse direction. Fortunately, we can eliminate the side-effect by using only the monolingual KD (Mono. KD), which still consistently outperforms the standard KD without introducing any computation cost. Larger-Scale WMT Benchmarks To verify the effectiveness of our method across different data sizes, we further experimented on two widely-used large-scale MT benchmarks, i.e. WMT17 En  Zh and WMT19 En  De. As listed in Table 6, our bidi-Model En-Zh En-De     AT Teacher 35.6 24.6 40.2 40.1 MaskT +Stand. rectional monolingual KD outperforms standard KD by averagely +1.9 and +2.3 BLEU points on En  Zh and En  De datasets, respectively, demonstrating the robustness and effectiveness of our monolingual KD approach. By combining with standard KD, our methods can achieve further +1.8 and +0.9 BLEU improvements. In this section, we provide some insights into how monolingual KD works. We report the results on WMT14 En-De data using Mask-Predict. Monolingual KD Reduces Complexity of Training Data by Improving Low-Frequency Word Alignment We first present data-level qualitative analyses to study how monolingual KD complements bilingual KD. Zhou et al. (2020) revealed that standard KD improves NAT models by reducing the complexity of original bilingual data. Along this thread, we used the data complexity metric to measure different distilled datasets. Formally, the translation uncertainty of a source sentence x can be operationalized as conditional entropy: H ( Y | X = x) =  (cid:88) y  Y p (y | x) log p (y | x)  T x (cid:88) t =1 H ( y | x = x t ) , where T x denotes the length of the source sentence, x and y represent a word in the source and target vocabularies, respectively. We run fast-align on each parallel corpus to obtain word alignment. For fair comparison, we sampled the subsets (i.e. 4.5M) of   KDM  and   KDM +  KDB  to perform complexity computation. As seen in Table 7, standard KD significantly reduces the data complexity compared to that of the Data WMT14 En-De WMT14 De-En H M L H M L AT Teacher Raw Data 84.7 80.2 73.0 85.4 81.1 74.2 NAT Student  KDB 82.4 78.2 68.4 83.7 79.6 69.9  KDM 82.9 78.4 69.5 83.9 80.1 71.2 +  KDB 83.1 78.7 70.8 84.3 80.5 72.1  KDM 84.1 79.1 72.7 85.0 80.9 73.4 +  KDB 84.6 79.7 73.6 85.2 81.4 75.2 Table 8: Accuracy of word translation. bilingual data (1.95 vs. 3.67), and monolingual KD reduces even more data complexity. Additionally, the data complexity can be further reduced by combining with standard KD. Monolingual KD Mainly Improves Low-Frequency Word Translation We first followed Ding et al. (2021c) to measure the translation accuracy of words with different frequencies, as shown in Table 8. The improvements over low-frequency words are the major reason for the performance gains, where the monolingual KD and bidirectional monolingual KD outperform the standard KD by averagely +1.2% and +3.9%, respectively. These findings confirm our hypothesis that monolingual KD can improve the translation of low-frequency words by redistributing them in the new monolingual data. Combining with standard KD can further improve the accuracy of translating low-frequency words, which reconfirms our hypothesis on the complementarity between the two KD methods on low-frequency words. In this section, we provide some potential directions to further improve NAT performance by making the most of monolingual data. Exploiting Monolingual Data at Scale One strength of monolingual KD is the potential to exploit more monolingual data to further improve translation performance. To validate our claim, we scaled the size of monolingual data by { 2  , 5  , 10  }, which are randomly sampled from the full set of monolingual data. As shown in Table 9, 2423 Mono WMT14 En-De WMT14 De-En Size MaskT LevT MaskT LevT Bidirectional Monolingual KD 1  29.1 29.5 32.6 33.6 2  29.7 30.1 33.1 33.9 5  30.6 30.9 33.9 34.5 10  30.4 30.8 33.3 34.4 Combining with Standard KD 1  30.1 30.4 33.7 34.1 2  30.7 30.9 34.2 34.5 5  31.3 31.7 34.5 34.7 10  30.9 31.5 34.2 34.6 Table 9: BLEU scores of using monolingual data at scale. enlarging the monolingual data consistently improves the BLEU scores, while this trend does not hold when further scaling the monolingual data (i.e. 10  ). One possible reason is that the limited capacity of NAT-base models cannot fully exploit the large data, which suggests future exploration of larger NAT architectures. Augmenting AT Teacher with Monolingual KD An alternative to exploit monolingual data is to strength the AT teacher with monolingual KD, as listed in Table 10. Applying monolingual KD for AT teacher is less effective than using it for NAT training, which we attribute to the information loss when transferred from AT teacher to NAT student. Applying monolingual KD to both AT teacher and NAT student can further improve the NAT performance, at the cost of more computational cost. To bridge the performance gap, a number of recent efforts have explored, including model architectures (Ghazvininejad et al., 2019; Gu et al., 2019; Ding et al., 2020; Guo et al., 2020), training objectives and methods (Shao et al., 2019; Ghazvininejad et al., 2020; Ding et al., 2021a). Another thread of work focus on understanding and improving distillation training for NAT (Zhou et al., 2020; Ding et al., 2021c,b; Huang et al., 2022). Sequence-level KD (Kim and Rush, 2016) is a preliminary step for training NAT models to reduce the intrinsic uncertainty and learning diffi-culty (Zhou et al., 2020; Ren et al., 2020). Recent studies have revealed that KD reduces the modes (i.e. multiple lexical choices for a source word) in the original data by re-weighting the training examples (Furlanello et al., 2018; Tang et al., 2020), at the cost of losing some important information, leading to more errors on predicting low-frequency words (Ding et al., 2021c). In response to this problem, Ding et al. (2021b) proposed to rejuvenate low-frequency words by pretraining NAT models on the raw bilingual data. In this study, we attempt to solve this problem from a different perspective  rediscovering low-frequency words from external monolingual data, which can simultaneously exploit the knowledge of bilingual data (implicitly encoded in the parameters of AT teacher). Closely related to our work, Zhou and Keung (2020) improved NAT models by augmenting source-side monolingual data. Their work can be regarded as a special case of our approach (i.e.  Mono. KD + Standard KD  in Section 3.3), and our work has several more contributions. Firstly, we demonstrated the effectiveness of using only monolingual KD for NAT models, which can achieve better performance than the standard KD without introducing any computational cost. Secondly, we proposed a novel bidirectional monolingual KD to exploit both the source-side and target-side monolingual data. Finally, we provide insights into how monolingual KD complements the standard KD. In this work, we propose a simple, effective and scalable approach  monolingual KD to redistribute the low-frequency words in the bilingual data using external monolingual data. Monolingual KD consistently outperforms the standard KD with more translation accuracy of low-frequency words, 2424 which attribute to its strength of exploiting both the knowledge of the original bilingual data (implicitly encoded in the parameters of AT teacher) and that of the new monolingual data. Monolingual KD enjoys appealing expandability, and can be further enhanced by (1) combining with a reverse monolingual KD to recall more alignments for low-frequency target words; (2) combining with the standard KD to explicitly combine both types of complementary knowledge; (3) enlarging the scale of monolingual data that is cheap to acquire. Our study empirically indicates the potential to make NAT a practical translation system. Future directions include designing advanced monolingual KD techniques and validating on larger-capacity NAT models (e.g. BIG setting) to strengthen the power of monolingual KD, and fully NAT models (Gu and Kong, 2021; Du et al., 2021) to show the universality of monolingual KD. Besides, it will be interesting to follow Liu et al. (2021) and Wang et al. (2022) to investigate the complementarity between our monolingual KD and pretrained language models to further enhance the NAT models. We are grateful to the anonymous reviewers and the area chair for their insightful comments and suggestions.
abstain method abstain abstain abstain abstain result other abstain abstain abstain abstain abstain abstain objective method abstain abstain abstain abstain method result objective abstain abstain objective objective objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain method abstain abstain abstain abstain method abstain abstain abstain abstain method method method method method method method method abstain abstain method abstain method method other abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result result abstain abstain result abstain result abstain abstain abstain other result abstain other abstain abstain abstain abstain abstain result result abstain abstain other abstain abstain abstain abstain abstain other other other other other objective abstain objective objective objective method objective abstain abstain abstain abstain abstain other