Non-Autoregressive machine Translation (NAT) models have demonstrated significant inference speedup but suffer from inferior translation accuracy. The common practice to tackle the problem is transferring the Autoregressive machine Translation (AT) knowledge to NAT models, e.g., with knowledge distillation. In this work, we hypothesize and empirically verify that AT and NAT encoders capture different linguistic properties of source sentences. Therefore, we propose to adopt multi-task learning to transfer the AT knowledge to NAT models through encoder sharing. Specifically, we take the AT model as an auxiliary task to enhance NAT model performance. Experimental results on WMT14 English  German and WMT16 English  Romanian datasets show that the proposed MULTI-TASKNAT achieves significant improvements over the baseline NAT models. Furthermore, the performance on large-scale WMT19 and WMT20 English  German datasets confirm the consistency of our proposed method. In addition, experimental results demonstrate that our MULTI-TASKNAT is complementary to knowledge distillation, the standard knowledge transfer method for NAT. 1 1 Introduction Neural machine translation (NMT), as the state-of-the-art machine translation paradigm, has recently been approached with two different sequence decoding strategies. The first type autoregressive  The first two authors contributed equally to this work. translation (AT) models generate output tokens one by one following the left to right direction (Vaswani et al., 2017; Bahdanau et al., 2015), but it is often criticized for its slow inference speed (Gu et al., 2018). The second type non-autoregressive translation (NAT) models adopt a parallel decoding algorithm to produce output tokens simultaneously (Gu et al., 2019; Ghazvininejad et al., 2019; Ma et al., 2020), but the translation quality of it is often inferior to auto-regressive models (Gu et al., 2018). Many researchers have investigated the collaboration between AT and NAT models. For instance, ENCODER-NAD-AD (Zhou et al., 2020) leverages NAT models to improve the performance of AT. Specifically, their method inserts a NAT decoder between the conventional AT encoder and decoder to generate coarse target sequences for the final autoregressive decoding. A line of research (Wang et al., 2019b; Guo et al., 2020; Ding et al., 2020) holds the opinion that the lack of contextual dependency on target sentences potentially leads to the deteriorated performance of NAT models. To boost the NAT translation performance, many re-cent works resort to the knowledge transfer from a well-trained AT model. Typical knowledge transfer methods include sequence-level knowledge distillation with translation outputs generated by strong AT models (Gu et al., 2019; Ghazvininejad et al., 2019), word-level knowledge distillation with AT decoder representations (Wei et al., 2019; Li et al., 2019), and fine-tuning on AT model by curriculum learning (Guo et al., 2020), etc. In this work, we first verify our our hypothesis that AT and NAT encoders  although they belong to the same sequence-to-sequence learning task  capture different linguistic properties of source sentences. We conduct our verification by evaluating the encoder on a set of probing tasks (Conneau Task AT NAT Surface SeLen 91.7 93.4 WC 76.0 79.1 Syntactic TrDep 45.8 46.0 ToCo 78.3 79.7 BShif 74.8 73.4 Semantic Tense 89.2 89.2 SubN 86.2 87.5 ObjN 85.2 85.3 SoMo 54.0 53.0 CoIn 64.9 62.8 Table 1: Performance on the probing tasks of evaluating linguistic properties embedded in the learned representations of AT and NAT models. et al., 2018; Raganato and Tiedemann, 2018) for AT and NAT models. Further, by leveraging the linguistic differences, we then adopt a multi-task learning framework with a shared encoder (i.e., MULTITASKNAT) to transfer the AT model knowledge into the NAT model. Specifically, we employ an additional AT task as the auxiliary task of which the encoder parameters are shared with the NAT task while parameters of the decoder are exclusive. Since many works (Cipolla et al., 2018; Liu et al., 2019) suggest that the weights for each task are critical to the multi-task learning, in this work, the multi-task weight assigned to the AT task is dynamically annealed from 1 to 0 . We name this scheme importance annealing . We empirically show the benefit of importance annealing in both directions of the original WMT14 English  German dataset. Further with knowledge distillation, our proposed MULTI-TASKNAT achieves significant improvements on WMT14 English  German and WMT16 English  Romanian datasets. This con-firms the effectiveness of our proposed model on machine translation tasks. We propose a multi-task learning framework to boost NAT translation quality by transferring the AT knowledge to the NAT model. Our analyses reveal that the encoder sharing is necessary for capturing more linguistic and semantic information. Experiments on standard benchmark datasets demonstrate the effectiveness of the proposed MULTI-TASKNAT. To verify our hypothesis that AT and NAT encoders capture different linguistic properties of source sentences and can thereby complement each other, we probe the linguistic knowledge (Conneau et al., 2018) that embedded in the AT and NAT encoders on a set of tasks to investigate to what extent an encoder captures the linguistic properties. We present the detail for each probing tasks in Appendix B. Moreover, in Appendix C, we also provide a qualitative investigation to capture the difference between high-dimensional representations of AT and NAT encoders from another perspective. The AT and NAT models referred to in the following experiments are TRANSFORMER and MASK-PREDICT . We train the models on the WMT14 English  German dataset, and the details of the experiments are introduced in the Appendix. Probing Tasks Probing tasks (Conneau et al., 2018) can quantitatively measure the linguistic knowledge embedded in the model representation. We follow Wang et al. (2019a) to set model config-urations. The experimental results are depicted in Table 1. Table 1 shows the AT and NAT encoders capture different linguistic properties of source sentences. We observe that on average, the NAT model captures more surface features but less semantic features than the AT model. For example, on the sentence length prediction (SeLen) task, NAT models significantly outperform AT models since the sentence length prediction is a key component in NAT models. However, for sentence modification (SoMo) and coordinate clauses invertion (CoIn) tasks, the AT model outperforms the NAT model by a large margin. The linguistic probing results reveal that AT and NAT models capture different linguistic properties, which thereby leaves space for the encoder sharing structure. In this section, we introduce that our shared encoder structure between AT and NAT models under the multi-task learning framework. Multi-Task NAT Given the AT and NAT models under the standard encoder-decoder structure, we employ the hard parameter sharing method (Ruder, 2017) to share their encoder parameters. Therefore, as shown in Figure 1, the proposed model MULTI-TASKNAT consists of three com-Figure 1: The architecture of our proposed model. We introduce an extra AT decoder to boost the training at the beginning, and gradually lower the importance weight of AT task by increasing  . ponents: shared encoder, AT decoder, and NAT decoder. Their parameters are jointly optimized towards minimizing the multi-task loss function, as introduced in the next section. Multi-Task Framework The loss function of the proposed MULTI-TASKNAT L at iteration step t is defined as the weighted sum of AT loss and NAT loss: L =  t L nat (cid:0) X, Y ;  enc ,  natdec (cid:1) + (1   t ) L at (cid:0) X, Y ;  enc ,  atdec (cid:1) (1) where L at and L nat are AT loss and NAT loss. enc ,  natdec , and  atdec are parameters of the shared encoder, NAT decoder, and AT decoder respectively. t is the importance factor to balance the preference between the AT and NAT models at time step t as illustrated bellow. Importance Annealing The term L at only serves as an auxiliary and does not directly affect the inference of NAT. Therefore, we intuitively determine to lower the importance of the AT loss when the training process is close to the ending, which we named importance annealing . Formally, we set  t = t T where T is the total steps of training. Under such a scheme, the weight for L at is linearly annealed from 1 . 0 to 0 . 0 along the training process, while the weight for L nat is increased from 0 . 0 to 1 . 0 . Training and Inference During the model training with training pairs ( X, Y ) , we feed the source sentence X to the encoder and the target sentence Y to two decoders separately. The target sentence Model WMT14 En  De De  En MASK-PREDICT 1 24.61  TRANSFORMER-LEV 2 25.20  MASK-PREDICT 3 24.70 29.52 MULTI-TASKNAT 25.66 30.09 + Importance Annealing 25.79 30.32 1 Ghazvininejad et al. (2019); 2 Gu et al. (2019); 3 Our implementation. Y can be either the target sentence in the raw training data (4.1) or the generated target sentence with knowledge distillation (4.2). During the model inference, we only use the NAT decoder to generate the target tokens simultaneously while ignoring the AT decoder. Therefore, the inference overhead is the same as the NAT model before sharing. We conducted experiments on two widely used WMT14 English  German and WMT16 English  Romanian benchmark datasets, which consist of 4.5M and 610K sentence pairs, respectively. We applied BPE (Sennrich et al., 2016) with 32K merge operations for both language pairs. The experimental results are evaluated in case-sensitive BLEU score (Papineni et al., 2002). We use TRANSFORMER (Vaswani et al., 2017) as our baseline autoregressive translation model and the MASK-PREDICT (Ghazvininejad et al., 2019) as our baseline non-autoregressive model. We integrate the TRANSFORMER decoder into the MASK-PREDICT to implement the proposed MULTI-TASKNAT model. For  t , we use the annealing scheme described in Section 3. Since the major NAT architecture of our method is exactly the MASK-PREDICT model, any established decoding latency results (Kasai et al., 2021) for MASK-PREDICT can also be applied to ours. All of the parameters are randomly initialized for a fair comparison with the MASK-PREDICT . More training details are introduced in Appendix A. 4.1 Ablation Study Table 2 shows that the performance of our MULTITASKNAT model and baseline models on WMT14 Model WMT14 WMT16 En  De De  En En  Ro Ro  En Baseline Models Transformer (Ghazvininejad et al., 2019) 27.74 31.09 34.28 33.99 Hint-based NAT (Li et al., 2019) 25.20 29.52   NAT-REG (Wang et al., 2019b) 24.61 28.90   FCL-NAT (Guo et al., 2020) 25.75 29.50   Levenshtein Transformer (Gu et al., 2019) 27.27   33.26 Mask-Predict (Ghazvininejad et al., 2019) 27.03 30.53 33.08 33.31 Mask-Predict w/ Raw Data Prior (Ding et al., 2021) 27.8   33.7 Our Experiments Mask-Predict 27.18 30.86 33.03 32.71 MULTI-TASKNAT (w/ IA) 27.98  31.27  33.80  33.60  Table 3: Evaluation of translation performance on WMT14 En  De and WMT16 En  Ro test sets. En  De datasets without using the knowledge distillation. The vanilla MULTI-TASKNAT model with the the  fixed as 0 . 5 outperforms the baseline MASK-PREDICT model by 0.96 and 0.57 BLEU score in En  De and De  En direction respectively and even surpasses the strong baseline TRANSFORMER-LEV by 0.46 BLEU points in En  De translation. With the importance annealing , the MULTI-TASKNAT model achieves slight but consistent improvements over the vanilla model (+ Importance Annealing  in Table 2). The improvements demonstrate the effectiveness of our proposed model using multi-task learning. We further evaluate the proposed MULTI-TASKNAT model with the standard practice of knowledge distillation. Table 3 depicts the performances of our model as well as strong baseline models. Our proposed MULTI-TASKNAT model achieves a significant improvement of 0.80 and 0.41 BLEU point over the strong baseline MASK-PREDICT model on En  De and De  En translation. On En  Ro translation, our model outperforms the baseline model by 0.77 and 0.89 BLEU scores respectively. We use the compare-mt (Neubig et al., 2019) 2 to determine the significance. Details for significance tests are described in Appendix A.4. We conduct probing tasks to empirically recon-firm our hypothesis in Section 2 and better understand our MULTI-TASKNAT in terms of linguistic properties. The results are presented in Table 4. In most of the cases, our MULTI-TASKNAT could learn better surface, syntactic, and semantic information than the TRANSFORMER and MASK-PREDICT baseline models, indicating that our multi-task learning framework can indeed take the advantages of two separate tasks and capture better linguistic properties. Notably, on the sen-Model WMT19 WMT20 En  De De  En En  De De  En MASK-PREDICT 34.79 37.04 25.24 36.36 MULTI-TASKNAT 35.38  37.62  25.72  36.58 Table 5: Evaluation of translation performance on WMT19 En  De and WMT20 En  De test sets with knowledge distillation. tence length (Selen) prediction task and tree depth (TrDep) task, the MULTI-TASKNAT shows significantly better performance. On other tasks, our model demonstrates better or on-par performance compared to the NAT model. Regarding the coordination inversion (CoIn) task, though the MULTITASKNAT shows certainly lower performance than the TRANSFORMER , it still outperforms the MASK-PREDICT by 0 . 5 . We conduct the larger-scale experiments on the WMT English  German. We adopt newstest2019 and newstest2020 as the test sets. The parallel data consists of about 36 . 8 M sentence pairs. We average the last 5 checkpoints as the final model. The results are listed in Table 5. The improvements suggest that our model are consistently effective on various scale of data. In this paper, we have presented a novel multitask learning approach for NAT model with a hard parameter sharing mechanism. Experimental results confirm the significant effect of the proposed MULTI-TASKNAT model, which shows the complementary effects of multi-task learning to the knowledge distillation method. Based on our MULTI-TASKNAT, there are many promising directions for future research. For example, 1) decoder interaction: knowledge distillation in an online fashion between AT and NAT decoders; 2) share-all framework: shared-encoder and shared-decoder with two decoding strategies, and the model can dynamically choose the optimal decoding strategy during model inference. 3) data manipulation strategies: such as data rejuvenation (Jiao et al., 2020), lexical frequency discrepancy (Ding et al., 2021). The authors sincerely thank Liang Ding for the advice of experiments settings, and the anonymous reviewers for their insightful suggestions on various aspects of this work.
abstain abstain objective objective method abstain objective objective abstain objective abstain abstain abstain abstain abstain abstain abstain other objective method method method abstain abstain result objective objective objective result objective abstain method abstain method abstain method abstain abstain abstain abstain abstain abstain method method method method abstain abstain other abstain abstain abstain method method abstain abstain abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain other