Recent evidence reveals that Neural Machine Translation (NMT) models with deeper neural networks can be more effective but are difficult to train. In this paper, we present a M ulti S cale C ollaborative (MSC ) framework to ease the training of NMT models that are substantially deeper than those used previously. We explicitly boost the gradient back-propagation from top to bottom levels by introducing a block-scale collaboration mechanism into deep NMT models. Then, instead of forcing the whole encoder stack directly learns a desired representation, we let each encoder block learns a fine-grained representation and enhance it by encoding spatial dependencies using a context-scale collaboration . We provide empirical evidence showing that the MSC nets are easy to optimize and can obtain improvements of translation quality from considerably increased depth. On IWSLT translation tasks with three translation directions, our extremely deep models (with 72-layer encoders) surpass strong baselines by +2.2  +3.1 BLEU points. In addition, our deep MSC achieves a BLEU score of 30.56 on WMT14 English  German task that significantly outperforms state-of-the-art deep NMT models. Neural machine translation (NMT) directly models the entire translation process using a large neural network and has gained rapid progress in recent years (Sutskever et al., 2014; Sennrich et al., 2016). The structure of NMT models has evolved quickly, such as RNN-based (Wu et al., 2016), CNN-based (Gehring et al., 2017) and attention-based (Vaswani et al., 2017) systems. All of these models follow the encoder-decoder framework with attention (Cho et al., 2014; Bahdanau et al., 2015; Luong et al., 2015) paradigm. Deep neural networks have revolutionized the state-of-the-art in various communities, from computer vision to natural language processing. However, training deep neural networks has been always a challenging problem. To encourage gradient flow and error propagation, researchers in the field of computer vision have proposed various approaches, such as residual connections (He et al., 2016), densely connected networks (Huang et al., 2017) and deep layer aggregation (Yu et al., 2018). In natural language processing, constructing deep architectures has shown effectiveness in language modeling, question answering, text clas-sification and natural language inference (Peters et al., 2018; Radford et al., 2018; Al-Rfou et al., 2019; Devlin et al., 2019). However, among existing NMT models, most of them are generally equipped with 4-8 encoder and decoder layers (Wu et al., 2016; Vaswani et al., 2017). Deep neural network has been explored relatively little in NMT. Recent evidence (Bapna et al., 2018; Wang et al., 2019a) shows that model depth is indeed of importance to NMT, but a degradation problem has been exposed: by simply stacking more layers, the translation quality gets saturated and then degrades rapidly. To address this problem, Bapna et al. (2018) proposed a transparent attention mechanism to ease the optimization of the models with deeper encoders. Wang et al. (2019a) continued this line of research but construct a much deeper encoder for Transformer by adopting the pre-norm method that establishes a direct way to propagate error gradients from the top layer to bottom levels, and passing the combination of previous layers to the next. While notable gains have been reported over shallow models, the improvements of translation quality are limited when the model depth is beyond 20. In addition, degeneration of translation quality is still observed when the depth is beyond 30. As a result, two questions arise naturally: How to break the limitation of depth in NMT models? and How to fully utilize the deeper structure to further improve the translation quality? In this paper, we address the degradation problem by proposing a M ulti S cale C ollaborative (MSC ) framework for constructing NMT models with very deep encoders. 1 In particular, the encoder and decoder of our model have the same number of blocks , each consisting of one or several stacked layers. Instead of relying on the whole encoder stack directly learns a desired representation, we let each encoder block learn a fine-grained representation and enhance it by encoding spatial dependences using a bottom-up network. For coordination, we attend each block of the decoder to both the corresponding representation of the encoder and the contextual representation with spatial dependences. This not only shortens the path of error propagation, but also helps to prevent the lower level information from being forgotten or diluted. We conduct extensive experiments on WMT and IWSLT translation tasks, covering three translation directions with varying data conditions. On IWSLT translation tasks, we show that:  While models with traditional stacking architecture exhibit worse performance on both training and validation data when depth increases, our framework is easy to optimize. The deep MSC nets (with 72-layer encoders) bring great improvements on translation quality from increased depth, producing results that substantially better than existing systems. On the WMT14 English  German task, we obtain improved results by deep MSC networks with a depth of 48 layers, outperforming strong baselines by +2.5 BLEU points, and also defeat state-of-the-art deep NMT models (Wu et al., 2019; Zhang et al., 2019a) with identical or less parameters. 2 2 Background Given a bilingual sentence pair ( x , y ) , an NMT model learns a set of parameters  by maximizing the log-likelihood P ( y | x ;  ) , which is typically 1 In our scenario, we mainly study the depth of encoders. The reason is similar in (Wang et al., 2019a): 1) encoders have a greater impact on performance than decoders; 2) increasing the depth of the decoder will significantly increase the complexity of inference. 2 MSC not only performs well on NMT but also is generalizable to other sequence-to-sequence generation tasks, such as abstractive summarization that is introduced in Appendix A. decomposed into the product of the conditional probability of each target word: P ( y | x ;  ) = (cid:81) T y t =1 P ( y t | y <t , x ;  ) , where T y is the length of sentence y , y <t is the partial translation that contains the target tokens before position t . An encoder-decoder framework is commonly adopted to model the conditional probability P ( y | x ;  ) , in which the encoder and decoder can be implemented as RNN (Wu et al., 2016), CNN (Gehring et al., 2017), or Self-Attention network (Vaswani et al., 2017). Despite variant types of NMT architectures, multiple-layer encoder and decoder are generally employed to perform the translation task, and residual connections (He et al., 2016) are naturally introduced among layers, as H l = LAYER (H l  1 ;  l ) + H l  1 , where H l is the output of the l -th layer, LAYER (  ) is the layer function and  l be the parameters. We take the state-of-the-art Transformer as our baseline model. Specifically, the encoder consists of a stack of L identical layers, each of which comprises two subcomponents: a self-attention mechanism followed by a feed-forward network. Layer normalization (Ba et al., 2016) is applied to the input of each subcomponent (i.e., pre-norm ) and a residual skip connection (He et al., 2016) adds each subcomponent's input to its output. Formally, O le = ATTN (Q le , K le , V le ;  le ) + H l  1 e , H le = FNN ( LN (O le );  le ) + O le , (1) where LN (  ) , ATTN (  ) and FFN (  ) are layer normalization, attention mechanism, and feed-forward networks with ReLU activation in between, respectively. { Q le , K le , V le } are query, key and value vectors that are transformed from the normalized ( l  1 )-th encoder layer LN (H l  1 e ) . The decoder is similar in structure to the encoder except that it includes a standard attention mechanism after each self-attention network, which attends to the output of the encoder stack H Le : O ld = ATTN (Q ld , K ld , V ld ;  ld ) + H l  1 d , S ld = ATTN ( LN (O ld ) , K Le , V Le ;  ld ) + O ld , H ld = FNN ( LN (S ld );  ld ) + C ld , (2) where { Q ld , K ld , V ld } are transformed from the normalized ( l  1 )-th decoder layer LN (H l  1 d ) and { K Le , V Le } are transformed from the top layer of the encoder. The top layer of the decoder H Ld is used to generate the final output sequence. In the BLOCK 1 LAYERBLOCK 2 LAYERSOURCEINPUTSBLOCK NLAYERBLOCK 1 LAYERBLOCK 2 BLAYERTARGETINPUTSBLOCK NBLAYEROUTPUTPROBABILITIES (cid:117) 1 M (cid:117) 2 M (cid:117) NM 0 CNC 2 C 1 C following sections, we simplify the equations as H le = F (H l  1 e ;  le ) + H l  1 e , H ld = G (H l  1 d , H Le ;  ld ) + H l  1 d , (3) for the encoder and decoder, respectively. As discussed by Wang et al. (2019a), applying layer normalization to the input of each subcomponent is the key to learning deep encoders, as it establishes a direct way to pass gradient from the top-most layer to bottom layers:  L  H le =  L  H Le  (1 + L  1 (cid:88) j = l  F (H je ;  j +1 e )  H le ) , (4) where L is the cross entropy loss. However, as pointed out by Wang et al. (2019a) that it can be difficult to deepen the encoder for better translation quality. We argue that as the right-most term in Eq. (4) approaches 0 for the lower levels of the encoder, the parameters of which cannot be suffi-ciently trained using the error gradient  L  H Le only. To solve this problem, we propose a novel approach to shorten the path of error propagation from L to bottom layers of the encoder. In this section, we introduce the details of the proposed approach, a M ulti S cale C ollaborative (MSC ) framework for constructing extremely deep NMT models. The framework of our method consists of two main components shown in Figure 1(a). First, a block-scale collaboration mechanism establishes shortcut connections from the lower levels of the encoder to the decoder (as described in 3.1), which is the key to training very deep NMT models. We give explanation by seeing the gradient propagation process. Second, we further enhance source representations with spatial dependencies by contextual collaboration , which is discussed in Section 3.2. An intuitive extension of naive stacking of layers is to group few stacked layers into a block . We suppose that the encoder and decoder of our model have the same number of blocks (i.e., N ). Each block of the encoder has M n ( n  { 1 , 2 , ..., N } ) identical layers, while each decoder block contains one layer. Thus, we can adjust the value of each M n flexibly to increase the depth of the encoder. Formally, for the n -th block of the encoder: B ne = BLOCK e (B n  1 e ) , (5) where BLOCK e (  ) is the block function, in which the layer function F (  ) is iterated M n times, i.e. B ne = H n,M n e , H n,le = F (H n,l  1 e ;  n,le ) + H n,l  1 e , H n, 0 e = B n  1 e , (6) where l  { 1 , 2 , ..., M n } , H n,le and  n,le are the representation and parameters of the l -th layer in the n -th block, respectively. Each block of the decoder attends to the corresponding encoder block. He et al. (2018) proposed a model that learns the hidden representations in two corresponding encoder and decoder layers as the same semantic level through layer-wise coordination and parameter sharing. Inspired by this, we focus on efficiently training extremely deep NMT models through directly attending decoder to the lower-level layers of the encoder, rather than only to the final representation of the encoder stack. The proposed block-scale collaboration (BSC ) mechanism can effectively boost gradient propagation from prediction loss to lower level encoder layers. For explaining this, see again Eq. (4), which explains the error back-propagation of pre-norm Transformer. Formally, we let L be the prediction loss. The differential of L with respect to the l -th layer in the n -th block H n,le can be calculated as: 3  L  H n,le =  L  B Ne   B Ne  H n,le +  L  B ne   B ne  H n,le =  L  B Ne  (1+ M n (cid:88) k = l +1  H n,ke  H n,le + N (cid:88) i = n +1 M i (cid:88) j =1  H i,je  H n,le ) (cid:124) (cid:123)(cid:122) (cid:125) ( a ) +  L  B ne  (1+ M n (cid:88) k = l +1  H n,ke  H n,le ) (cid:124) (cid:123)(cid:122) (cid:125) ( b ) , (8) where term ( a ) is equal to Eq. (4). In addition to the straightforward path  L  B Ne for parameter update from the top-most layer to lower ones, Eq. (8) also provides a complementary way to directly pass error gradient  L  B ne from top to bottom in the current block. Another benefit is that BSC shortens the length of gradient pass chain (i.e., M n (cid:28) L ). 2014) cell Q ( c , x ) , which maps a hidden state and an additional input x into a new hidden state: where E e is the embedding matrix of the source input x . The new state C n can be fused with each layer of the subsequent blocks in both the encoder and the decoder. Formally, B ne in Eq.(5) can be re-calculated in the following way: B ne = H n,M n e , H n,le = F (H n,l  1 e , C n  1 ;  n,le ) + H n,l  1 e , H n, 0 e = B n  1 e . (10) Similarly, for decoder, we have B nd = BLOCK d (B n  1 d , B ne ) = G (B n  1 d , B ne , C n ;  nd ) + B n  1 d . (11) The above design is inspired by multiscale RNNs (MRNN ) (Schmidhuber, 1992; El Hihi and Bengio, 1996; Koutnik et al., 2014; Chung et al., 2016), which encode temporal dependencies with different timescales. Unlike MRNN , our MSC enables each decoder block to attend to multi-granular source information with different space-scales, which helps to prevent the lower level information from being forgotten or diluted. Feature Fusion: We fuse the contextual representation with each layer of the encoder and decoder through attention. A detailed illustration of our algorithm is shown in Figure 1(b). In particular, the l -th layer of the n -th encoder block F (  ;  n,le ) , l  [1 , M n ] and n  [1 , N ] , O n,le = g e (cid:12) ATTN h (H n,l  1 e , H n,l  1 e , H n,l  1 e ;  n,le ) + (1  g e ) (cid:12) ATTN c (H n,l  1 e , C n  1 , C n  1 ;  n,le ) + H n,l  1 e , g e =  ( W 1 ATTN h (  ) + W 2 ATTN c (  ) + b ) , (12) where g e is a gate unit, ATTN h (  ) and ATTN c (  ) are attention models (see Eq. (1)) with different parameters. O n,le is further processed by FFN (  ) to output the representation H n,le . Symmetrically, in the decoder, S nd in Eq. (2) can be calculated as S nd = g d (cid:12) ATTN h (O nd , B ne , B ne ;  nd ) + (1  g d ) (cid:12) ATTN c (O nd , C n , C n ;  nd ) + O ld (13) where O nd is the output of the self-attention sublayer defined in Eq. (2). g d is another gate unit. We first evaluate the proposed method on IWSLT14 English  German (En  De) and IWSLT17 English  French (En  Fr) benchmarks. To make the results more convincing, we also experiment on a larger WMT14 English  German (En  De) dataset. Dataset. The dataset for IWSLT14 En  De are as in Ranzato et al. (2016), with 160 k sentence pairs for training and 7584 sentence pairs for validation. The concatenated validation sets are used as the test set (dev2010, dev2012, tst2010, tst2011, tst2012). For En  Fr, there are 236 k sentence pairs for training and 10263 for validation. The concatenated validation sets are used as the test set (dev2010, tst2010, tst2011, tst2012, tst2013, tst2014, tst2015). For all IWSLT translation tasks, we use a joint source and target vocabulary with 10 k byte-pair-encoding (BPE) types (Sennrich et al., 2016). For the WMT14 En  De task, the training corpus is identical to previous work (Vaswani et al., 2017; Wang et al., 2019a), which consists of about 4.5 million sentence pairs. All the data are tokenized using the script tokenizer.pl of Moses (Koehn et al., 2007) and segmented into subword symbols using jointly BPE with 32 k merge operations. The shared source-target vocabulary contains about 37 k BPE tokens. We use newstest2013 as the development set and newstest2014 as the test set. Following previous work, we evaluate IWSLT tasks with tokenized case-insensitive BLEU and report tokenized case-sensitive BLEU (Papineni et al., 2002) for WMT14 En  De. Model Settings. For IWSLT, the model configuration is transformer iwslt , representing a small model with embedding size 256 and FFN layer dimension 512 . We train all models using the Adam optimizer (  1 / 2 = 0 . 9 / 0 . 98 ) with adaptive learning rate schedule (warm-up step with 4K for shallow models, 8K for deep models) as in (Vaswani et al., 2017) and label smoothing of 0 . 1 . Sentence pairs containing 16K  32K tokens are grouped into one batch. Unless otherwise stated, we train small models with 15K maximum steps, Depth 36-layer 54-layer 72-layer dec. ( N ) 6 6 6 enc. and decode sentences using beam search with a beam size of 5 and length penalty of 1 . 0 . For WMT14 En  De, the model configuration is transformer base/big , with a embedding size of 512 / 1024 and a FFN layer dimension of 2048 / 4096 . Experiments on WMT are conducted on 8 P100 GPUs. Following Ott et al. (2018), we accumulate the gradient 8 iterations and then update to simulate a 64-GPU environment with a batch-size of 65K tokens per step. The Adam optimizer (  1 / 2 = 0 . 9 / 0 . 98 for base ,  1 / 2 = 0 . 9 / 0 . 998 ) for big ) and the warm-up strategy (8K steps for base , 16K steps for big ) are also adopted. We use relatively larger batch size and dropout rate for deeper and bigger models for better convergence. The transformer base/big is updated for 100K/300K steps. For evaluation, we average the last 5/20 checkpoints for base/big , each of which is saved at the end of an epoch. Beam search is adopted with a width of 4 and a length penalty of 0 . 6 . We use multi-bleu.perl to evaluate both IWSLT and WMT tasks for fair comparison with previous work. We first evaluate 36-layer, 54-layer and 72-layer MSC nets on IWSLT tasks. Table 1 summarizes the architecture. As shown in Table 2, applying MSC to the vanilla Transformer with 6 layers slightly increases translation quality by +0.26  +0.37 BLEU ( 1 (cid:13) 2 (cid:13) ). When the depth is increasing to 36, we use relatively larger dropout rate of 0 . 3 and achieve substantially improvements (+1.4  +1.8 BLEU) over its shallow counterparts ( 3 (cid:13) v.s. 2 (cid:13) ). After that, we continue deepening the encoders in order, however, our extremely deep models (72 layers, 5 (cid:13) ) suffer from overfitting issue on the small IWSLT corpora, which cannot be solved by simply enlarging the dropout rate. We seek to solve this issue by applying L2 regularization to the weights of encoders with greatly increased depth. Results show that this works for deeper encoders ( 6 (cid:13) ). We also report the inference speed in Table 2 (the last column). As expected, the speed decreases # Model Param. with the depth of MSC increasing, which is consistent with observation of Wang et al. (2019a). Compared to the baseline, MSC (72 layers) reduces decoding speed by 26%. We leave further investigation on this issue to future work. For fair comparisons, we implement existing methods (Bapna et al., 2018; Wang et al., 2019a) on the same vanilla Transformer backbone. We separately list the results of 36-layer and 72-layer encoders on the IWSLT14 En  De task in Table 3. The method of Bapna et al. (2018) fail to train a very deep architecture while the method of Wang et al. (2019a) is exposed a degradation phenomenon (28.63  28.34). In contrast, MSC in both 36-layer and 72-layer cases outperform these methods. This suggests that our extremely deep models can easily bring improvements on translation quality from greatly increased depth, producing results substantially better than existing systems. Table 4 lists the results on the WMT14 En  De translation task and the comparison with the current state-of-the-art systems. The architectures ( N  M ) of the 18-layer, 36-layer and 48-layer encoders Model Param. are set as 6  3, 6  6 and 6  8 respectively. We can see that incorporating our MSC into the shallow base/big contributes to +0.24/+0.31 BLEU (27.44  27.68/28.86  29.17) improvements under the same depth. When the depth grows, MSC demonstrates promising improvements of +1.39  +2.51 BLEU points over its shallow counterparts. It is worth noting that deep MSC with the base setting significantly outperforms the shallow one with the big setting (29.17  30.19), though both of them have around the same number of parameters. Compared to existing models, our MSC outperforms the transparent model (Bapna et al., 2018) (+2.2 BLEU) and the DLCL model (+0.9 BLEU) (Wang et al., 2019a), two recent approaches for deep encoding. Compared to both the depth scaled model (Zhang et al., 2019a) and the current SOTA (Wu et al., 2019), our MSC achieves better performance with identical or less parameters. Analysis of Degradation. We examine 36-layer and 72-layer plain and MSC nets, respectively. For plain networks, we simply stack dozens of layers. As we can see from Figure 2(a), the plain nets suffer from the degradation problem, which is not caused by overfitting, as they exhibit lower training BLEU. In contrast, the 72-layer MSC exhibits higher training BLEU than the 36-layer counterpart and is generalizable to the validation data. This indicates that our MSC can be more easily optimized with greatly increased depth. Analysis of Handling Complicated Semantics. Although our MSC can enjoy improvements of BLEU score from increased depth, what does the models benefit from which is still implicit. To better understand this, we show the performance of deep MSC nets in handling sentences with complicated semantics. We assume that complicated sentences are difficult to fit with high prediction losses. Then we propose to use the modified prediction losses to identify these sentences: s ( x , y ) = E (cid:2)  log P ( y | x ;  ) (cid:3) + Std (cid:2)  log P ( y | x ;  ) (cid:3) , (14) where E (cid:2)  log P ( y | x ;  ) (cid:3) is approximated by: E (cid:2)  log P ( y | x ;  ) (cid:3)  1 KK (cid:88) k =1  log P ( y | x ;  ( k ) ) , (15) where {  ( k ) } Kk =1 indicates model parameters for the last K ( K = 20 ) checkpoints. Std[  ] is the 0.5 0.4 1.1 0.4 0.3 0.5 0.5 0.2 0.2 0.4 0.2 0.1 0.5 1.0 1.5 0.9 0.5 1.3 2.1 1.7 0 0.5 1 1.5 2 2.5 Simple (50.3) Ordinary (32.0) Difficult (23.7) Challenging (15.2) Plain-36 Plain-72 MSC-6 MSC-36 MSC-72 Figure 3: Comparison between plain nets and MSC nets on fine-grained test sets with increasing translation difficulty from Simple to Challenging. standard deviation of prediction loss of sentence y given sentence x , and the introduction of which aims to prevent training oscillations from affecting complicated sentences identification. We adopt a shallow plain net ( small , 6 layers) to assign the prediction loss s ( x , y ) to each sentence pair. Further, we split the IWSLT En  De test set into 4 equal parts according to the prediction losses, which are pre-defined to have Simple, Ordinary, Difficult and Challenging translation difficulties, respectively. 4 Results on these fine-grained test sets are shown in Figure 3. First of all, all methods yield minor BLEU improvements over the baseline on the first sub-set that containing sentences with little difficulties to be translated. However, when the translation difficulty increases, the improvements of the deep MSC nets are expanded to around 2 BLEU. These results indicate that our MSC framework deals with sentences which are difficult to be translated well. 4 The fine-grained test sets are publicly available at https://github.com/pemywei/MSC-NMT/tree/master/IWSLT_En2De_Split_Test . 4(a), when generating the next token of tun, the shallow MSC attends to diverse tokens, such as to, that, . and eos, which causes the generation of eos and the phrase be able to is mistakenly untranslated. Remarkably, the deep MSC (Figure 4(b)) mostly focuses on the source tokens be, able and to, and translates this complicated sentence successfully. More cases can be found in Appendix C. This kind of cases show the advantages of constructing extremely deep models for translating semantic-complicated sentences. Analysis of Error Propagation. To understand the propagation process of training signals, we col-lect the gradient norm of each encoder layer during training. Results in Figure 5 show that with the MSC framework each layer enjoys a certain value of gradient for parameter update, and the error signals traverse along the depth of the model without hindrance. MSC helps balance the gradient norm between top and bottom layers in deep models. Ablation Study. We conduct ablation study to investigate the performance of each component of our model. The results are reported in Table 5: (1) We use simple element-wise addition for feature fusion instead of using a gated combination as introduced in Section 3.2. This method achieves a 29.45 BLEU, which is lower than the best result. We additionally modify the implementation of the contextual collaboration cell Q (  ) as FFN (  ) , which shows that the performance is reduced by 0.5 BLEU. (2) Removing CXT-ENCATTENTION and/or contextual collaboration makes the BLEU score drop by  0.7, which suggests that multiscale Model BELU MSC , 72 layers 29.67 feature fusion with addition 29.45 implement Q (  ) in Eq. collaboration helps in constructing extremely deep models. (3) Considering that the deep MSC introduces more parameters, we also train another two MSC models with about the same or double number of parameters: with 18/36 layers, embedding size 512 and FFN layer dimension 1024. These models underperform the deeper 72-layer model, which shows that the number of parameters is not the key to the improvement. Researchers have constructed deep NMT models that use linear connections to reduce the gradient propagation length inside the topology (Zhou et al., 2016; Wang et al., 2017; Zhang et al., 2018b) or read-write operations on stacked layers of memories (Meng et al., 2015). Such work has been conducted on the basis of the conventional RNN architectures and may not be fully applicable to the advanced Transformer. Recently, Bapna et al. (2018) introduced a transparent network into NMT models to ease the optimization of models with deeper encoders. To improve gradient flow they let each decoder layer find an unique weighted combination of all encoder layer outputs, instead of just the top encoder layer. Wang et al. (2019a) found that adopting the proper use of layer normalization helps to learn deep encoders. A method was further proposed to combine layers and encourage gradient flow by simple shortcut connections. Zhang et al. (2019a) introduced a depth-scaled initialization to improve norm preservation and proposed a merged attention sublayer to avoid the computational overhead for deep models. Researchers have also explored growing NMT models in two stages (Wu et al., 2019), in which shallow encoders and decoders are trained in the first stage and subsequently held constant, when another set of shallow layers are stacked on the top. In concurrent work, Xu et al. (2019) studied the effect of the computation order of residual connection and layer normalization, and proposed an parameter initialization method with Lipschitz restrictions to ensure the convergence of deep Transformers. Our method significantly differs from these methods, solving the problem by associating the decoder with the encoder with multi-granular dependencies in different space-scales. Exploiting deep representations have been studied to strengthen feature propagation and encourage feature reuse in NMT (Shen et al., 2018; Dou et al., 2018, 2019; Wang et al., 2019b). All of these works mainly attend the decoder to the final output of the encoder stack, we instead coordinate the encoder and the decoder at earlier stage. In this paper, we propose a multisacle collaborative framework to ease the training of extremely deep NMT models. Specifically, instead of the top-most representation of the encoder stack, we attend the decoder to multi-granular source information with different space-scales. We have shown that the proposed approach boosts the training of very deep models and can bring improvements on translation quality from greatly increased depth. Experiments on various language pairs show that the MSC achieves prominent improvements over strong baselines as well as previous deep models. In the future, we would like to extend our model to extremely large datasets, such as WMT'14 English-to-French with about 36M sentence-pairs. And the deeper MSC model results in high computational overhead, to address this issue, we would like to apply the average attention network (Zhang et al., 2018a) to our deep MSC models. We would like to thank the anonymous reviewers for the helpful comments. We also thank Xingxing Zhang, Luxi Xing and Kaixin Wu for their instructive suggestions and invaluable help. This work is supported by the National Key Research and Development Program (Grant No.2017YFB0803301).
abstain method method method result result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain method abstain abstain method result objective result method abstain other abstain other method abstain abstain other abstain other abstain abstain method other abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain method abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain result abstain abstain abstain abstain abstain result abstain other abstain abstain abstain abstain abstain abstain result other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain other other other other other other other other other objective other abstain objective abstain objective abstain objective result other other other