The Transformer translation model employs residual connection and layer normalization to ease the optimization difficulties caused by its multi-layer encoder/decoder structure. Previous research shows that even with residual connection and layer normalization, deep Transformers still have difficulty in training, and particularly Transformer models with more than 12 encoder/decoder layers fail to converge. In this paper, we first empirically demonstrate that a simple modification made in the official implementation, which changes the computation order of residual connection and layer normalization, can significantly ease the optimization of deep Transformers. We then compare the subtle differences in computation order in considerable detail, and present a parameter initialization method that leverages the Lipschitz constraint on the initialization of Transformer parameters that effectively ensures training convergence. In contrast to find-ings in previous research we further demonstrate that with Lipschitz parameter initialization, deep Transformers with the original computation order can converge, and obtain significant BLEU improvements with up to 24 layers. In contrast to previous research which focuses on deep encoders, our approach additionally enables Transformers to also benefit from deep decoders. Neural machine translation has achieved great success in the last few years (Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017). The Transformer (Vaswani et al., 2017), which has outperformed previous RNN/CNN based translation models (Bahdanau et al., 2015; Gehring et al., 2017), is based on multi-layer self-attention networks and can be trained very efficiently. The  Corresponding author. multi-layer structure allows the Transformer to model complicated functions. Increasing the depth of models can increase their capacity but may also cause optimization difficulties (Mhaskar et al., 2017; Telgarsky, 2016; Eldan and Shamir, 2016; He et al., 2016; Bapna et al., 2018). In order to ease optimization, the Transformer employs residual connection and layer normalization techniques which have been proven useful in reducing optimization difficulties of deep neural networks for various tasks (He et al., 2016; Ba et al., 2016). However, even with residual connections and layer normalization, deep Transformers are still hard to train: the original Transformer (Vaswani et al., 2017) only contains 6 encoder/decoder layers. Bapna et al. (2018) show that Transformer models with more than 12 encoder layers fail to converge, and propose the Transparent Attention (TA) mechanism which combines outputs of all encoder layers into a weighted encoded representation. Wang et al. (2019) find that deep Transformers with proper use of layer normalization are able to converge and propose to aggregate previous layers' outputs for each layer. Wu et al. (2019) explore incrementally increasing the depth of the Transformer Big by freezing pre-trained shallow layers. Concurrent work closest to ours is Zhang et al. (2019). They address the same issue, but propose a different layer-wise initialization approach to reduce the standard deviation. Our contributions are as follows:  We empirically demonstrate that a simple modification made in the Transformer's official implementation (Vaswani et al., 2018) which changes the computation order of residual connection and layer normalization can effectively ease its optimization;  We deeply analyze how the subtle difference of computation order affects convergence in Process Dropout + Norm Input/Norm Process Dropout + Norm Input deep Transformers, and propose to initialize deep Transformers under the Lipschitz constraint;  In contrast to previous works, we empirically show that with proper parameter initialization, deep Transformers with the original computation order can converge;  Our simple approach effectively ensures the convergence of deep Transformers with up to 24 layers, and achieves +1 . 50 and +0 . 92 BLEU improvements over the baseline on the WMT 14 English to German task and the WMT 15 Czech to English task;  We further investigate deep decoders for the Transformer in addition to the deep encoders studied in previous works, and show that deep decoders can also benefit the Transformer. In this paper we focus on the convergence of the training of deep transformers. To alleviate the training problem for the standard Transformer model, Layer Normalization (Ba et al., 2016) and Residual Connection (He et al., 2016) are adopted. The official implementation (Vaswani et al., 2018) of the Transformer uses a different computation order (Figure 1 b) compared to the published version (Vaswani et al., 2017) (Figure 1 a), since it (Fig-ure 1 b) seems better for harder-to-learn models. 1 Even though several studies (Chen et al., 2018; Domhan, 2018) have mentioned this change and although Wang et al. (2019) analyze the difference between the two computation orders during back-propagation, and Zhang et al. (2019) point out the effects of normalization in their work, how this modification impacts on the performance of the Transformer, especially for deep Transformers, has not been deeply studied before. Here we present both empirical convergence experiments (Table 1) and a theoretical analysis of the effect of the interaction between layer normalization and residual connection (Table 2). In order to compare with Bapna et al. (2018), we used the same datasets from the WMT 14 English to German task and the WMT 15 Czech to English task for our experiments. We applied joint Byte-Pair Encoding (BPE) (Sennrich et al., 2016) with 32k merge operations. We used the same setting as the Transformer base (Vaswani et al., 2017) except the number of warm-up steps was set to 8 k . Parameters were initialized with Glorot Initialization 2 (Glorot and Bengio, 2010) like in many other Transformer implementations (Klein et al., 2017; Hieber et al., 2017; Vaswani et al., 2018). We conducted experiments based on the Neutron implementation (Xu and Liu, 2019) of the Transformer translation model. Our experiments run on 2 GTX 1 https://github.com/tensorflow/ tensor2tensor/blob/v1.6.5/tensor2tensor/ layers/common_hparams.py#L110-L112 . 1080 Ti GPUs, and a batch size of 25 k target tokens is achieved through gradient accumulation of small batches. We used a beam size of 4 for decoding, and evaluated tokenized case-sensitive BLEU with the averaged model of the last 5 checkpoints saved with an interval of 1,500 training steps. Results of the two different computation orders are shown in Table 1, which shows that deep Transformers with the computation order of the official implementation (v2) have no convergence issue. Since the subtle change of computation order results in large differences in convergence, we further analyze the differences between the computation orders to investigate how they affect convergence. We conjecture that the convergence issue of deep Transformers is perhaps due to the fact that layer normalization over residual connections in Figure 1 (a) effectively reduces the impact of residual connections due to subsequent layer normalization, in order to avoid a potential explosion of combined layer outputs (Chen et al., 2018), which is also studied by Wang et al. (2019); Zhang et al. (2019). We therefore investigate how the layer normalization and the residual connection are computed in the two computation orders, shown in Table 2. Table 2 shows that the computation of residual connection in v1 is weighted by w compared to v2, and the residual connection of previous layers will be shrunk if w < 1 . 0 , which makes it difficult for deep Transformers to converge. Since the diminished residual connections (Table 2) may cause the convergence issue of deep v1 Transformers, is it possible to constrain w  1 . 0 ? Given that w is initialized with 1 , we suggest that the standard deviation of in model + in res should be constrained as follows: 0 . in which case w will be greater than or at least equal to 1 . 0 , and the residual connection of v1 will not be shrunk anymore. To achieve this goal, we can constrain elements of in model + in res to be in [ a, b ] and ensure that their standard deviation is smaller than 1 . 0 . Let's define P ( x ) as any probability distribution of x between [ a, b ] : b (cid:90) a P ( x ) dx = 1 . then the standard deviation of x is: Thus, as long as b  a  1 (the range of elements of the representation x ), the requirements for the corresponding  described in Equation 1 can be satisfied. To achieve this goal, we can simply constrain the range of elements of x to be smaller than 1 and initialize the sub-model before layer normalization to be a k-Lipschitz function, where k  1 . Because if the function F of the sub-layer is a k-Lipschitz function, for inputs x , y  [a , b] , | F ( x )  F ( y ) | < k | x  y | holds. Given that | x  y |  b  a , we can get | F ( x )  F ( y ) | < k ( b  a ) , the range of the output of that sub-layer is constrained by making it a k-Lipschitz function with constrained input. The k-Lipschitz constraint can be satisfied effectively through weight clipping, 3 and we empirically find that deep Transformers are only hard to train at the beginning and only applying a constraint to parameter initialization is sufficient, which is more efficient and can avoid a potential risk of weight clipping on performance. Zhang et al. (2019) also show that decreasing parameter variance at the initialization stage is sufficient for ensuring the convergence of deep Transformers, which is consistent with our observation. We use the training data described in Section 2 to examine the effectiveness of the proposed Lipschitz constrained parameter initialization approach. In practice, we initialize embedding matrices and weights of linear transformations with uniform distributions of [  e, + e ] and [  l, + l ] respectively. We use (cid:113) 2 esize + vsize as e and (cid:113) 1 isize as l where esize , vsize and isize stand for the size of embedding, vocabulary size and the input dimension of the linear transformation respectively. 4 Results for two computation orders with the new parameter initialization method are shown in Table 3. v1-L indicates v1 with Lipschitz constrained parameter initialization, the same for v2-L. Table 3 shows that deep v1-L models do not suffer from convergence problems anymore with our new parameter initialization approach. It is also worth noting that unlike Zhang et al. (2019), our parameter initialization approach does not degrade the translation quality of the 6-layer Transformer, and the 12-layer Transformer with our approach already achieves performance comparable to the 20-layer Transformer in Zhang et al. (2019) (shown in Table 1). 3 Note that the weight of the layer normalization cannot be clipped, otherwise residual connections will be more heavily shrunk. 4 To preserve the magnitude of the variance of the weights in the forward pass. While previous approaches (Bapna et al., 2018; Wang et al., 2019) only increase the depth of the encoder, we suggest that deep decoders should also be helpful. We analyzed the influence of deep encoders and decoders separately and results are shown in Table 4. Table 4 shows that the deep decoder can indeed benefit performance in addition to the deep encoder, especially on the Czech to English task. In contrast to previous works (Bapna et al., 2018; Wang et al., 2019; Wu et al., 2019) which show that deep Transformers with the computation order as in Vaswani et al. (2017) have difficulty in convergence, we show that deep Transformers with the original computation order can converge as long as proper parameter initialization is performed. We first investigate convergence differences between the published Transformer (Vaswani et al., 2017) and its official implementation (Vaswani et al., 2018), and compare the differences of computation orders between them. We conjecture that the convergence issue of deep Transformers is because layer normalization sometimes shrinks residual connections, we support our conjecture with a theoretical analysis (Table 2), and propose a Lipschitz constrained parameter initialization approach for solving this problem. Our experiments show the effectiveness of our simple approach on the convergence of deep Transformers, which achieves significant improvements on the WMT 14 English to German and the WMT 15 Czech to English news translation tasks. We also study the effects of deep decoders in addition to deep encoders extending previous works. We thank anonymous reviewers for their insightful comments. Hongfei Xu acknowledges the support of China Scholarship Council ([2018]3101, 201807040056). Deyi Xiong is supported by the National Natural Science Foundation of China (Grant No. 61861130364), the Natural Science Foundation of Tianjin (Grant No. 19JCZDJC31400) and the Royal Society (Lon-don) (NAF \ R1 \ 180122). Hongfei Xu, Josef van Genabith and Jingyi Zhang are supported by the German Federal Ministry of Education and Research (BMBF) under the funding code 01IW17001 (Deeplee).
abstain abstain objective method objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective objective method abstain abstain abstain abstain abstain other method method method method abstain method other abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain objective result objective other other other other