The Transformer architecture is popularly used in natural language processing tasks. To train a Transformer model, a carefully designed learning rate warm-up stage is usually needed: the learning rate has to be set to an extremely small value at the beginning of the optimization and then gradually increases in some given number of iterations. Such a stage is shown to be crucial to the final performance and brings more hyper-parameter tunings. In this paper, we study why the learning rate warm-up stage is important in training the Transformer and theoretically show that the location of layer normalization matters. It can be proved that at the beginning of the optimization, for the original Transformer, which places the layer normalization between the residual blocks, the expected gradients of the parameters near the output layer are large. Then using a large learning rate on those gradients makes the training unstable. The warm-up stage is practically helpful to avoid this problem. Such an analysis motivates us to investigate a slightly modified Transformer architecture which locates the layer normalization inside the residual blocks. We show that the gradients in this Transformer architecture are well-behaved at initialization. Given these findings, we are the first to show that this Transformer variant is easier and faster to train. The learning rate warm-up stage can be safely removed, and the training time can be largely reduced on a wide range of applications.
1. Specific problem tackled by the paper:
Moving the LayerNorm layer to be inside the residual connection in a stack of transformers can remove the need for learning rate warm up. This paper provides a theoretical motivation for doing this.
2. Motivation of the paper:
The authors motivate the problem clearly, performing experiments to demonstrate the problem. Their experiments provide good context for the problem they are solving and acts as a solid reference point for their (and other peoples' future) work. To be more convincing  the authors should have performed multiple runs and shown the standard deviations across runs.
3. Claims of the paper:
The authors claim that layer norm should be placed inside the residual connection (Pre-LN) rather than outside (Post-LN) it. The authors show, theoretically, that in a Post-LN transformer the magnitude of the gradients in a transformer scale with the number of layers and that the magnitude of the hidden states scale linearly with the layers that that they are output by. While in a Pre-LN transformer the magnitudes of the gradients and states are independent of the number of layers.
i.e. During training a Post-LN transformer is likely to have weaker gradients in the lower (closer to input layers) than at the output layers.
This theory is backed by an empirical study. It is good that these experiments are repeated ten times however the authors should show the standard deviations too.
The authors show machine translation results, demonstrating that using Pre-LN rather than Post-LN leads to faster convergence, however the models converge to the same result. However, there is benefit in not having to design a training schedule. Wang et al. have also shown the benefits of Pre-LN rather than Post-LN transformers for machine translation.
4. Decision (accept or reject) with one or two key reasons for this choice and reasons for the decision.
Weak accept.
Pros:
(1) The authors provide theory that supports the use of Pre-LN rather than Post-LN transformers. Using Pre-LN rather than Post-LN transformers may save a lot of time by avoiding hyper-parameter tuning, without loss in performance and this looks very easy to implement.
(2) The paper is well organised and the problem is very well motivated.
(3) The experiments are sufficient. They could be improved with repeats and by showing the standard deviations (as mentioned above).
Cons:
(1) This work is incremental. Wang et al. have already shown that Pre-LN transformers are better that Post-LN transformers when the network has many transformer layers, they explain theoretically why this is the case. 
(2) While the organisation of the paper is good, the paper is not well written. The gramma is poor.
(3) The paper is very long for an incremental improvement.
5. Additional feedback with the aim to improve the paper. 
[a] Ideally, it would be good to see repeats for Figure 2 and the standard deviations for Figure 3.
[b] Without reading the appendix it is not clear where the assumption that W^Q and W^K are zero is used. Making some connection with how this assumption relates to the lemmas would be useful. Additionally, you should explain what this means qualitatively, because this makes the assumption more acceptable. I am assuming that it means that the attention is uniform.
[c] In Lemma two you are comparing the magnitudes for the input in the Pre-LN and the output in the Post-LN transformer according to how x_{l,i}^post and x_{l, i}^pre are defined in Table 1.
[d] In Figure 3(b) the gradients are clearly decreasing with the number of layers, are there any comments on this? In the limit this could cause vanishing gradients?
[e] On the surface Figure 2 and 4 appear to contradict. Is the difference a result of using RAdam? If so, this should be made very clear. If not, why are the results contradictory?
More minor comments:
[1] Many gramma errors.
[2] Figure 1 is referenced before explaining what an FFN is. Figure 1 could also be enhanced by labelling Post-LN as previous work and Pre-LN as current work.
[3] MultiHeadAtt is not well defined. Multi-Head( ), Attention( ) and Head( ) each take three arguments, while MutliHeadAtt( ) takes two. It would be worth connecting these.
[4]"sub-layer --> ..." here sub-layer is not defined. Should this say self-attention sub-layer?
[5] Where does equation (4) come from? There should be a citation and/or explanation.
[6] The BLEU score is not defined (this could be with a footnote).
[7] Top of page 7: "As most of the parameters are initialized by Gaussian distributions" --> Using the word "most" is very vague. The authors should be specific about which parameters they are referring to.
[8] There is a good balance of equations in the main text with most of the proofs in the appendix.
[9] Inconsistent use of LN and LayerNorm, both are used.