Answers.Body: <p>Two things:</p>
  <ul>
  <li>Layer Norm wasn't invented before ResNet. ResNet still uses the regular Batch Norm.</li>
  <li>The model to use Layer Norm is residual block is ConvNeXt. Based on <a href="https://github.com/facebookresearch/ConvNeXt/blob/048efcea897d999aed302f2639b6270aedf8d4c8/models/convnext.py#L18" rel="nofollow noreferrer">this line</a>, it applies LayerNorm after the first Conv layer, before the activation layer.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-22T15:42:13.610Z
Answers.Text: I am aware of (1), but I should have added that I cannot use batch-norm because of a gradient penalty, which penalizes the gradient with respect
  to each input independently, rendering batch-norm ineffective.
  
  Regarding (2), I am not using any convolutional layers, as I don't work with image data. Do you know of any ResNet implemenatations with MLPs that use layer norm?
Answers.UserDisplayName: postnubilaphoebus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-22T15:39:27.127Z
Answers.LastActivityDate: 2023-03-22T15:39:27.127Z
Answers.UserDisplayName: Minh-Long Luu
Answers.Body: <p>Usually you insert the normalization layer (be it BatchNorm, LayerNorm or whatever) after the convolutional layer and before the activation layer, i.e. Conv + Norm + ReLU.</p>
  <p>The original ResNet applies the addition (skip connection) just before the last ReLU, but this design was revised in a <a href="https://arxiv.org/pdf/1603.05027.pdf" rel="nofollow noreferrer">follow-up paper</a> by the same authors. You can also see <a href="https://pi-tau.github.io/posts/res-nets/#the-residual-block-empowering-deeper-networks" rel="nofollow noreferrer">here</a> for figures and explanations.</p>
  <p>Regarding using skip connections with fully-connected layers, I think you can take a look at the transformer architecture. The transformer encoder (and decoder) architecture uses skip connections around fully connected layers. It also uses LayerNorm instead of BatchNorm. However, with this architecture there are again debates where exactly to place the LayerNorm layer. <a href="https://arxiv.org/pdf/2002.04745.pdf" rel="nofollow noreferrer">This paper</a> suggests using Pre-LayerNorm instead of Post-LayerNorm, because it stabilizes the training of the Transformer model. You can also see <a href="https://pi-tau.github.io/posts/transformer/#encoder-block" rel="nofollow noreferrer">here</a> for figures and explanations.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-23T18:25:19.073Z
Answers.LastActivityDate: 2023-07-23T18:25:19.073Z
Answers.UserDisplayName: pi-tau
Body: <p>Where do you insert layer norm in a residual block?
  After the addition or before the activation function (RELU in this case)?</p>
  
Tags: <residual-networks><layer-normalization>
Title: Layer Norm in a ResNet MLP
UserDisplayName: postnubilaphoebus
ThreadText: # Layer Norm in a ResNet MLP
  
  (Asked by: username_0 on 2023-03-22)
  
  Where do you insert layer norm in a residual block?
  After the addition or before the activation function (RELU in this case)?
  
  **Tags:** residual-networks, layer-normalization
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-03-22. Score: 1
  
  Two things:
  
  * Layer Norm wasn't invented before ResNet. ResNet still uses the regular Batch Norm.
  * The model to use Layer Norm is residual block is ConvNeXt. Based on [this line](https://github.com/facebookresearch/ConvNeXt/blob/048efcea897d999aed302f2639b6270aedf8d4c8/models/convnext.py#L18), it applies LayerNorm after the first Conv layer, before the activation layer.
  
  > **Comments:**
  > **username_0**: I am aware of (1), but I should have added that I cannot use batch-norm because of a gradient penalty, which penalizes the gradient with respect to each input independently, rendering batch-norm ineffective.  Regarding (2), I am not using any convolutional layers, as I don't work with image data. Do you know of any ResNet implemenatations with MLPs that use layer norm?  
  
  ---
  
  ### Answer by username_2 on 2023-07-23. Score: 1
  
  Usually you insert the normalization layer (be it BatchNorm, LayerNorm or whatever) after the convolutional layer and before the activation layer, i.e. Conv + Norm + ReLU.
  
  The original ResNet applies the addition (skip connection) just before the last ReLU, but this design was revised in a [follow-up paper](https://arxiv.org/pdf/1603.05027.pdf) by the same authors. You can also see [here](https://username_2.github.io/posts/res-nets/#the-residual-block-empowering-deeper-networks) for figures and explanations.
  
  Regarding using skip connections with fully-connected layers, I think you can take a look at the transformer architecture. The transformer encoder (and decoder) architecture uses skip connections around fully connected layers. It also uses LayerNorm instead of BatchNorm. However, with this architecture there are again debates where exactly to place the LayerNorm layer. [This paper](https://arxiv.org/pdf/2002.04745.pdf) suggests using Pre-LayerNorm instead of Post-LayerNorm, because it stabilizes the training of the Transformer model. You can also see [here](https://username_2.github.io/posts/transformer/#encoder-block) for figures and explanations.
  
  
  ---
  
  