Answers.Body: <p>This is how I understand it.</p>
  <p>Batch normalization is used to remove internal &quot;covariate shift&quot; (wich may be not the case) by normalizing the input for each hidden layer using the statistics across the entire mini-batch, which averages each individual sample, so the input for each layer is always in the same range. This can be seen from the BN equation:</p>
  <p><span class="math-container">$$
  \textrm{BN}(x)= \gamma\left(\frac{x-\mu(x)}{\sigma(x)}\right)+\beta
  $$</span></p>
  <p>where <span class="math-container">$\gamma$</span> and <span class="math-container">$\beta$</span> are affine parameters learned from data; <span class="math-container">$\mu(x)$</span> and <span class="math-container">$\sigma(x)$</span> are the mean and standard deviation, computed <strong>across batch size and spatial dimensions independently for each feature channel</strong>. First, we normalize each channel with 0 mean and standard deviation of 1 according to batch statistics. We then scale and shift each channel with <span class="math-container">$\gamma$</span> and <span class="math-container">$\beta$</span>.</p>
  <p>This is fine if you want to classify an average object on an image from different viewing angles and lighting conditions. It is defined similarly to BN:</p>
  <p><span class="math-container">$$
  \textrm{LN}(x)= \gamma\left(\frac{x-\mu(x)}{\sigma(x)}\right)+\beta
  $$</span></p>
  <p>but now <span class="math-container">$\mu(x)$</span> and <span class="math-container">$\sigma(x)$</span> are computed <strong>across all channels for each individual sample.</strong> Here's an illustration of the difference:</p>
  <p><a href="https://i.sstatic.net/fAowJ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/fAowJ.png" alt="enter image description here" /></a></p>
  <p>So layer normalization averages input across channels (for 2d input), which preserves the statistics of an individual sample. In some cases, we want to penalize the weights norm with respect to an individual sample rather than to the entire batch, as was done in WGAN-GP.</p>
  <p>In terms of style transfer for images, it is also important to preserve the individual color statistics of a sample. Therefore, StyleGAN uses adaptive instance normalization, which is an extension of the original instance normalization, where each channel is normalized individually.</p>
  <p>In addition, BN has several problems: the batch size must be large enough to capture overall statistics, which is sometimes impossible if you are working with large images since the model won't fit in memory. The concept of a batch is not always present, or it may change from time to time.</p>
  <p>I strongly encourage you to read the original <a href="https://arxiv.org/pdf/1502.03167v3.pdf" rel="nofollow noreferrer">BN paper</a> and also:<br />
  <a href="https://arxiv.org/pdf/1703.06868.pdf" rel="nofollow noreferrer">Adaptive instance normalization</a> <br />
  <a href="https://arxiv.org/pdf/1803.08494.pdf" rel="nofollow noreferrer">Group Normalization</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-27T12:30:01.430Z
Answers.Text: Thanks for your thoughts Aray. I'm just not sure about some of the things you say. For instance, I don't think batch norm "averages each individual sample". I also don't think layer norm "averages input across channels". It's very possible though, that what you mean to say is correct. I think my two key takeaways from your response are 1) Layer normalization might be useful if you want to maintain the distribution of pixels (or whatever constitutes a sample), and 2) batch norm might not make sense with small batch sizes.
Answers.UserDisplayName: Alexander Soare
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-27T13:33:36.763Z
Answers.Text: I updated my answer by adding equations and illustrations to support my assertions. As I am also looking for better intuition and use cases, I look forward to any feedback and additions.
Answers.UserDisplayName: Aray Karjauv
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-27T18:03:32.443Z
Answers.Text: "Batch normalization is used to remove internal covariate shift". There's some controversy about it. https://ai.stackexchange.com/questions/27260/why-does-batch-normalization-work
Answers.UserDisplayName: Kostya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-27T18:30:11.517Z
Answers.Text: This is the original statement from the paper. The paper itself is titled "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift"
Answers.UserDisplayName: Aray Karjauv
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-01T21:30:21.360Z
Answers.Text: can you explain what is the meaning of `affine parameters`? I am not able to understand what/why they are be called `affine parameters`?
Answers.UserDisplayName: Alex
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-02T08:54:46.790Z
Answers.Text: @Alex you can also call them scaling and offset parameters. The name is not important. What's important is that they are learnable and that they rescale and shift the sample statistics.
Answers.UserDisplayName: Aray Karjauv
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-27T00:30:26.987Z
Answers.LastActivityDate: 2023-05-27T18:15:39.827Z
Answers.LastEditDate: 2023-05-27T18:15:39.827Z
Answers.UserDisplayName: Aray Karjauv
Body: <p>I'll start with my understanding of the literal difference between these two. First, let's say we have an input tensor to a layer, and that tensor has dimensionality <span class="math-container">$B \times D$</span>, where <span class="math-container">$B$</span> is the size of the batch and <span class="math-container">$D$</span> is the dimensionality of the input corresponding to a single instance within the batch.</p>
  <ul>
  <li>Batch norm does the normalization across the batch dimension <span class="math-container">$B$</span></li>
  <li>Layer norm does the normalization across <span class="math-container">$D$</span></li>
  </ul>
  <p>What are the differences in terms of the consequences of this choice?</p>
  
Tags: <deep-learning><comparison><batch-normalization><layer-normalization>
Title: What are the consequences of layer norm vs batch norm?
UserDisplayName: Alexander Soare
ThreadText: # What are the consequences of layer norm vs batch norm?
  
  (Asked by: username_0 on 2021-04-13)
  
  I'll start with my understanding of the literal difference between these two. First, let's say we have an input tensor to a layer, and that tensor has dimensionality $B \times D$, where $B$ is the size of the batch and $D$ is the dimensionality of the input corresponding to a single instance within the batch.
  
  * Batch norm does the normalization across the batch dimension $B$
  * Layer norm does the normalization across $D$
  
  What are the differences in terms of the consequences of this choice?
  
  **Tags:** deep-learning, comparison, batch-normalization, layer-normalization
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-04-27. Score: 5
  
  This is how I understand it.
  
  Batch normalization is used to remove internal "covariate shift" (wich may be not the case) by normalizing the input for each hidden layer using the statistics across the entire mini-batch, which averages each individual sample, so the input for each layer is always in the same range. This can be seen from the BN equation:
  
  $$
  \textrm{BN}(x)= \gamma\left(\frac{x-\mu(x)}{\sigma(x)}\right)+\beta
  $$
  
  where $\gamma$ and $\beta$ are affine parameters learned from data; $\mu(x)$ and $\sigma(x)$ are the mean and standard deviation, computed **across batch size and spatial dimensions independently for each feature channel**. First, we normalize each channel with 0 mean and standard deviation of 1 according to batch statistics. We then scale and shift each channel with $\gamma$ and $\beta$.
  
  This is fine if you want to classify an average object on an image from different viewing angles and lighting conditions. It is defined similarly to BN:
  
  $$
  \textrm{LN}(x)= \gamma\left(\frac{x-\mu(x)}{\sigma(x)}\right)+\beta
  $$
  
  but now $\mu(x)$ and $\sigma(x)$ are computed **across all channels for each individual sample.** Here's an illustration of the difference:
  
  [![enter image description here](https://i.sstatic.net/fAowJ.png)](https://i.sstatic.net/fAowJ.png)
  
  So layer normalization averages input across channels (for 2d input), which preserves the statistics of an individual sample. In some cases, we want to penalize the weights norm with respect to an individual sample rather than to the entire batch, as was done in WGAN-GP.
  
  In terms of style transfer for images, it is also important to preserve the individual color statistics of a sample. Therefore, StyleGAN uses adaptive instance normalization, which is an extension of the original instance normalization, where each channel is normalized individually.
  
  In addition, BN has several problems: the batch size must be large enough to capture overall statistics, which is sometimes impossible if you are working with large images since the model won't fit in memory. The concept of a batch is not always present, or it may change from time to time.
  
  I strongly encourage you to read the original [BN paper](https://arxiv.org/pdf/1502.03167v3.pdf) and also:  
  [Adaptive instance normalization](https://arxiv.org/pdf/1703.06868.pdf)   
  [Group Normalization](https://arxiv.org/pdf/1803.08494.pdf)
  
  > **Comments:**
  > **username_0**: Thanks for your thoughts Aray. I'm just not sure about some of the things you say. For instance, I don't think batch norm "averages each individual sample". I also don't think layer norm "averages input across channels". It's very possible though, that what you mean to say is correct. I think my two key takeaways from your response are 1) Layer normalization might be useful if you want to maintain the distribution of pixels (or whatever constitutes a sample), and 2) batch norm might not make sense with small batch sizes.  
  > **username_1**: I updated my answer by adding equations and illustrations to support my assertions. As I am also looking for better intuition and use cases, I look forward to any feedback and additions.  
  > **username_2**: "Batch normalization is used to remove internal covariate shift". There's some controversy about it. https://ai.stackexchange.com/questions/27260/why-does-batch-normalization-work  
  > **username_1**: This is the original statement from the paper. The paper itself is titled "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift"  
  > **username_3**: can you explain what is the meaning of `affine parameters`? I am not able to understand what/why they are be called `affine parameters`?  
  > **username_1**: @Alex you can also call them scaling and offset parameters. The name is not important. What's important is that they are learnable and that they rescale and shift the sample statistics.  
  
  ---
  
  