Answers.Body: <p>Look, this is currently a quite contentious issue. J3soons answer already links to the original paper and another one. However, most of the evidence for batch normalization is still empirical, and there is very little theoretical explanation for why it works.</p>
  <p>There are a number of competing theories out there. Also, there are a number of papers providing explanations for how batch normalization works in certain settings. However, there is no general theorem/explanation that can explain why the technique works.</p>
  <p>Here's <a href="http://proceedings.mlr.press/v89/kohler19a/kohler19a.pdf" rel="nofollow noreferrer">one paper</a> analyzing how it works in simplified settings. <a href="https://papers.nips.cc/paper/2018/file/36072923bfc3cf47745d704feb489480-Paper.pdf" rel="nofollow noreferrer">This one</a> provides another analysis of the batch norm (you can find their appendices on <a href="https://arxiv.org/pdf/1806.02375.pdf" rel="nofollow noreferrer">Arxiv</a>). Here's <a href="https://openreview.net/pdf?id=d-XzF81Wg1" rel="nofollow noreferrer">another one</a> which is the most recent peer reviewed one I could find. To quote this last paper:</p>
  <blockquote>
  <p>Perhaps at the core of the confusion is that BatchNorm has many
  effects. It has been correlated to reducing covariate shift (Ioffe and
  Szegedy, 2015), enabling higher learning rates (Bjorck et al., 2018),
  improving initialization (Zhang et al., 2019), and improving
  conditioning (Desjardins et al., 2015), to name a few. These entangled
  effects make it difficult to properly study the technique.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-14T19:42:32.260Z
Answers.LastActivityDate: 2022-01-14T19:42:32.260Z
Answers.UserDisplayName: Avatrin
Answers.Body: <p><strong>Definition and Explaination</strong></p>
  
  <p>For how Batch Normalization works exactly, I'll suggest you to read the following papers:</p>
  
  <ul>
  <li><a href="https://arxiv.org/abs/1502.03167" rel="nofollow noreferrer">Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift</a></li>
  <li><a href="https://arxiv.org/abs/1805.11604" rel="nofollow noreferrer">How Does Batch Normalization Help Optimization?</a></li>
  </ul>
  
  <p>The recent interpretation on How BN works is that it can reduce the high-order effect as mentioned in <a href="https://www.youtube.com/embed/Xogn6veSyxA?start=325&amp;end=664&amp;version=3" rel="nofollow noreferrer">Ian Goodfellow's lecture</a>. So it's not really about reducing the internal covariate shift.</p>
  
  <p><strong>Intuition</strong></p>
  
  <p>For how it works intuitively, you can think that we want to normalize the intermediate outputs (zero mean and unit variance) if the normalization won't remove too much useful information.</p>
  
  <p>However, normalization may not be suitable for all intermediate outputs. So <span class="math-container">$\beta$</span> and <span class="math-container">$\gamma$</span> is introduced to provide additional flexibility, if normalization removes too much useful information then <span class="math-container">$\beta$</span> and <span class="math-container">$\gamma$</span> will learn to become the original mean and variance, making the BN layer an identity transformation, as if it doesn't exist.</p>
  
  <p>In practice, <span class="math-container">$\beta$</span> and <span class="math-container">$\gamma$</span> won't become the original mean and variance, since all intermediate outputs can be normalized in some certain way without losing too much useful information. So you can think of it to be a customized normalization for each BN layer.</p>
  
  <p><strong>tl;dr</strong></p>
  
  <p>BN layer normalize the intermediate outputs in default, however, if the neural network find out that these intermediate outputs should not be normalized, then the neural network undos or provide more flexibility to the normalization.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-24T21:39:55.137Z
Answers.Text: What is the "high-order effect"?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-26T08:46:44.083Z
Answers.Text: @nbro You can see Ian Goodfellow's lecture or [this post](https://mlexplained.com/2018/01/10/an-intuitive-explanation-of-why-batch-normalization-really-works-normalization-in-deep-learning-part-1/). It's mainly about the high-order interactions.
Answers.UserDisplayName: J3soon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-19T14:49:32.610Z
Answers.Text: It's funny because the post doesn't actually explain how high order terms interact with normalized vs not. Only saying that it does.
Answers.UserDisplayName: FourierFlux
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-24T10:47:30.020Z
Answers.LastActivityDate: 2019-12-24T10:47:30.020Z
Answers.UserDisplayName: J3soon
Answers.Body: <p>Andrej Karpathy did a <a href="https://www.youtube.com/watch?v=P6sfmUTpUmc" rel="nofollow noreferrer">video</a> explaining initialization of neural networks.</p>
  <p>To summarize, activations like tanh and relu cause too many vanishing gradients if the distribution of inputs has too larger standard deviation (or the mean negative for relu).</p>
  <p>Putting batch norm (without β and γ) before your tanh will prevent it becoming saturated (i.e. prevent vanishing gradients) as less input values will be on the flat tails (i.e. <span class="math-container">$&lt;-3$</span> or <span class="math-container">$&gt;3$</span>).</p>
  <p>However, adding in β and γ will allow the neural network to control the saturation optimally of the tanh.</p>
  <p>Similarity, zero mean input to relu will mean half of the inputs are saturated (and have vanishing gradient). A positive β parameter will stop this from happening.</p>
  <p>So you can think of the step 2 as optimizing the saturation of the activation function.</p>
  <p>Of course you can put the batch norm after the activation function, and perhaps the second step wouldn't be as needed here.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-27T12:56:13.053Z
Answers.Text: I thought it is weird to put BN between the model weights and its activation.  IMO the order should be BN -> weights -> activation. so beta and gamma are absorbed to the layer's weights and don't show up as separate parameters to be optimized. I'm sure this has caused lots of discussion here and elsewhere.
Answers.UserDisplayName: NikoNyrh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-27T21:02:28.810Z
Answers.Text: @NikoNyrh yes I probably should have said beta and gamma are completely useless before a linear layer. Although I'm not sure about conv layers
Answers.UserDisplayName: Tom Huntington
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-26T21:36:08.763Z
Answers.LastActivityDate: 2022-10-26T21:36:08.763Z
Answers.UserDisplayName: Tom Huntington
Body: <p>I understood that we normalize to <strong>input</strong> features in order to bring them on the same scale so that weights won't be learned in arbitrary fashion and training would be faster. </p>
  
  <p>Then I studied about <strong>batch-normalization</strong> and observed that we can do the normalization for outputs of the <strong>hidden</strong> layers in following way:</p>
  
  <p><strong>Step 1:</strong> normalize the output of the hidden layer in order to have zero mean and unit variance a.k.a. <strong>standard normal</strong> (i.e. subtract by mean and divide by std dev of that minibatch).</p>
  
  <p><strong>Step 2:</strong> rescale this normalized vector to a new vector with new distribution having <span class="math-container">$\beta$</span> mean and <span class="math-container">$\gamma$</span> standard deviation, where both <span class="math-container">$\beta$</span> and <span class="math-container">$\gamma$</span> are trainable.</p>
  
  <p>I did not understand the <strong>purpose of the second step</strong>. Why can't we just do the first step, make the vector standard normal, and then move forward? Why do we need to rescale the input of each hidden neuron to an arbitrary distribution which is learned (through beta and gamma parameters)?</p>
  
Tags: <neural-networks><deep-learning><batch-normalization>
Title: How does a batch normalization layer work?
UserDisplayName: exAres
ThreadText: # How does a batch normalization layer work?
  
  (Asked by: username_0 on 2019-12-23)
  
  I understood that we normalize to **input** features in order to bring them on the same scale so that weights won't be learned in arbitrary fashion and training would be faster.
  
  Then I studied about **batch-normalization** and observed that we can do the normalization for outputs of the **hidden** layers in following way:
  
  **Step 1:** normalize the output of the hidden layer in order to have zero mean and unit variance a.k.a. **standard normal** (i.e. subtract by mean and divide by std dev of that minibatch).
  
  **Step 2:** rescale this normalized vector to a new vector with new distribution having $\beta$ mean and $\gamma$ standard deviation, where both $\beta$ and $\gamma$ are trainable.
  
  I did not understand the **purpose of the second step**. Why can't we just do the first step, make the vector standard normal, and then move forward? Why do we need to rescale the input of each hidden neuron to an arbitrary distribution which is learned (through beta and gamma parameters)?
  
  **Tags:** neural-networks, deep-learning, batch-normalization
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2022-01-14. Score: 2
  
  Look, this is currently a quite contentious issue. J3soons answer already links to the original paper and another one. However, most of the evidence for batch normalization is still empirical, and there is very little theoretical explanation for why it works.
  
  There are a number of competing theories out there. Also, there are a number of papers providing explanations for how batch normalization works in certain settings. However, there is no general theorem/explanation that can explain why the technique works.
  
  Here's [one paper](http://proceedings.mlr.press/v89/kohler19a/kohler19a.pdf) analyzing how it works in simplified settings. [This one](https://papers.nips.cc/paper/2018/file/36072923bfc3cf47745d704feb489480-Paper.pdf) provides another analysis of the batch norm (you can find their appendices on [Arxiv](https://arxiv.org/pdf/1806.02375.pdf)). Here's [another one](https://openreview.net/pdf?id=d-XzF81Wg1) which is the most recent peer reviewed one I could find. To quote this last paper:
  
  > Perhaps at the core of the confusion is that BatchNorm has many
  > effects. It has been correlated to reducing covariate shift (Ioffe and
  > Szegedy, 2015), enabling higher learning rates (Bjorck et al., 2018),
  > improving initialization (Zhang et al., 2019), and improving
  > conditioning (Desjardins et al., 2015), to name a few. These entangled
  > effects make it difficult to properly study the technique.
  
  
  ---
  
  ### Answer by username_2 on 2019-12-24. Score: 1
  
  **Definition and Explaination**
  
  For how Batch Normalization works exactly, I'll suggest you to read the following papers:
  
  * [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://arxiv.org/abs/1502.03167)
  * [How Does Batch Normalization Help Optimization?](https://arxiv.org/abs/1805.11604)
  
  The recent interpretation on How BN works is that it can reduce the high-order effect as mentioned in [Ian Goodfellow's lecture](https://www.youtube.com/embed/Xogn6veSyxA?start=325&end=664&version=3). So it's not really about reducing the internal covariate shift.
  
  **Intuition**
  
  For how it works intuitively, you can think that we want to normalize the intermediate outputs (zero mean and unit variance) if the normalization won't remove too much useful information.
  
  However, normalization may not be suitable for all intermediate outputs. So $\beta$ and $\gamma$ is introduced to provide additional flexibility, if normalization removes too much useful information then $\beta$ and $\gamma$ will learn to become the original mean and variance, making the BN layer an identity transformation, as if it doesn't exist.
  
  In practice, $\beta$ and $\gamma$ won't become the original mean and variance, since all intermediate outputs can be normalized in some certain way without losing too much useful information. So you can think of it to be a customized normalization for each BN layer.
  
  **tl;dr**
  
  BN layer normalize the intermediate outputs in default, however, if the neural network find out that these intermediate outputs should not be normalized, then the neural network undos or provide more flexibility to the normalization.
  
  > **Comments:**
  > **username_3**: What is the "high-order effect"?  
  > **username_2**: @nbro You can see Ian Goodfellow's lecture or [this post](https://mlexplained.com/2018/01/10/an-intuitive-explanation-of-why-batch-normalization-really-works-normalization-in-deep-learning-part-1/). It's mainly about the high-order interactions.  
  > **username_4**: It's funny because the post doesn't actually explain how high order terms interact with normalized vs not. Only saying that it does.  
  
  ---
  
  ### Answer by username_5 on 2022-10-26. Score: 0
  
  Andrej Karpathy did a [video](https://www.youtube.com/watch?v=P6sfmUTpUmc) explaining initialization of neural networks.
  
  To summarize, activations like tanh and relu cause too many vanishing gradients if the distribution of inputs has too larger standard deviation (or the mean negative for relu).
  
  Putting batch norm (without β and γ) before your tanh will prevent it becoming saturated (i.e. prevent vanishing gradients) as less input values will be on the flat tails (i.e. $<-3$ or $>3$).
  
  However, adding in β and γ will allow the neural network to control the saturation optimally of the tanh.
  
  Similarity, zero mean input to relu will mean half of the inputs are saturated (and have vanishing gradient). A positive β parameter will stop this from happening.
  
  So you can think of the step 2 as optimizing the saturation of the activation function.
  
  Of course you can put the batch norm after the activation function, and perhaps the second step wouldn't be as needed here.
  
  > **Comments:**
  > **username_6**: I thought it is weird to put BN between the model weights and its activation.  IMO the order should be BN -> weights -> activation. so beta and gamma are absorbed to the layer's weights and don't show up as separate parameters to be optimized. I'm sure this has caused lots of discussion here and elsewhere.  
  > **username_5**: @NikoNyrh yes I probably should have said beta and gamma are completely useless before a linear layer. Although I'm not sure about conv layers  
  
  ---
  
  