Answers.Body: <p>I'm not sure how just training the batch normalisation layer, you can get an accuracy of 83%. The batch normalisation layer parameters <span class="math-container">$\gamma^{(k)}$</span> and <span class="math-container">$\beta^{(k)}$</span>, are used to scale and shift the normalised batch outputs. These parameters are learnt during the back-propagation step. For the <span class="math-container">$k$</span>th layer, <span class="math-container">$$y^{(k)} = \gamma^{(k)}\hat{x}^{(k)} + \beta^{(k)}$$</span> The scaling and shifting are done to ensure a non - linear activation being outputted by each layer. Because batch normalisation scales outputs between 0-1, some activation functions are linear within that range (Eg. <span class="math-container">$tahh$</span> and <span class="math-container">$sigmoid$</span>)</p>
  <p>Regarding the second fact however, the difference between training and inference mode is this. During training mode, the statistics of each batch norm layer <span class="math-container">$\mu_B$</span> and <span class="math-container">$\sigma^2_B$</span> is computed. This statistic is used in scaling and normalising the outputs of the batch norm layer to have 0 mean and unit variance. At the same time, the current batch statistic computed is also used to update the running mean and running variance of the population. <span class="math-container">$\mu_B[t]$</span> represents the current batch mean, <span class="math-container">$\sigma^2_B[t]$</span> represents the current batch variance, while <span class="math-container">$\mu'_B[t]$</span> and <span class="math-container">$\sigma'_B[t]$</span> represent the accumulated means and variance from the previous batches. The running mean and variance of the population is then updated as <span class="math-container">$$\mu'_B[t]=\mu'_B[t]× momentum+ \mu_B[t]×(1−momentum)$$</span>
  <span class="math-container">$$\sigma'^2_B[t]=\sigma'^2_B[t] × momentum + \sigma^2_B[t]×(1−momentum)$$</span></p>
  <p>In inference mode, the batch normalisation uses the running mean and variance computed during training mode to scale and normalise inputs in the batch norm layer instead of the current batch mean and variance.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-10T08:52:10.530Z
Answers.LastActivityDate: 2020-07-10T08:52:10.530Z
Answers.UserDisplayName: calveeen
Answers.Body: <p>The original paper by Sergey Ioffe and Christian Szegedy; <a href="https://arxiv.org/abs/1502.03167" rel="nofollow noreferrer">https://arxiv.org/abs/1502.03167</a> &quot;Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift&quot; is very good. Make sure to go through the paper slowly and make annotations to truly understand it.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-11T09:47:55.823Z
Answers.LastActivityDate: 2020-07-11T09:47:55.823Z
Answers.UserDisplayName: BlueTurtle
Answers.Body: <p>A batch normalisation layer is like a standard FC layer but instead of learning weights and bias', you learn means and variances and scale the whole layer by said means and variances.</p>
  <p><strong>Fact 1:</strong></p>
  <p>Because it behaves just like a normal layer, and can learn, with the right structure it will learn to get a high enough accuracy.</p>
  <p><strong>Fact 2</strong></p>
  <p>Disabling learning on a batch norm layer is just like disabling learning on any other layer. It will not updated any of its parameters, and in this case the parameters are the means and variances, and so these will not be updated.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-10T06:08:41.893Z
Answers.LastActivityDate: 2020-07-10T06:08:41.893Z
Answers.UserDisplayName: Recessive
Body: <p>I actually went through the Keras' batch normalization tutorial and the description there puzzled me more.</p>
  <p>Here are some facts about batch normalization that I read recently and want a deep explanation on it.</p>
  <ol>
  <li><p>If you froze all layers of neural networks to their random initialized weights, except for batch normalization layers, you can still get 83% accuracy on CIFAR10.</p>
  </li>
  <li><p>When setting the trainable layer of batch normalization to false, it will run in inference mode and will not update its mean and variance statistics.</p>
  </li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-12T13:26:58.137Z
Comments.Text: Which tutorial are you exactly referring to? Can you please edit your post to include the link to the tutorial that states those facts?
Comments.UserDisplayName: nbro
Tags: <machine-learning><transfer-learning><batch-normalization>
Title: How does batch normalisation actually work?
UserDisplayName: Devansh Khandekar
ThreadText: # How does batch normalisation actually work?
  
  (Asked by: username_0 on 2020-07-10)
  
  I actually went through the Keras' batch normalization tutorial and the description there puzzled me more.
  
  Here are some facts about batch normalization that I read recently and want a deep explanation on it.
  
  1. If you froze all layers of neural networks to their random initialized weights, except for batch normalization layers, you can still get 83% accuracy on CIFAR10.
  2. When setting the trainable layer of batch normalization to false, it will run in inference mode and will not update its mean and variance statistics.
  
  **Tags:** machine-learning, transfer-learning, batch-normalization
  > **Comments:**
  > **username_1**: Which tutorial are you exactly referring to? Can you please edit your post to include the link to the tutorial that states those facts?  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2020-07-10. Score: 0 | Accepted answer
  
  I'm not sure how just training the batch normalisation layer, you can get an accuracy of 83%. The batch normalisation layer parameters $\gamma^{(k)}$ and $\beta^{(k)}$, are used to scale and shift the normalised batch outputs. These parameters are learnt during the back-propagation step. For the $k$th layer, $$y^{(k)} = \gamma^{(k)}\hat{x}^{(k)} + \beta^{(k)}$$ The scaling and shifting are done to ensure a non - linear activation being outputted by each layer. Because batch normalisation scales outputs between 0-1, some activation functions are linear within that range (Eg. $tahh$ and $sigmoid$)
  
  Regarding the second fact however, the difference between training and inference mode is this. During training mode, the statistics of each batch norm layer $\mu\_B$ and $\sigma^2\_B$ is computed. This statistic is used in scaling and normalising the outputs of the batch norm layer to have 0 mean and unit variance. At the same time, the current batch statistic computed is also used to update the running mean and running variance of the population. $\mu\_B[t]$ represents the current batch mean, $\sigma^2\_B[t]$ represents the current batch variance, while $\mu'\_B[t]$ and $\sigma'\_B[t]$ represent the accumulated means and variance from the previous batches. The running mean and variance of the population is then updated as $$\mu'\_B[t]=\mu'\_B[t]× momentum+ \mu\_B[t]×(1−momentum)$$
  $$\sigma'^2\_B[t]=\sigma'^2\_B[t] × momentum + \sigma^2\_B[t]×(1−momentum)$$
  
  In inference mode, the batch normalisation uses the running mean and variance computed during training mode to scale and normalise inputs in the batch norm layer instead of the current batch mean and variance.
  
  
  ---
  
  ### Answer by username_3 on 2020-07-11. Score: 0
  
  The original paper by Sergey Ioffe and Christian Szegedy; <https://arxiv.org/abs/1502.03167> "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift" is very good. Make sure to go through the paper slowly and make annotations to truly understand it.
  
  
  ---
  
  ### Answer by username_4 on 2020-07-10. Score: 0
  
  A batch normalisation layer is like a standard FC layer but instead of learning weights and bias', you learn means and variances and scale the whole layer by said means and variances.
  
  **Fact 1:**
  
  Because it behaves just like a normal layer, and can learn, with the right structure it will learn to get a high enough accuracy.
  
  **Fact 2**
  
  Disabling learning on a batch norm layer is just like disabling learning on any other layer. It will not updated any of its parameters, and in this case the parameters are the means and variances, and so these will not be updated.
  
  
  ---
  
  