Answers.Body: <p>I think it'd be helpful to refer to the <a href="https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html" rel="nofollow noreferrer">batchnorm formula</a> given in the PyTorch implementation.</p>
  <p>In particular, given an input <span class="math-container">$x$</span>, you would get the mean and variance (<span class="math-container">$\mathbf{E}[x]$</span> and <span class="math-container">$\text{Var}[x]$</span>), and normalize it as such:
  <span class="math-container">$$ \hat{x} = \frac{x - \mathbf{E}[x]}{\sqrt{\text{Var}[x] + \epsilon}} $$</span>
  where <span class="math-container">$\epsilon$</span> is a small number to improve stability. There is also an additional affine transform applied in most implementations of batchnorm in standard libraries, so the output is really <span class="math-container">$\gamma \hat{x} + \beta$</span>, where <span class="math-container">$\gamma$</span> and <span class="math-container">$\beta$</span> are the affine parameters (scale and shift respectively).</p>
  <p>To answer your question (if I understood correctly), yes batchnorm would change the output of the previous layer, due to the normalization and affine transformation.</p>
  <p>There are some details to consider:</p>
  <ul>
  <li>In most implementations, the <span class="math-container">$\mathbf{E}[x]$</span> and <span class="math-container">$\text{Var}[x]$</span> are running mean estimates, meaning that there would be additional momentum hyperparameter to control how fast the estimates are updated as new inputs come in.</li>
  <li>For batchnorm, you would normalize and apply the affine transformations to each output channel separately, meaning that each output channel would hold its own set of running mean estimates and affine parameters.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-06T06:28:05.160Z
Answers.Text: Thanks for the reply, just have a quick question. When you say moving mean do you mean in the mini-batch or for the final inference stage? Specifically are these moving means and variances used only during the inference stage? I am asking this because it seems like a lot of work to go through the training sample all over again with normalized outputs for each node, rather than just having a moving mean and average from the start of the mini-batch.
Answers.UserDisplayName: liyu zerihun
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-06T06:31:18.813Z
Answers.Text: @liyuzerihun The moving means are only updated during training, and updated per mini-batch when fed forward through the network. For inference, the standard practice is to "freeze" them (meaning that they do not get updated).
Answers.UserDisplayName: PeaBrane
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-06T05:49:04.600Z
Answers.LastActivityDate: 2022-09-06T05:49:04.600Z
Answers.UserDisplayName: PeaBrane
Body: <p>I am asking this question because while designing my own model, I had repeated gradient explosion issues, so I wanted to try batch normalization. I really want to understand the details and math surrounding how it is implemented. Specifically wouldn't batch normalization effectively change the output of each layer as the activations are now normalized? Also, any details on alpha and beta parameters would be greatly appreciated! Thank you!</p>
  
Tags: <convolutional-neural-networks><gradient-descent><batch-normalization><normalisation>
Title: During batch normalization is the mini-batch gone through twice, one to calculate the mean and variance and then to normalize them?
UserDisplayName: liyu zerihun
ThreadText: # During batch normalization is the mini-batch gone through twice, one to calculate the mean and variance and then to normalize them?
  
  (Asked by: username_0 on 2022-09-05)
  
  I am asking this question because while designing my own model, I had repeated gradient explosion issues, so I wanted to try batch normalization. I really want to understand the details and math surrounding how it is implemented. Specifically wouldn't batch normalization effectively change the output of each layer as the activations are now normalized? Also, any details on alpha and beta parameters would be greatly appreciated! Thank you!
  
  **Tags:** convolutional-neural-networks, gradient-descent, batch-normalization, normalisation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-09-06. Score: 1
  
  I think it'd be helpful to refer to the [batchnorm formula](https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html) given in the PyTorch implementation.
  
  In particular, given an input $x$, you would get the mean and variance ($\mathbf{E}[x]$ and $\text{Var}[x]$), and normalize it as such:
  $$ \hat{x} = \frac{x - \mathbf{E}[x]}{\sqrt{\text{Var}[x] + \epsilon}} $$
  where $\epsilon$ is a small number to improve stability. There is also an additional affine transform applied in most implementations of batchnorm in standard libraries, so the output is really $\gamma \hat{x} + \beta$, where $\gamma$ and $\beta$ are the affine parameters (scale and shift respectively).
  
  To answer your question (if I understood correctly), yes batchnorm would change the output of the previous layer, due to the normalization and affine transformation.
  
  There are some details to consider:
  
  * In most implementations, the $\mathbf{E}[x]$ and $\text{Var}[x]$ are running mean estimates, meaning that there would be additional momentum hyperparameter to control how fast the estimates are updated as new inputs come in.
  * For batchnorm, you would normalize and apply the affine transformations to each output channel separately, meaning that each output channel would hold its own set of running mean estimates and affine parameters.
  
  > **Comments:**
  > **username_0**: Thanks for the reply, just have a quick question. When you say moving mean do you mean in the mini-batch or for the final inference stage? Specifically are these moving means and variances used only during the inference stage? I am asking this because it seems like a lot of work to go through the training sample all over again with normalized outputs for each node, rather than just having a moving mean and average from the start of the mini-batch.  
  > **username_1**: @username_0 The moving means are only updated during training, and updated per mini-batch when fed forward through the network. For inference, the standard practice is to "freeze" them (meaning that they do not get updated).  
  
  ---
  
  