Answers.Body: <p>I think this phrasing is a bit misleading. If I understand this passage correctly, another way to put it would be:</p>
  <p>Applying batch normalization distorts the true data distribution: An arbitrarily distributed batch of data is transformed into a distribution with mean = 0 and standard deviation = 1. This might not be beneficial for the downstream task, so two scalars beta and gamma are introduced that are trainable and can modify the output of the BatchNorm. This formula for BatchNorm makes this clear:</p>
  <p><span class="math-container">$\text{BatchNorm}({\mathcal{X}}_{b}) = \gamma \cdot \frac{(\mathcal{X}_{b} - \text{mean}(\mathcal{X}_{b}))}{\text{std}(\mathcal{X}_{b})} + \beta$</span></p>
  <p>Without the gamma and beta, you would simply normalize the batch. But SGD can modify beta and gamma and thereby shift the distribution around according to what the optimization task demands. By acting on beta and gamma, the SGD can then somewhat 'counteract' this normalization.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-09T17:53:22.533Z
Answers.Text: Thank you for the detailed answer, it's crystal clear now!
Answers.UserDisplayName: Arighna
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-10T15:22:29.650Z
Answers.Text: I'm glad it helped :)
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-09T15:53:58.917Z
Answers.LastActivityDate: 2022-03-09T16:09:22.150Z
Answers.LastEditDate: 2022-03-09T16:09:22.150Z
Answers.UserDisplayName: Chillston
Body: <p>I want to understand the handshake between SGD (or mini-batch GD) and batch normalization. Below, an explanation quoted from <a href="https://jelal.medium.com/batch-normalization-8fb27a96b9c7" rel="nofollow noreferrer">this Medium article</a>. However, I am confused about the denormalization by the SGD.</p>
  <blockquote>
  <p>SGD ( Stochastic gradient descent) undoes this normalization if it’s a way for it to minimize the loss function.</p>
  <p>Consequently, batch normalization adds two trainable parameters to each layer, so the normalized output is multiplied by a “standard deviation” parameter (gamma) and add a “mean” parameter (beta). In other words, batch normalization lets SGD do the denormalization by changing only these two weights for each activation, instead of losing the stability of the network by changing all the weights.</p>
  </blockquote>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-03T15:00:04.317Z
Comments.Text: [This](https://jelal.medium.com/batch-normalization-8fb27a96b9c7) Medium article!
Comments.UserDisplayName: Arighna
Tags: <neural-networks><deep-learning><hyperparameter-optimization><batch-normalization><stochastic-gradient-descent>
Title: How does stochastic gradient descent undo the normalization done by the batch normalization?
UserDisplayName: Arighna
ThreadText: # How does stochastic gradient descent undo the normalization done by the batch normalization?
  
  (Asked by: username_0 on 2022-03-02)
  
  I want to understand the handshake between SGD (or mini-batch GD) and batch normalization. Below, an explanation quoted from [this Medium article](https://jelal.medium.com/batch-normalization-8fb27a96b9c7). However, I am confused about the denormalization by the SGD.
  
  > SGD ( Stochastic gradient descent) undoes this normalization if it’s a way for it to minimize the loss function.
  >
  > Consequently, batch normalization adds two trainable parameters to each layer, so the normalized output is multiplied by a “standard deviation” parameter (gamma) and add a “mean” parameter (beta). In other words, batch normalization lets SGD do the denormalization by changing only these two weights for each activation, instead of losing the stability of the network by changing all the weights.
  
  **Tags:** neural-networks, deep-learning, hyperparameter-optimization, batch-normalization, stochastic-gradient-descent
  > **Comments:**
  > **username_0**: [This](https://jelal.medium.com/batch-normalization-8fb27a96b9c7) Medium article!  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-03-09. Score: 0 | Accepted answer
  
  I think this phrasing is a bit misleading. If I understand this passage correctly, another way to put it would be:
  
  Applying batch normalization distorts the true data distribution: An arbitrarily distributed batch of data is transformed into a distribution with mean = 0 and standard deviation = 1. This might not be beneficial for the downstream task, so two scalars beta and gamma are introduced that are trainable and can modify the output of the BatchNorm. This formula for BatchNorm makes this clear:
  
  $\text{BatchNorm}({\mathcal{X}}\_{b}) = \gamma \cdot \frac{(\mathcal{X}\_{b} - \text{mean}(\mathcal{X}\_{b}))}{\text{std}(\mathcal{X}\_{b})} + \beta$
  
  Without the gamma and beta, you would simply normalize the batch. But SGD can modify beta and gamma and thereby shift the distribution around according to what the optimization task demands. By acting on beta and gamma, the SGD can then somewhat 'counteract' this normalization.
  
  > **Comments:**
  > **username_0**: Thank you for the detailed answer, it's crystal clear now!  
  > **username_1**: I'm glad it helped :)  
  
  ---
  
  