Answers.Body: <p><em>"Naturally, this will affect the gradient through the network."</em> this statement is only partially true, let's see why by starting explaining the real aim of batch normalisation.</p>
  
  <p>As the title of the paper suggest, the aim of batch normalisation is to decrease training time by reducing covariance shift. What is covariance shift? We can conceive it as the variation that can happen between between the values of two layers of a network. We are all familiar with the concept that if we have input features with different unite scale, like kilos and euros, most likely lot of values will have different magnitude orders, like thousands for the weight could appear often with hundred of thousands for the money. When applying the activation function to different order of magnitude this discrepancy will remain, causing the values in the first layer to assume a really broad range of values. This is not good, since high fluctuation means more time to converge to stable values. This is why values fed to neural nets are always standardised. </p>
  
  <p>The authors applied the same logic to the hidden layers, arguing that a deep neural net can be conceived as a repetition of itself (every hidden layer is an input layer that send features to another hidden layer) ergo the features should be normalised at every layer. How can we do it? Normalising every batch is the most natural way to procede, but in this way there is the risk to end up transforming the internal representation of a layer, because normalising is not a linear transformation. This is way the authors propose a clever way to perform hidden layers normalisation, which consists in a classic normalisation followed by a linear scaling performed with two trainable parameters <span class="math-container">$\beta$</span> and <span class="math-container">$\gamma$</span> (which appear in the last step of the batch norm scratch code below). </p>
  
  <p><a href="https://i.sstatic.net/0eObf.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0eObf.png" alt="enter image description here"></a></p>
  
  <p>A really important thing to notice is that the mean and variance used to perform the classic normalisation are <strong>mean and variance calculated on the mini batch</strong>. I will explain why this is important in a sec, first I want to stress out that the <span class="math-container">$\beta$</span> parameter can actually bring to increase overfitting when batch norm is randomly stucked on top of other layer. The reason why is that, as we see in the scale and shift step, <span class="math-container">$\beta$</span> is nothing but a biased term, added to perform the shift of the mean of the batch hidden values. So, to avoid having an extra bias that lead to overfit, the bays on the previous layer should be removed, leaving only the classic matrix of weights parameters.</p>
  
  <p>Back to the gradient problem, we can see that in itself doesn't necessarily lead to increased performances, but it does provide an advantage in terms of hidden layer values convergence. The x axis on the two right sub plots of the figure below represent the variation of the hidden values of net trained with and without batch norm. When trained with batch norm the hidden values reach stable ranges after few iteration. This help the net to reach high accuracy in less iterations (first subplot on the left) but we can see that even without batch norm the net reach eventually high accuracy.  </p>
  
  <p><a href="https://i.sstatic.net/6zpxF.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/6zpxF.png" alt="enter image description here"></a></p>
  
  <p>The only help provided by batch norm to the gradient is the fact that, as noticed before, the normalisation is firstly performed by calculating the mean and variance on individual batches. This is important because this partial estimation of mean and variance introduces noice. Similarly to drop out, which has a regularisation effect due to the noice generated by randomly deactivating part of the weights, batch norm can introduce regularisation by adding noice due to the larger or smaller mean and variance estimated on individual batches. But still, batch norm was not introduced as a regularisation technique, and the equation you put on the question simply prove that it is possible to calculate the derivatives of the equations applied to perform the batch norm. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-28T00:06:01.113Z
Answers.Text: This seems consistent with my knowledge (although, to be honest, I am not really an expert on the topic). However, I think you don't really address why and how batch normalization affects the gradient, if it mitigates or not the vanishing and/or exploding gradients.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-28T00:59:29.477Z
Answers.Text: It is also dubious as to whether the "covariate shift" is indeed why batch norm helps.  http://gradientscience.org/batchnorm/ and https://blog.paperspace.com/busting-the-myths-about-batch-normalization/
Answers.UserDisplayName: information_interchange
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-03T15:54:11.653Z
Answers.Text: @information_interchange Thank's for the links! Had a chance to read them yesterday and they are really interesting, it's a shame that I was not taught them during real classes, apparently, even some professors give for granted some explanations without digging deeper.
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-27T23:36:34.270Z
Answers.LastActivityDate: 2020-03-27T23:36:34.270Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>Batch norm is a technique where they essentially standardize the activations at each layer, before passing it on to the next layer. Naturally, this will affect the gradient through the network. I have seen the equations that derive the back-propagation equations for the batch norm layers. From the original paper: <a href="https://arxiv.org/pdf/1502.03167.pdf" rel="noreferrer">https://arxiv.org/pdf/1502.03167.pdf</a></p>
  
  <p><a href="https://i.sstatic.net/iWINr.png" rel="noreferrer"><img src="https://i.sstatic.net/iWINr.png" alt="From the paper: https://arxiv.org/pdf/1502.03167.pdf"></a></p>
  
  <p>However, I have trouble understanding if there is an intuitive understanding of what effect it actually has on the network. For instance, does it help with the exploding gradient problem, since the activations are rescaled, and the variance of them is constrained?</p>
  
Tags: <deep-learning><optimization><batch-normalization><vanishing-gradient-problem><exploding-gradient-problem>
Title: What effect does batch norm have on the gradient?
UserDisplayName: information_interchange
ThreadText: # What effect does batch norm have on the gradient?
  
  (Asked by: username_0 on 2020-03-27)
  
  Batch norm is a technique where they essentially standardize the activations at each layer, before passing it on to the next layer. Naturally, this will affect the gradient through the network. I have seen the equations that derive the back-propagation equations for the batch norm layers. From the original paper: <https://arxiv.org/pdf/1502.03167.pdf>
  
  [![From the paper: https://arxiv.org/pdf/1502.03167.pdf](https://i.sstatic.net/iWINr.png)](https://i.sstatic.net/iWINr.png)
  
  However, I have trouble understanding if there is an intuitive understanding of what effect it actually has on the network. For instance, does it help with the exploding gradient problem, since the activations are rescaled, and the variance of them is constrained?
  
  **Tags:** deep-learning, optimization, batch-normalization, vanishing-gradient-problem, exploding-gradient-problem
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-03-27. Score: 3
  
  *"Naturally, this will affect the gradient through the network."* this statement is only partially true, let's see why by starting explaining the real aim of batch normalisation.
  
  As the title of the paper suggest, the aim of batch normalisation is to decrease training time by reducing covariance shift. What is covariance shift? We can conceive it as the variation that can happen between between the values of two layers of a network. We are all familiar with the concept that if we have input features with different unite scale, like kilos and euros, most likely lot of values will have different magnitude orders, like thousands for the weight could appear often with hundred of thousands for the money. When applying the activation function to different order of magnitude this discrepancy will remain, causing the values in the first layer to assume a really broad range of values. This is not good, since high fluctuation means more time to converge to stable values. This is why values fed to neural nets are always standardised.
  
  The authors applied the same logic to the hidden layers, arguing that a deep neural net can be conceived as a repetition of itself (every hidden layer is an input layer that send features to another hidden layer) ergo the features should be normalised at every layer. How can we do it? Normalising every batch is the most natural way to procede, but in this way there is the risk to end up transforming the internal representation of a layer, because normalising is not a linear transformation. This is way the authors propose a clever way to perform hidden layers normalisation, which consists in a classic normalisation followed by a linear scaling performed with two trainable parameters $\beta$ and $\gamma$ (which appear in the last step of the batch norm scratch code below).
  
  [![enter image description here](https://i.sstatic.net/0eObf.png)](https://i.sstatic.net/0eObf.png)
  
  A really important thing to notice is that the mean and variance used to perform the classic normalisation are **mean and variance calculated on the mini batch**. I will explain why this is important in a sec, first I want to stress out that the $\beta$ parameter can actually bring to increase overfitting when batch norm is randomly stucked on top of other layer. The reason why is that, as we see in the scale and shift step, $\beta$ is nothing but a biased term, added to perform the shift of the mean of the batch hidden values. So, to avoid having an extra bias that lead to overfit, the bays on the previous layer should be removed, leaving only the classic matrix of weights parameters.
  
  Back to the gradient problem, we can see that in itself doesn't necessarily lead to increased performances, but it does provide an advantage in terms of hidden layer values convergence. The x axis on the two right sub plots of the figure below represent the variation of the hidden values of net trained with and without batch norm. When trained with batch norm the hidden values reach stable ranges after few iteration. This help the net to reach high accuracy in less iterations (first subplot on the left) but we can see that even without batch norm the net reach eventually high accuracy.
  
  [![enter image description here](https://i.sstatic.net/6zpxF.png)](https://i.sstatic.net/6zpxF.png)
  
  The only help provided by batch norm to the gradient is the fact that, as noticed before, the normalisation is firstly performed by calculating the mean and variance on individual batches. This is important because this partial estimation of mean and variance introduces noice. Similarly to drop out, which has a regularisation effect due to the noice generated by randomly deactivating part of the weights, batch norm can introduce regularisation by adding noice due to the larger or smaller mean and variance estimated on individual batches. But still, batch norm was not introduced as a regularisation technique, and the equation you put on the question simply prove that it is possible to calculate the derivatives of the equations applied to perform the batch norm.
  
  > **Comments:**
  > **username_2**: This seems consistent with my knowledge (although, to be honest, I am not really an expert on the topic). However, I think you don't really address why and how batch normalization affects the gradient, if it mitigates or not the vanishing and/or exploding gradients.  
  > **username_0**: It is also dubious as to whether the "covariate shift" is indeed why batch norm helps.  http://gradientscience.org/batchnorm/ and https://blog.paperspace.com/busting-the-myths-about-batch-normalization/  
  > **username_1**: @username_0 Thank's for the links! Had a chance to read them yesterday and they are really interesting, it's a shame that I was not taught them during real classes, apparently, even some professors give for granted some explanations without digging deeper.  
  
  ---
  
  