Answers.Body: <p>Theoretically, Yes and batchnorm can be omitted. Practically, No. Most of the intermediate features aren't normalized. For example, after a 2x2 Max pool of a normalized feature, the SD and mean would be vastly shifted to 0.8 and 1.2 (I don't remember the exact number) of the originals. It will take an immense amount of iterations for subsequent learnable parameters to absorb these scales and offsets. Before the absorption happens, those low SD feature map will be suppressed and lowering the overall network capacity. Other parameters are trained according the condition where some features are suppressed. The overall dynamics of unnormalized features are unpredictable. Batch Norm first normalize x, then the learnable gamma and beta are updated under the same scale of other normalized features.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-30T02:55:22.453Z
Answers.LastActivityDate: 2022-11-30T02:55:22.453Z
Answers.UserDisplayName: Max Law
Body: <p>Batch norm is a normalizing layer that is shown to help deep networks learn faster and with higher generalization accuracy. It normalizes the activations of the previous layer to a mean <span class="math-container">$\beta$</span> and variance <span class="math-container">$\gamma^2$</span> to prevent things like activations from exploding or shifting during the learning process.</p>
  <p>More specifically:
  <span class="math-container">$$\hat{x} = \displaystyle \frac{x - \mu_t}{\sqrt{\sigma_t^2 + \epsilon}}\label{1}\tag{1}$$</span>
  <span class="math-container">$$ BatchNorm_{\mu_t, \sigma_t}(x) = \gamma \hat{x} + \beta \label{2}\tag{2}$$</span></p>
  <p>where</p>
  <ul>
  <li><span class="math-container">$x$</span> is the layer input of the layer</li>
  <li><span class="math-container">$\mu_t, \sigma_t$</span> is the sample mean and standard deviation at time step <span class="math-container">$t$</span></li>
  <li><span class="math-container">$\epsilon$</span> is a small constant, and</li>
  <li><span class="math-container">$\gamma$</span> and <span class="math-container">$\beta$</span> are learnable parameters so that the output is not necessarily standardized to mean <span class="math-container">$0$</span> and variance <span class="math-container">$1$</span>, but possibly to another mean and variance that may be better for the neural network.</li>
  </ul>
  <p>My question is, why does BatchNorm first standardize the input <span class="math-container">$x$</span> to <span class="math-container">$\hat{x}$</span> before applying the learnable parameters <span class="math-container">$\gamma$</span> and <span class="math-container">$\beta$</span>? Isn't this redundant? The parameters <span class="math-container">$\gamma$</span> and <span class="math-container">$\beta$</span> could learn to standardize the input themselves right?</p>
  <p>In fact, as training progresses, <span class="math-container">$\mu_t$</span> and <span class="math-container">$\sigma_t$</span> becomes updated to new values <span class="math-container">$\mu_{t+1}$</span> and <span class="math-container">$\sigma_{t+1}$</span>, so the learned parameters at that time step, <span class="math-container">$\gamma_t$</span> and <span class="math-container">$\beta_t$</span>, no longer apply for time step <span class="math-container">$t+1$</span> since that involves a different standardization process with a different mean and variance. So by adding this standardization step, it may even hurt the convergence of the layer during learning, since it is adding the gradient of <span class="math-container">$BatchNorm_{\mu_{t+1}, \sigma_{t+1}}(x)$</span> to <span class="math-container">$BatchNorm_{\mu_t, \sigma_t}(x)$</span>, which are two different functions right?</p>
  <p>Why not just simply make it like this?</p>
  <p><span class="math-container">$$BatchNorm(x) = \gamma x + \beta \label{3}\tag{3}$$</span></p>
  <p>This would simplify the calculation of the gradients, which would make learning faster to compute.</p>
  <p>BatchNorm is one of the most successful developments of deep learning, so I know my intuition on these things is wrong -- I'm just curious as to what I am missing.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-13T10:55:52.733Z
Comments.Text: That second formula reminds me of the _re-parametrization trick_ (used e.g. in VAEs). Not sure if it's related or not, though, because I'm not currently familiar with the details of batch normalization. So, you could investigate this option.
Comments.UserDisplayName: nbro
Tags: <neural-networks><deep-learning><batch-normalization><normalisation><standardisation>
Title: Why does batch norm standardize with sample mean/variance, when it also learns parameters to scale the mean/variance?
UserDisplayName: user3667125
ThreadText: # Why does batch norm standardize with sample mean/variance, when it also learns parameters to scale the mean/variance?
  
  (Asked by: username_0 on 2021-01-13)
  
  Batch norm is a normalizing layer that is shown to help deep networks learn faster and with higher generalization accuracy. It normalizes the activations of the previous layer to a mean $\beta$ and variance $\gamma^2$ to prevent things like activations from exploding or shifting during the learning process.
  
  More specifically:
  $$\hat{x} = \displaystyle \frac{x - \mu\_t}{\sqrt{\sigma\_t^2 + \epsilon}}\label{1}\tag{1}$$
  $$ BatchNorm\_{\mu\_t, \sigma\_t}(x) = \gamma \hat{x} + \beta \label{2}\tag{2}$$
  
  where
  
  * $x$ is the layer input of the layer
  * $\mu\_t, \sigma\_t$ is the sample mean and standard deviation at time step $t$
  * $\epsilon$ is a small constant, and
  * $\gamma$ and $\beta$ are learnable parameters so that the output is not necessarily standardized to mean $0$ and variance $1$, but possibly to another mean and variance that may be better for the neural network.
  
  My question is, why does BatchNorm first standardize the input $x$ to $\hat{x}$ before applying the learnable parameters $\gamma$ and $\beta$? Isn't this redundant? The parameters $\gamma$ and $\beta$ could learn to standardize the input themselves right?
  
  In fact, as training progresses, $\mu\_t$ and $\sigma\_t$ becomes updated to new values $\mu\_{t+1}$ and $\sigma\_{t+1}$, so the learned parameters at that time step, $\gamma\_t$ and $\beta\_t$, no longer apply for time step $t+1$ since that involves a different standardization process with a different mean and variance. So by adding this standardization step, it may even hurt the convergence of the layer during learning, since it is adding the gradient of $BatchNorm\_{\mu\_{t+1}, \sigma\_{t+1}}(x)$ to $BatchNorm\_{\mu\_t, \sigma\_t}(x)$, which are two different functions right?
  
  Why not just simply make it like this?
  
  $$BatchNorm(x) = \gamma x + \beta \label{3}\tag{3}$$
  
  This would simplify the calculation of the gradients, which would make learning faster to compute.
  
  BatchNorm is one of the most successful developments of deep learning, so I know my intuition on these things is wrong -- I'm just curious as to what I am missing.
  
  **Tags:** neural-networks, deep-learning, batch-normalization, normalisation, standardisation
  
  > **Comments:**
  > **username_1**: That second formula reminds me of the _re-parametrization trick_ (used e.g. in VAEs). Not sure if it's related or not, though, because I'm not currently familiar with the details of batch normalization. So, you could investigate this option.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-11-30. Score: 0
  
  Theoretically, Yes and batchnorm can be omitted. Practically, No. Most of the intermediate features aren't normalized. For example, after a 2x2 Max pool of a normalized feature, the SD and mean would be vastly shifted to 0.8 and 1.2 (I don't remember the exact number) of the originals. It will take an immense amount of iterations for subsequent learnable parameters to absorb these scales and offsets. Before the absorption happens, those low SD feature map will be suppressed and lowering the overall network capacity. Other parameters are trained according the condition where some features are suppressed. The overall dynamics of unnormalized features are unpredictable. Batch Norm first normalize x, then the learnable gamma and beta are updated under the same scale of other normalized features.
  
  
  ---
  
  