Answers.Body: <p>Your intuition is correct. We will be normalizing the inputs of the layer under consideration (just right before applying the activation function). </p>
  
  <p>So, if this layer receives an input <span class="math-container">$\mathrm{x}=\left(x^{(1)} \ldots x^{(d)}\right)$</span>, the formula for normalizing the <span class="math-container">$k^{th}$</span> dimension of <span class="math-container">$\mathrm{x}$</span> would look as follows:
  <span class="math-container">$$\widehat{x}^{(k)}=\frac{x^{(k)}-\mathrm{E}\left[x^{(k)}\right]}{\sqrt{\operatorname{Var}\left[x^{(k)}\right]}}$$</span></p>
  
  <p>Note that in practice a constant <span class="math-container">$\epsilon$</span> is also added under the square root in the denominator to ensure stability. </p>
  
  <p>Source: The original <a href="https://arxiv.org/pdf/1502.03167.pdf" rel="nofollow noreferrer">Batch Normalization paper</a> (Section 3). </p>
  
  <p>Andrew Ng's <a href="https://www.youtube.com/watch?v=em6dfRxYkYU" rel="nofollow noreferrer">video</a> on this topic might also be useful for illustration. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-08T12:45:13.877Z
Answers.Text: Thanks for clarifying this! it was actually the original paper I read that confused me.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-08T17:26:07.183Z
Answers.Text: Right. There are a few locations in the text where they define it in an explicit manner (like in the abstract or when they define $x$ ). In some places, they may also refer to the process as “normalizing the activations”.
Answers.UserDisplayName: user5093249
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-08T12:42:52.353Z
Answers.LastActivityDate: 2020-06-08T17:26:43.317Z
Answers.LastEditDate: 2020-06-08T17:26:43.317Z
Answers.UserDisplayName: user5093249
Body: <p>In Batch Normalisation, are the sample mean and standard deviation we normalise by the mean/sd of the original data put into the network, or of the inputs in the layer we are currently BN'ing over? </p>
  
  <p>For instance, suppose I have a mini-batch size of 2 which contains <span class="math-container">$\textbf{x}_1, \textbf{x}_2$</span>. Suppose now we are at the <span class="math-container">$k$</span>th layer and the outputs from the previous layer are <span class="math-container">$\tilde{\textbf{x}}_1,\tilde{\textbf{x}}_2$</span>. When we perform batch norm at this layer would be subtract the sample mean of <span class="math-container">$\textbf{x}_1, \textbf{x}_2$</span> or of <span class="math-container">$\tilde{\textbf{x}}_1,\tilde{\textbf{x}}_2$</span>? </p>
  
  <p>My intuition tells me that it must be the mean,sd of <span class="math-container">$\tilde{\textbf{x}}_1,\tilde{\textbf{x}}_2$</span> otherwise I don't think it would be normalised to have 0 mean and sd of 1. </p>
  
Tags: <neural-networks><deep-learning><batch-normalization>
Title: In Batch Normalisation, are $\hat{\mu}$, $\hat{\sigma}$ the mean and stdev of the original mini-batch or of the input into the current layer?
UserDisplayName: David
ThreadText: # In Batch Normalisation, are $\hat{\mu}$, $\hat{\sigma}$ the mean and stdev of the original mini-batch or of the input into the current layer?
  
  (Asked by: username_0 on 2020-06-07)
  
  In Batch Normalisation, are the sample mean and standard deviation we normalise by the mean/sd of the original data put into the network, or of the inputs in the layer we are currently BN'ing over?
  
  For instance, suppose I have a mini-batch size of 2 which contains $\textbf{x}\_1, \textbf{x}\_2$. Suppose now we are at the $k$th layer and the outputs from the previous layer are $\tilde{\textbf{x}}\_1,\tilde{\textbf{x}}\_2$. When we perform batch norm at this layer would be subtract the sample mean of $\textbf{x}\_1, \textbf{x}\_2$ or of $\tilde{\textbf{x}}\_1,\tilde{\textbf{x}}\_2$?
  
  My intuition tells me that it must be the mean,sd of $\tilde{\textbf{x}}\_1,\tilde{\textbf{x}}\_2$ otherwise I don't think it would be normalised to have 0 mean and sd of 1.
  
  **Tags:** neural-networks, deep-learning, batch-normalization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-08. Score: 1 | Accepted answer
  
  Your intuition is correct. We will be normalizing the inputs of the layer under consideration (just right before applying the activation function).
  
  So, if this layer receives an input $\mathrm{x}=\left(x^{(1)} \ldots x^{(d)}\right)$, the formula for normalizing the $k^{th}$ dimension of $\mathrm{x}$ would look as follows:
  $$\widehat{x}^{(k)}=\frac{x^{(k)}-\mathrm{E}\left[x^{(k)}\right]}{\sqrt{\operatorname{Var}\left[x^{(k)}\right]}}$$
  
  Note that in practice a constant $\epsilon$ is also added under the square root in the denominator to ensure stability.
  
  Source: The original [Batch Normalization paper](https://arxiv.org/pdf/1502.03167.pdf) (Section 3).
  
  Andrew Ng's [video](https://www.youtube.com/watch?v=em6dfRxYkYU) on this topic might also be useful for illustration.
  
  > **Comments:**
  > **username_0**: Thanks for clarifying this! it was actually the original paper I read that confused me.  
  > **username_1**: Right. There are a few locations in the text where they define it in an explicit manner (like in the abstract or when they define $x$ ). In some places, they may also refer to the process as “normalizing the activations”.  
  
  ---
  
  