Body: <p>Let us assume your dataset has <span class="math-container">$n$</span> training samples each of size <span class="math-container">$s$</span> and you divided them into <span class="math-container">$k$</span> batches for training. Then each batch has <span class="math-container">$n_k = \dfrac{n}{k}$</span> training samples.</p>
  <p>Batch normalization can be applied to <a href="https://ai.stackexchange.com/questions/28744/where-does-batch-normalization-layers-present-in-a-neural-network#comment45697_28744">any input or hidden layer</a> in a neural network. So, assume that I am applying batch normalization at every possible place I can.</p>
  <p>Now, consider a particular batch normalization layer (say <span class="math-container">$b$</span>) of a hidden layer <span class="math-container">$\ell$</span>. Now, I am confused about the working frequency of <span class="math-container">$b$</span>.</p>
  <p>Will it be activated only after every <span class="math-container">$n_k - 1$</span> forward passes i.e, once per batch at the end of the batch? If no, then how <span class="math-container">$b$</span> calculates the mean and standard deviation for every forward pass while training if <span class="math-container">$n_k$</span> output vectors of <span class="math-container">$\ell$</span> are not available at that instant?</p>
  <p>Will <span class="math-container">$b$</span> calculates the mean and standard deviated, for every forward pass, based on the outputs of <span class="math-container">$\ell$</span> that are calculated so far? If yes, then why it is called <strong>batch</strong> normalization?</p>
  <p><strong>To put it concisely, are batch normalization layers active for every iteration? If yes then how they are normalizing a &quot;batch&quot; of vectors?</strong></p>
  <hr />
  <p>You can check <a href="https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html" rel="nofollow noreferrer">here</a> which says</p>
  <blockquote>
  <p>The mean and standard-deviation are calculated per-dimension over the
  <strong>mini-batches</strong></p>
  </blockquote>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-31T19:30:27.747Z
Comments.Text: If I understand your question correctly, I think this is related to the momentum which calculates the moving average, hence it can be applied at every iteration. [More Details](https://keras.io/api/layers/normalization_layers/batch_normalization/).
Comments.UserDisplayName: Yahya
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-31T23:39:45.177Z
Comments.Text: @Yahya Does the word "batch" here also need to be interpreted as the "collection" rather than collection of $\dfrac{n}{k}$ samples?
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-31T23:45:29.213Z
Comments.Text: @Yahya you can see that "the layer normalizes its output using the mean and standard deviation of the **current batch of inputs**" this statement says that we use mean and standard deviation on a batch. Even we use moving averages, the number of samples we are considering for normalization depends on number of iterations occurred till now. So, batch in batch normalization also refers to the collection of vectors we encountered till now.
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-31T23:47:55.780Z
Comments.Text: And it is also useful to check [here](https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html) which says **The mean and standard-deviation are calculated per-dimension over the mini-batches**. The word mini-batches here imposes the requirement that at-least mini batch number of vectors should be available to proceed. Am i wrong?
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-31T23:49:31.190Z
Comments.Text: @Yahya I am opining that the words batch and mini-batch are used loosely If they are considering moving averages at every forward pass.
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-01T12:54:15.283Z
Comments.Text: As far as I understand it, it is always the **mini-batch** that the mean and STD are derived from. Yet, the mini-batch can be as large as the whole dataset, hence they are using the two terms (batch and mini-batch) interchangeably. It's even required statistically to have a relatively large mini-batch to get representative mean and STD. Moreover, at inference time, we need the mean and STD of the whole dataset, (continue in next comment).
Comments.UserDisplayName: Yahya
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-01T12:55:52.507Z
Comments.Text: but since we are using mini-batches, we calculate the **moving** average and STD at every call of the layer during training, this is valid regardless of the vectorization, ..well ...because it is a *moving* measure. Still, how many mini-batches should be involved to calculate the latter? Here we introduce the momentum to decide the history that should be involved, where 0.0 means calculate the *moving* average and STD only for the very last mini-batch (hence highly biased), and 1 is for the very first mini-batch.
Comments.UserDisplayName: Yahya
Tags: <neural-networks><deep-learning><batch-normalization>
Title: When does a batch normalization layer becomes active?
UserDisplayName: hanugm
ThreadText: # When does a batch normalization layer becomes active?
  
  (Asked by: username_0 on 2021-07-30)
  
  Let us assume your dataset has $n$ training samples each of size $s$ and you divided them into $k$ batches for training. Then each batch has $n\_k = \dfrac{n}{k}$ training samples.
  
  Batch normalization can be applied to [any input or hidden layer](https://ai.stackexchange.com/questions/28744/where-does-batch-normalization-layers-present-in-a-neural-network#comment45697_28744) in a neural network. So, assume that I am applying batch normalization at every possible place I can.
  
  Now, consider a particular batch normalization layer (say $b$) of a hidden layer $\ell$. Now, I am confused about the working frequency of $b$.
  
  Will it be activated only after every $n\_k - 1$ forward passes i.e, once per batch at the end of the batch? If no, then how $b$ calculates the mean and standard deviation for every forward pass while training if $n\_k$ output vectors of $\ell$ are not available at that instant?
  
  Will $b$ calculates the mean and standard deviated, for every forward pass, based on the outputs of $\ell$ that are calculated so far? If yes, then why it is called **batch** normalization?
  
  **To put it concisely, are batch normalization layers active for every iteration? If yes then how they are normalizing a "batch" of vectors?**
  
  ---
  
  You can check [here](https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html) which says
  
  > The mean and standard-deviation are calculated per-dimension over the
  > **mini-batches**
  
  **Tags:** neural-networks, deep-learning, batch-normalization
  
  > **Comments:**
  > **username_1**: If I understand your question correctly, I think this is related to the momentum which calculates the moving average, hence it can be applied at every iteration. [More Details](https://keras.io/api/layers/normalization_layers/batch_normalization/).  
  > **username_0**: @username_1 Does the word "batch" here also need to be interpreted as the "collection" rather than collection of $\dfrac{n}{k}$ samples?  
  > **username_0**: @username_1 you can see that "the layer normalizes its output using the mean and standard deviation of the **current batch of inputs**" this statement says that we use mean and standard deviation on a batch. Even we use moving averages, the number of samples we are considering for normalization depends on number of iterations occurred till now. So, batch in batch normalization also refers to the collection of vectors we encountered till now.  
  > **username_0**: And it is also useful to check [here](https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html) which says **The mean and standard-deviation are calculated per-dimension over the mini-batches**. The word mini-batches here imposes the requirement that at-least mini batch number of vectors should be available to proceed. Am i wrong?  
  > **username_0**: @username_1 I am opining that the words batch and mini-batch are used loosely If they are considering moving averages at every forward pass.  
  > **username_1**: As far as I understand it, it is always the **mini-batch** that the mean and STD are derived from. Yet, the mini-batch can be as large as the whole dataset, hence they are using the two terms (batch and mini-batch) interchangeably. It's even required statistically to have a relatively large mini-batch to get representative mean and STD. Moreover, at inference time, we need the mean and STD of the whole dataset, (continue in next comment).  
  > **username_1**: but since we are using mini-batches, we calculate the **moving** average and STD at every call of the layer during training, this is valid regardless of the vectorization, ..well ...because it is a *moving* measure. Still, how many mini-batches should be involved to calculate the latter? Here we introduce the momentum to decide the history that should be involved, where 0.0 means calculate the *moving* average and STD only for the very last mini-batch (hence highly biased), and 1 is for the very first mini-batch.  
  
  ---
  
  ## 0 answers
  
  