Body: <p>I am trying to understand how batch normalization (BN) works in CNNs. Suppose I have a feature map tensor <span class="math-container">$T$</span> of shape <span class="math-container">$(N, C, H, W)$</span></p>
  <p>where <span class="math-container">$N$</span> is the mini-batch size,</p>
  <p><span class="math-container">$C$</span> is the number of channels, and</p>
  <p><span class="math-container">$H,W$</span> is the spatial dimension of the tensor.</p>
  <p>Then it seems there could a few ways of going about this:</p>
  <p><strong>Method 1:</strong> <span class="math-container">$T_{n,c,x,y} :=  \gamma*\frac {T_{c,x,y} - \mu_{x,y}} {\sqrt{\sigma^2_{x,y} + \epsilon}} + \beta$</span> where <span class="math-container">$\mu_{x,y} = \frac{1}{NC}\sum_{n, c} T_{n,c,x,y}$</span> is the mean for all channels <span class="math-container">$c$</span> for each batch element <span class="math-container">$n$</span> at spatial location <span class="math-container">$x,y$</span> over the minibatch, and</p>
  <p><span class="math-container">$\sigma^2_{x,y} = \frac{1}{NC} \sum_{n, c} (T_{n, c,x,y}-\mu_{c})^2$</span> is the variance of the minibatch for all channels <span class="math-container">$c$</span> at spatial location <span class="math-container">$x,y$</span>.</p>
  <p><strong>Method 2:</strong> <span class="math-container">$T_{n,c,x,y} :=  \gamma*\frac {T_{c,x,y} - \mu_{c,x,y}} {\sqrt{\sigma^2_{c,x,y} + \epsilon}} + \beta$</span> where <span class="math-container">$\mu_{c,x,y} = \frac{1}{N}\sum_{n} T_{n,c,x,y}$</span> is the mean for a specific channels <span class="math-container">$c$</span> for each batch element <span class="math-container">$n$</span> at spatial location <span class="math-container">$x,y$</span> over the minibatch, and</p>
  <p><span class="math-container">$\sigma^2_{c,x,y} = \frac{1}{N} \sum_{n} (T_{n, c,x,y}-\mu_{c})^2$</span> is the variance of the minibatch for a channel <span class="math-container">$c$</span> at spatial location <span class="math-container">$x,y$</span>.</p>
  <p><strong>Method 3:</strong> For each channel <span class="math-container">$c$</span> we compute the mean/variance over the entire spatial values for <span class="math-container">$x,y$</span> and apply the formula as</p>
  <p><span class="math-container">$T_{n, c,x,y} := \gamma*\frac {T_{n, c,x,y} - \mu_{c}} {\sqrt{\sigma^2_{c} + \epsilon}} + \beta$</span>, where now <span class="math-container">$\mu_c = \frac{1}{NHW} \sum_{n,x,y} T_{n,c,x,y}$</span> and <span class="math-container">$\sigma^2{_c} = \frac{1}{NHW} \sum_{n,x,y} (T_{n,c,x,y}-\mu_c)^2 $</span></p>
  <p>In practice which of these methods is used (if any) are correct for?</p>
  <p>The original paper on batch normalization , <a href="https://arxiv.org/pdf/1502.03167.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1502.03167.pdf</a> , states on page 5 section 3.2, last paragraph, left side of the page:</p>
  <blockquote>
  <p>For convolutional layers, we additionally want the normalization to
  obey the convolutional property – so that different elements of the
  same feature map, at different locations, are normalized in the same
  way. To achieve this, we jointly normalize all the activations in a
  minibatch, over all locations. In Alg. 1, we let <span class="math-container">$\mathcal{B}$</span> be the set of all
  values in a feature map across both the elements of a mini-batch and
  spatial locations – so for a mini-batch of size <span class="math-container">$m$</span> and feature maps of
  size <span class="math-container">$p \times q$</span>, we use the effective mini-batch of size <span class="math-container">$m^\prime = \vert \mathcal{B} \vert = m \cdot pq$</span>. We learn a pair of parameters <span class="math-container">$\gamma^{(k)}$</span> and <span class="math-container">$\beta^{(k)}$</span> per feature map,
  rather than per activation. Alg. 2 is modified similarly, so that
  during inference the BN transform applies the same linear
  transformation to each activation in a given feature map.</p>
  </blockquote>
  <p>I'm not sure what the authors mean by &quot;per feature map&quot;, does this mean per channel?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-20T01:01:53.273Z
Comments.Text: Are you proposing the three methods or are you pulling them out of a document?  If you are pulling them from somewhere else please cite them. Also, you want the variance and not the standard deviation in the denominator.
Comments.UserDisplayName: Brian O'Donnell
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-20T01:11:26.503Z
Comments.Text: @BrianO'Donnell edited.
Comments.UserDisplayName: IntegrateThis
Tags: <convolutional-neural-networks><batch-normalization>
Title: Understanding Batch Normalization for CNNs
UserDisplayName: IntegrateThis
ThreadText: # Understanding Batch Normalization for CNNs
  
  (Asked by: username_0 on 2021-01-19)
  
  I am trying to understand how batch normalization (BN) works in CNNs. Suppose I have a feature map tensor $T$ of shape $(N, C, H, W)$
  
  where $N$ is the mini-batch size,
  
  $C$ is the number of channels, and
  
  $H,W$ is the spatial dimension of the tensor.
  
  Then it seems there could a few ways of going about this:
  
  **Method 1:** $T\_{n,c,x,y} := \gamma\*\frac {T\_{c,x,y} - \mu\_{x,y}} {\sqrt{\sigma^2\_{x,y} + \epsilon}} + \beta$ where $\mu\_{x,y} = \frac{1}{NC}\sum\_{n, c} T\_{n,c,x,y}$ is the mean for all channels $c$ for each batch element $n$ at spatial location $x,y$ over the minibatch, and
  
  $\sigma^2\_{x,y} = \frac{1}{NC} \sum\_{n, c} (T\_{n, c,x,y}-\mu\_{c})^2$ is the variance of the minibatch for all channels $c$ at spatial location $x,y$.
  
  **Method 2:** $T\_{n,c,x,y} := \gamma\*\frac {T\_{c,x,y} - \mu\_{c,x,y}} {\sqrt{\sigma^2\_{c,x,y} + \epsilon}} + \beta$ where $\mu\_{c,x,y} = \frac{1}{N}\sum\_{n} T\_{n,c,x,y}$ is the mean for a specific channels $c$ for each batch element $n$ at spatial location $x,y$ over the minibatch, and
  
  $\sigma^2\_{c,x,y} = \frac{1}{N} \sum\_{n} (T\_{n, c,x,y}-\mu\_{c})^2$ is the variance of the minibatch for a channel $c$ at spatial location $x,y$.
  
  **Method 3:** For each channel $c$ we compute the mean/variance over the entire spatial values for $x,y$ and apply the formula as
  
  $T\_{n, c,x,y} := \gamma\*\frac {T\_{n, c,x,y} - \mu\_{c}} {\sqrt{\sigma^2\_{c} + \epsilon}} + \beta$, where now $\mu\_c = \frac{1}{NHW} \sum\_{n,x,y} T\_{n,c,x,y}$ and $\sigma^2{\_c} = \frac{1}{NHW} \sum\_{n,x,y} (T\_{n,c,x,y}-\mu\_c)^2 $
  
  In practice which of these methods is used (if any) are correct for?
  
  The original paper on batch normalization , <https://arxiv.org/pdf/1502.03167.pdf> , states on page 5 section 3.2, last paragraph, left side of the page:
  
  > For convolutional layers, we additionally want the normalization to
  > obey the convolutional property – so that different elements of the
  > same feature map, at different locations, are normalized in the same
  > way. To achieve this, we jointly normalize all the activations in a
  > minibatch, over all locations. In Alg. 1, we let $\mathcal{B}$ be the set of all
  > values in a feature map across both the elements of a mini-batch and
  > spatial locations – so for a mini-batch of size $m$ and feature maps of
  > size $p \times q$, we use the effective mini-batch of size $m^\prime = \vert \mathcal{B} \vert = m \cdot pq$. We learn a pair of parameters $\gamma^{(k)}$ and $\beta^{(k)}$ per feature map,
  > rather than per activation. Alg. 2 is modified similarly, so that
  > during inference the BN transform applies the same linear
  > transformation to each activation in a given feature map.
  
  I'm not sure what the authors mean by "per feature map", does this mean per channel?
  
  **Tags:** convolutional-neural-networks, batch-normalization
  
  > **Comments:**
  > **username_1**: Are you proposing the three methods or are you pulling them out of a document?  If you are pulling them from somewhere else please cite them. Also, you want the variance and not the standard deviation in the denominator.  
  > **username_0**: @username_1 edited.  
  
  ---
  
  ## 0 answers
  
  