Answers.Body: <p>I think the answer to your question is much more a rule of thumb than an appropriate analytical answer. First of all, I would like to remark that Batch Normalization [1] are applied most commonly to convolutional layer, constituting what is called a &quot;convolutional block&quot; <strong>(Convolution + Batch Normalization + Activation)</strong>. Thus, for giving you an idea on where to put the normalization layers, I will analyze three papers that make use of Batch Normalization.</p>
  <p><strong>Unsupervised representation learning with deep convolutional generative adversarial networks</strong> [2]. In Section 3, Approach and Model Architecture, the authors make the following remark:</p>
  <blockquote>
  <p>Third is Batch Normalization [1] which stabilizes learning by normalizing the
  input to each unit to have zero mean and unit variance. This helps deal with training problems that
  arise due to poor initialization and helps gradient flow in deeper models. This proved critical to get
  deep generators to begin learning, preventing the generator from collapsing all samples to a single
  point which is a common failure mode observed in GANs. Directly applying batchnorm to all layers
  however, resulted in sample oscillation and model instability. This was avoided by not applying
  batchnorm to the generator output layer and the discriminator input layer.</p>
  </blockquote>
  <p>thus, the authors have experimented with Batch Normalization on all layers, but this approach resulted in model instability. They ultimately avoided <strong>Batch Normalization on the Generator's output layer, and Discriminator's input layer</strong>. These correspond to &quot;raw image layers&quot; (the output of the generator is an image, as well as the input of the discriminator). Using your notation, it would be something like:</p>
  <p><span class="math-container">$$G: z \rightarrow (C + BN + A)_{1} \rightarrow (C + A)_{2} \rightarrow O$$</span>
  <span class="math-container">$$D: I \rightarrow (C + A)_{1} \rightarrow (C + BN + A)_{2} \rightarrow h_{1} \rightarrow h_{2} \rightarrow O$$</span></p>
  <p>where <span class="math-container">$G$</span> stands for Generator, <span class="math-container">$D$</span> for Discriminator, <span class="math-container">$C$</span> for convolution, <span class="math-container">$BN$</span> for batchnorm, <span class="math-container">$A$</span> for activation and <span class="math-container">$D_{i}$</span> for dense layers.</p>
  <p><strong>Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising</strong> [3]. In Figure 1., the authors show their proposed architecture that applies the same logic. Neither the first convolutional layer, neither the last use Batch Normalization. Although the authors make their point in justifying the usage of residual learning + Batch Normalization, the paper does not justify this architectural choice. The network would look something like this:</p>
  <p><span class="math-container">$$I \rightarrow (C + A)_{1} \rightarrow (C + BN + A)_{2} \rightarrow \cdots \rightarrow (C + BN + A)_{n} \rightarrow C \rightarrow O$$</span></p>
  <p><strong>Deep Residual Learning for Image Recognition</strong> [4]. This paper proposes a different scheme, as it applies Batch Normalization after each convolution operation on convolutional layers (thus including the input).</p>
  <p><strong>To sum up</strong> some papers use it only on the &quot;hidden convolutional blocks&quot;, others on all of them. My advice is that, if you have the time, you should compare the two approaches. <strong>Remark:</strong> maybe I am unaware of some further development on the matter that gives a precise argument in favor of whether of these approaches.</p>
  <h2 id="references">References</h2>
  <p>[1] Ioffe, S., &amp; Szegedy, C. (2015, June). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning (pp. 448-456). PMLR.</p>
  <p>[2] Radford, A., Metz, L., &amp; Chintala, S. (2015). Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434.</p>
  <p>[3] Zhang, K., Zuo, W., Chen, Y., Meng, D., &amp; Zhang, L. (2017). Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE transactions on image processing, 26(7), 3142-3155.</p>
  <p>[4] He, K., Zhang, X., Ren, S., &amp; Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-22T02:43:42.800Z
Answers.LastActivityDate: 2021-07-22T02:50:52.460Z
Answers.LastEditDate: 2021-07-22T02:50:52.460Z
Answers.UserDisplayName: Eduardo Montesuma
Body: <p>Batch normalization is a procedure widely used to train neural networks. Mean and standard deviation are calculated in this step of training.</p>
  <p>Since we train a neural network by dividing training data into batches, we use the word <strong><a href="https://ai.stackexchange.com/questions/17270/what-is-a-batch-in-batch-normalization">batch</a></strong> normalization as we consider a batch of training vectors at a time.</p>
  <p>My doubt is about the position of batch normalization layers in a neural network.</p>
  <p>Is it present before the input layer only? Or is it before every layer? Or is it dependent on the underlying task?</p>
  <p>Suppose there is a neural network of 4 layers: <span class="math-container">$I \rightarrow h1 \rightarrow h2 \rightarrow h3 \rightarrow O$</span></p>
  <p>Which one of the following is true</p>
  <p><span class="math-container">$$bn \rightarrow I \rightarrow h1 \rightarrow h2 \rightarrow h3 \rightarrow O$$</span></p>
  <p><span class="math-container">$$bn1 \rightarrow I \rightarrow bn2 \rightarrow  h1 \rightarrow bn3 \rightarrow  h2 \rightarrow bn4 \rightarrow  h3 \rightarrow bn5 \rightarrow  O$$</span></p>
  <p>Here <span class="math-container">$I$</span> stands for input layer, <span class="math-container">$h$</span> for the hidden layer, <span class="math-container">$O$</span> for output layer, and <span class="math-container">$bn$</span> for batch normalization layer.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-22T03:30:01.173Z
Comments.Text: Batch normalization can be applied to any input
  or hidden layer in a network. [p:318](https://raw.githubusercontent.com/janishar/mit-deep-learning-book-pdf/master/complete-book-pdf/Ian%20Goodfellow%2C%20Yoshua%20Bengio%2C%20Aaron%20Courville%20-%20Deep%20Learning%20(2017%2C%20MIT).pdf)
Comments.UserDisplayName: hanugm
Tags: <neural-networks><architecture><batch-normalization>
Title: Where does batch normalization layers present in a neural network?
UserDisplayName: hanugm
ThreadText: # Where does batch normalization layers present in a neural network?
  
  (Asked by: username_0 on 2021-07-19)
  
  Batch normalization is a procedure widely used to train neural networks. Mean and standard deviation are calculated in this step of training.
  
  Since we train a neural network by dividing training data into batches, we use the word **[batch](https://ai.stackexchange.com/questions/17270/what-is-a-batch-in-batch-normalization)** normalization as we consider a batch of training vectors at a time.
  
  My doubt is about the position of batch normalization layers in a neural network.
  
  Is it present before the input layer only? Or is it before every layer? Or is it dependent on the underlying task?
  
  Suppose there is a neural network of 4 layers: $I \rightarrow h1 \rightarrow h2 \rightarrow h3 \rightarrow O$
  
  Which one of the following is true
  
  $$bn \rightarrow I \rightarrow h1 \rightarrow h2 \rightarrow h3 \rightarrow O$$
  
  $$bn1 \rightarrow I \rightarrow bn2 \rightarrow h1 \rightarrow bn3 \rightarrow h2 \rightarrow bn4 \rightarrow h3 \rightarrow bn5 \rightarrow O$$
  
  Here $I$ stands for input layer, $h$ for the hidden layer, $O$ for output layer, and $bn$ for batch normalization layer.
  
  **Tags:** neural-networks, architecture, batch-normalization
  
  > **Comments:**
  > **username_0**: Batch normalization can be applied to any input or hidden layer in a network. [p:318](https://raw.githubusercontent.com/janishar/mit-deep-learning-book-pdf/master/complete-book-pdf/Ian%20Goodfellow%2C%20Yoshua%20Bengio%2C%20Aaron%20Courville%20-%20Deep%20Learning%20(2017%2C%20MIT).pdf)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-07-22. Score: 1
  
  I think the answer to your question is much more a rule of thumb than an appropriate analytical answer. First of all, I would like to remark that Batch Normalization [1] are applied most commonly to convolutional layer, constituting what is called a "convolutional block" **(Convolution + Batch Normalization + Activation)**. Thus, for giving you an idea on where to put the normalization layers, I will analyze three papers that make use of Batch Normalization.
  
  **Unsupervised representation learning with deep convolutional generative adversarial networks** [2]. In Section 3, Approach and Model Architecture, the authors make the following remark:
  
  > Third is Batch Normalization [1] which stabilizes learning by normalizing the
  > input to each unit to have zero mean and unit variance. This helps deal with training problems that
  > arise due to poor initialization and helps gradient flow in deeper models. This proved critical to get
  > deep generators to begin learning, preventing the generator from collapsing all samples to a single
  > point which is a common failure mode observed in GANs. Directly applying batchnorm to all layers
  > however, resulted in sample oscillation and model instability. This was avoided by not applying
  > batchnorm to the generator output layer and the discriminator input layer.
  
  thus, the authors have experimented with Batch Normalization on all layers, but this approach resulted in model instability. They ultimately avoided **Batch Normalization on the Generator's output layer, and Discriminator's input layer**. These correspond to "raw image layers" (the output of the generator is an image, as well as the input of the discriminator). Using your notation, it would be something like:
  
  $$G: z \rightarrow (C + BN + A)\_{1} \rightarrow (C + A)\_{2} \rightarrow O$$
  $$D: I \rightarrow (C + A)\_{1} \rightarrow (C + BN + A)\_{2} \rightarrow h\_{1} \rightarrow h\_{2} \rightarrow O$$
  
  where $G$ stands for Generator, $D$ for Discriminator, $C$ for convolution, $BN$ for batchnorm, $A$ for activation and $D\_{i}$ for dense layers.
  
  **Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising** [3]. In Figure 1., the authors show their proposed architecture that applies the same logic. Neither the first convolutional layer, neither the last use Batch Normalization. Although the authors make their point in justifying the usage of residual learning + Batch Normalization, the paper does not justify this architectural choice. The network would look something like this:
  
  $$I \rightarrow (C + A)\_{1} \rightarrow (C + BN + A)\_{2} \rightarrow \cdots \rightarrow (C + BN + A)\_{n} \rightarrow C \rightarrow O$$
  
  **Deep Residual Learning for Image Recognition** [4]. This paper proposes a different scheme, as it applies Batch Normalization after each convolution operation on convolutional layers (thus including the input).
  
  **To sum up** some papers use it only on the "hidden convolutional blocks", others on all of them. My advice is that, if you have the time, you should compare the two approaches. **Remark:** maybe I am unaware of some further development on the matter that gives a precise argument in favor of whether of these approaches.
  
  ## References
  
  [1] Ioffe, S., & Szegedy, C. (2015, June). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning (pp. 448-456). PMLR.
  
  [2] Radford, A., Metz, L., & Chintala, S. (2015). Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434.
  
  [3] Zhang, K., Zuo, W., Chen, Y., Meng, D., & Zhang, L. (2017). Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE transactions on image processing, 26(7), 3142-3155.
  
  [4] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778).
  
  
  ---
  
  