Answers.Body: <p>In the literature, it differs. You will see models do it after or before pooling only, and sometimes you see it after every single convolution.   </p>
  
  <p>Batch normalization's assistance to neural networks wasn't really understood for the longest time, initially it was thought to assist with internal covariate shift (hypothesized by the initial paper: <a href="https://arxiv.org/abs/1502.03167" rel="nofollow noreferrer">Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift</a>) but lately has been tied to the optimization process (<a href="https://arxiv.org/pdf/1805.11604.pdf" rel="nofollow noreferrer">How Does Batch Normalization Help Optimization?</a>). </p>
  
  <p>This means, from an architectural perspective, it is difficult to correctly assume how it should be utilized, unless you really understand its impact on the loss landscape and how your optimization process will traverse it given some initialization (Note, by the way, a recent paper by Google showed that you can alleviate a lot of the benefits of batch normalization sheerly by understanding what issues it's resolving and attempting to mitigate them in the initialization process: <a href="https://arxiv.org/abs/1901.09321" rel="nofollow noreferrer">Fixup Initialization</a>). </p>
  
  <p>So I would recommend 3 things until it is more understood how to utilize it generally:</p>
  
  <ol>
  <li>Play around, get <em>frisky</em> and experiment. Use what works best.  </li>
  <li>Use block featurizers that are known to work well like residual blocks. Proven in practice and will probably work for you too.  </li>
  <li>Do the research and investigate it more, if you find the answer, you'll be helping a lot of people :)</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-11T21:14:55.687Z
Answers.LastActivityDate: 2019-07-11T21:46:06.870Z
Answers.LastEditDate: 2019-07-11T21:46:06.870Z
Answers.UserDisplayName: mshlis
Body: <p>For a deep NN, should I generally apply batch normalization after each convolution layer? Or only after some of them?  Which? Every 2nd, every 3rd, lowest, highest, etc.?</p>
  
Tags: <deep-learning><convolutional-neural-networks><batch-normalization>
Title: What is the most common practice to apply batch normalization?
UserDisplayName: user31264
ThreadText: # What is the most common practice to apply batch normalization?
  
  (Asked by: username_0 on 2019-07-11)
  
  For a deep NN, should I generally apply batch normalization after each convolution layer? Or only after some of them? Which? Every 2nd, every 3rd, lowest, highest, etc.?
  
  **Tags:** deep-learning, convolutional-neural-networks, batch-normalization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-07-11. Score: 1
  
  In the literature, it differs. You will see models do it after or before pooling only, and sometimes you see it after every single convolution.
  
  Batch normalization's assistance to neural networks wasn't really understood for the longest time, initially it was thought to assist with internal covariate shift (hypothesized by the initial paper: [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://arxiv.org/abs/1502.03167)) but lately has been tied to the optimization process ([How Does Batch Normalization Help Optimization?](https://arxiv.org/pdf/1805.11604.pdf)).
  
  This means, from an architectural perspective, it is difficult to correctly assume how it should be utilized, unless you really understand its impact on the loss landscape and how your optimization process will traverse it given some initialization (Note, by the way, a recent paper by Google showed that you can alleviate a lot of the benefits of batch normalization sheerly by understanding what issues it's resolving and attempting to mitigate them in the initialization process: [Fixup Initialization](https://arxiv.org/abs/1901.09321)).
  
  So I would recommend 3 things until it is more understood how to utilize it generally:
  
  1. Play around, get *frisky* and experiment. Use what works best.
  2. Use block featurizers that are known to work well like residual blocks. Proven in practice and will probably work for you too.
  3. Do the research and investigate it more, if you find the answer, you'll be helping a lot of people :)
  
  
  ---
  
  