Answers.Body: <p>I think the problems with batch normalization arise mainly due to the large misunderstanding of what it does, combined with a simple but confusing name.</p>
  <p>Despite the simple computation (everybody knows what normalization is), it is not so clear why batch normalization provides the benefits it does. In the original paper the main explanation given was that batch normalization reduced covariat shift for the parameters in very deep layers, but there are many papers out there who proved already that batch normalization also smooth classic loss functions, which means it can be considered as a regularization layer. And indeed some papers pointed out that combining batch norm with other regularization techniques like dropout doesn't really provide any benefit. You can find all the papers link in the first answer to this <a href="https://ai.stackexchange.com/questions/17228/how-does-a-batch-normalization-layer-work">question</a>.</p>
  <p>So this already answer how you can shoot yourself in the foot with batch normalization, said that it's hard to say a priori when you should use it or not, unfortunately like many other aspects of deep learning this is still a matter of trials and errors.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-26T10:34:42.533Z
Answers.LastActivityDate: 2022-10-26T10:34:42.533Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>Andrej Karpathy <a href="https://youtu.be/P6sfmUTpUmc?t=4620" rel="nofollow noreferrer">said</a> he's shot himself in foot over and over gain with batch norm.</p>
  <blockquote>
  <p>I mentioned no one likes this layer. It causes a huge amount of bugs and intuitively it's because it is coupling examples in the forward pass of a neural net and <strong>I've shot myself in the foot with this layer over and over again in my life</strong> and I don't want you to suffer the same so basically try to avoid it as much as possible. Some of the other alternatives to these layers are for example group normalization or layer normalization.</p>
  </blockquote>
  <p>How exactly can you shoot yourself in the foot with batch norm?</p>
  
Tags: <neural-networks><batch-normalization>
Title: What can go wrong using batch norm?
UserDisplayName: Tom Huntington
ThreadText: # What can go wrong using batch norm?
  
  (Asked by: username_0 on 2022-10-26)
  
  Andrej Karpathy [said](https://youtu.be/P6sfmUTpUmc?t=4620) he's shot himself in foot over and over gain with batch norm.
  
  > I mentioned no one likes this layer. It causes a huge amount of bugs and intuitively it's because it is coupling examples in the forward pass of a neural net and **I've shot myself in the foot with this layer over and over again in my life** and I don't want you to suffer the same so basically try to avoid it as much as possible. Some of the other alternatives to these layers are for example group normalization or layer normalization.
  
  How exactly can you shoot yourself in the foot with batch norm?
  
  **Tags:** neural-networks, batch-normalization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-10-26. Score: 3
  
  I think the problems with batch normalization arise mainly due to the large misunderstanding of what it does, combined with a simple but confusing name.
  
  Despite the simple computation (everybody knows what normalization is), it is not so clear why batch normalization provides the benefits it does. In the original paper the main explanation given was that batch normalization reduced covariat shift for the parameters in very deep layers, but there are many papers out there who proved already that batch normalization also smooth classic loss functions, which means it can be considered as a regularization layer. And indeed some papers pointed out that combining batch norm with other regularization techniques like dropout doesn't really provide any benefit. You can find all the papers link in the first answer to this [question](https://ai.stackexchange.com/questions/17228/how-does-a-batch-normalization-layer-work).
  
  So this already answer how you can shoot yourself in the foot with batch normalization, said that it's hard to say a priori when you should use it or not, unfortunately like many other aspects of deep learning this is still a matter of trials and errors.
  
  
  ---
  
  