Michael_Oliver:

Correct me if I’m wrong, but there is no reason the beta and gamma parameters in BatchNorm should ever be subject to weight decay, ie L2 regularization, that pulls them toward 0. In fact it seems like a very bad idea to pull them toward 0.


This finding is also backed by this publication:

  
      
      arXiv.org
  
  
    

L2 Regularization versus Batch and Weight Normalization 264

Batch Normalization is a commonly used trick to improve the training of deep
neural networks. These neural networks use L2 regularization, also called
weight decay, ostensibly to prevent overfitting. However, we show that L2
regularization has no...


  
  
    
    
  
  


While I very much appreciate the code snippet provided by @rwightman , I too would like to see at least an option in PyTorch to exclude normalization layers from weight decay.