Why not perform weight decay on layernorm/embedding? I am learning the code of minGPT. In the function, the author excluded layernorm and embedding layer from experiencing weight decay and I want to know the reasons. Besides, what about batchnorm?

The mechanism of weight decay seems to be not clearly understood in the research field. For example, a research paper [1] reported that "the regularization effect was concentrated in the BN layer.

As evidence, we found that almost all of the regularization effect of weight decay was due to applying it to layers with BN (for which weight decay is meaningless).

The reason why such an implementation is widely used in the first place might be that Google's public BERT implementation[2] and any other pioneer's works did so. However, since this trick is not described in the paper[3], I don't think there is clear evidence for it.
Therefore, I believe think there is no evidence beyond "that's what the pioneers did."

Note that, as discussed in the forum[4], the reason for excluding weight decay from updating Layer norm and bias might be based on the paper[5], where the author states "when Batch/Layer/Weight Normalization and weight decay are used together, weight decay regularization is not expected to be effective". However, this paper only points out that "setting |w|=1 avoids variation in the effective learning rate" and does not state whether weight decay should be disabled in the normalization layer.
Reference

*

*[1] https://arxiv.org/abs/1810.12281

*[2] https://github.com/google-research/bert/blob/master/optimization.py#L65

*[3] https://arxiv.org/abs/1908.08962

*[4] https://discuss.pytorch.org/t/weight-decay-in-the-optimizers-is-a-bad-idea-especially-with-batchnorm

*[5] https://arxiv.org/abs/1706.05350
