Why not perform weight decay on layernorm/embedding? I am learning the code of minGPT. In the function, the author excluded layernorm and embedding layer from experiencing weight decay and I want to know the reasons. Besides, what about batchnorm?

After some searching, I find some explanations:
karpathy's explanation and a disscusion in the pytorch forum.

@morganmcg1 the purpose of L2 regularization is to "spread out" the weights in dot products, ensuring that more "independent measurements" (dimensions of the input) get used more equally, instead of any one feature dominating the computation. This only makes sense for matrix multiply layers, which the embeddings and the layernorm parameters are not. L2 regularization is covered eg in CS231n.

