  L2 regularization for weights in neural networks is widely used as a standard
training trick. However, L2 regularization for gamma, a trainable parameter of
batch normalization, remains an undiscussed mystery and is applied in different
ways depending on the library and practitioner. In this paper, we study whether
L2 regularization for gamma is valid. To explore this issue, we consider two
approaches: 1) variance control to make the residual network behave like
identity mapping and 2) stable optimization through the improvement of
effective learning rate. Through two analyses, we specify the desirable and
undesirable gamma to apply L2 regularization and propose four guidelines for
managing them. In several experiments, we observed the increase and decrease in
performance caused by applying L2 regularization to gamma of four categories,
which is consistent with our four guidelines. Our proposed guidelines were
validated through various tasks and architectures, including variants of
residual networks and transformers.

Computer Vision and Pattern Recognition