TOWARDS UNDERSTANDING REGULARIZATION IN BATCH NORMALIZATION
Batch Normalization (BN) improves both convergence and generalization in training neural networks. This work understands these phenomena theoretically. We analyze BN by using a basic block of neural networks, consisting of a kernel layer, a BN layer, and a nonlinear activation function. This basic network helps us understand the impacts of BN in three aspects. First, by viewing BN as an implicit regularizer, BN can be decomposed into population normalization (PN) and gamma decay as an explicit regularization. Second, learning dynamics of BN and the regularization show that training converged with large maximum and effective learning rate. Third, generalization of BN is explored by using statistical mechanics. Experiments demonstrate that BN in convolutional neural networks share the same traits of regularization as the above analyses.
This is a thought provoking paper that aims to understand the regularization effects of batch-normalization (BN) under a probabilistic interpretation. The authors connect BN to population normalization (PN) and a gamma-decay term that penalizes the scale of the weights. They analyze the generalization error of BN for a single-layer perceptron using ideas in statistical physics.
Detailed comments:
1. Theorem 1 uses the loss function of a single-layer perceptron in the proof. This is not mentioned in the main writeup. This theorem is not valid in general.
2. The main contribution of this paper is Theorem 1 which connects BN to population normalization and weight normalization. It shows that the regularization of BN can be split into two components that depend on the mini-batch mean and variances: the former penalizes the magnitude of activations while the latter penalizes their correlation.
3. Although the theoretical analysis is conducted under simplistic models, this paper corroborates a number of widely-known observations about BN in practice. It validates these predictions on standard experiments.
4. The scaling of BN regularization with batch-size can be easily seen from Teye et al., 2018, so I think the experiments that validate this prediction are not strictly necessary.
5. It is difficult to use these techniques for deep non-linear networks.
6. The predictions in Section 3.3 are very interesting: it is often seen that fully-connected layers (where BN helps significantly) need small learning rates to train without BN; with BN one can use larger learning rates.
7. The experimental section is very rough. In particular the experiments on CIFAR-10 and downsampled-ImageNet with CNNs seem to have very high errors and it is difficult to understand whether some of the predictions about generalization error apply here. Why not use a more recent architecture for CIFAR-10?
8. There is a very large number of grammatical and linguistic errors in the narrative.
9. The presentation of the paper is very dense, I would advise the authors to move certain parts to the appendix and remove the inlining of important equations to improve readability.