We introduce a new normalization technique that exhibits the fast convergence properties of batch normalization using a transformation of layer weights instead of layer outputs. The proposed technique keeps the contribution of positive and negative weights to the layer output in equilibrium. We validate our method on a set of standard benchmarks including CIFAR-10/100, SVHN and ILSVRC 2012 ImageNet.
This manuscript introduces a new layer-wise transform, EquiNorm, to improve upon batch normalization. As with batch normalization and related techniques, the idea is to introduce a simple linear transform at each layer to reduce the dependency of the features to the data. Unlike batch normalization, the procedure does not modify the inputs to the layers but rather the layer weights. For this purpose, a scaling factor and a shift is computed on a mini batch, separating positive and negative weights to compute easily both running estimates of shift and of spread (here in the l1 sense). The method is compared to BatchNorm and GroupNorm on several classic computer vision datasets. Empirically, the method converges faster in the beginning of the optimization, in the sense that in the first few epochs the test accuracy is higher than for BatchNorm. However, this benefit decreases with more epochs and when the results are close to peak performance the difference between methods is a small. In addition, test accuracy can decrease at the end of the optimization, which the authors interpret as a sign of increased overfit, and tackle with clever data augmentation. The paper reads well.
The strengths of the paper are the elegance of the solution proposed, and the faster convergence. To me, the main drawback of the manuscript is that the results do not show a clear benefit at convergence of EquiNorm compared to BatchNorm. It seems that as the learning rate decreases there is no consistent difference: in none of the 4 datasets does EquiNorm give a gain larger than a quartile of the distribution of scores and in 2 out of 4 it performs less well.
My advice to improve this work would be to do more experiments and to show better that the lack of performance gain is due to overfit, and can be fixed with larger data or data augmentation.
The faster initial convergence could be a real benefit of this method. However, the time to useful convergence is the same than with BatchNorm. This is probably because the learning-rate decrease schedule is the same. It might be interesting to study adapted learning-rate decrease.
To argue for increased overfit, it would have been interesting to compare test error with train error, and to plot train loss alongside with figure 2.
While the fast convergence is a benefit, it would have been interesting to also compare EquiNorm with BatchNorm with learning scale schedules (ie outside "super convergence" settings). While costly, such an experiment would help separating the benefits of EquiNorm from choice of learning rate and give a standard baseline to compare to. Ideally, EquiNorm achieves this baseline with less compute time.
I would like to congratulate the authors for reporting multiple runs with different RNGs and the quantile of the distribution. This is absolutely good practice to judge the significance of improvements and is seldom done.
Why does figure 2 shows only test loss, while the other experimental figures show test accuracy and loss?