Despite its empirical success, the theoretical underpinnings of the stability, convergence and acceleration properties of batch normalization (BN) remain elusive. In this paper, we attack this problem from a modeling approach, where we perform a thorough theoretical analysis on BN applied to a simplified model: ordinary least squares (OLS). We discover that gradient descent on OLS with BN has interesting properties, including a scaling law, convergence for arbitrary learning rates for the weights, acceleration effects, as well as insensitivity to the choice of learning rates. We then demonstrate numerically that these findings are not specific to the OLS problem and hold qualitatively for more complex supervised learning problems. This points to a new direction towards uncovering the mathematical principles that underlies batch normalization.
The author analyze the convergence properties of batch normalization for the ordinary least square (OLS) objective. They also provide experimental results on the OLS objective as well as small scale neural networks. First of all, understanding the properties of batch normalization is an important topic in the machine learning community so in that sense, contributions that tackle this problem are of interest for the community. However, this paper has a significant number of problems that need to be addressed before publication, perhaps the most important one being the overlap with prior work. Please address this point clearly in your rebuttal.
1) Overlap with Kolher et al. 2018: The authors erroneously state that Kolher et al. considered the convergence properties of BNGD on linear networks while after taking a close look at their analysis, they first derive an analysis for least-squares and then also provide an extension of their analysis to perceptrons. The major problem is that this paper does not correctly state the difference between their analysis and Kolher et al who already derived similar results for OLS. I will come back to this aspect multiple times below.
2) Properties of the minimizer
The authors should clearly state that Kolher et al. first proved that a^* and w^* have similar properties to Eq. 8. If I understand correctly, the difference seem to be that the algorithm analyzed in Kohler relies on the optimal a^* while the analysis presented here alternates between optimizing a and w. Is this correct? Is there any advantage in not using a^*? I think this would be worth clarifying.
3) Scaling property
I find this section confusing. Specifically,
a) The authors say they rely on this property in the proof but it is not very clear why this is beneficial. Can you please elaborate?
b) It seems to me this scaling property is also similar to the analysis of Kolher et al. who showed that the reparametrized OLS objective yields a Rayleigh quotient objective. Can you comment on this?
c) The idea of “restarting” is not clear to me, are you saying that one the magnitude of the vector w goes above a certain threshold, then one can rescale the vector therefore going back to what you called an equivalent representation? I don’t see why the text has to make this part so unclear. Looking at the proof of Theorem 3.3, this “property” seem to be used to simply rescale the a and w parameters.
d) The authors claim that “the scaling law (Proposition 3.2) should play a significant role” to extend the analysis to more general models. This requires further explanation, why would this help for say neural networks or other more complex models?
4) Convergence rate
It seems to me that the results obtained in this paper are weaker than previous known results, I would have liked to see a discussion of these results. Specifically,
a) Theorem 3.3 is an asymptotic convergence result so it is much weaker than the linear rate of convergence derived in Kolher et al. The authors require a sufficiently small step size. Looking at the analysis of Kolher et al., they show that the reparametrized OLS objective yields a Rayleigh quotient objective. Wouldn’t a constant step size also yield convergence in that case?
b) Proposition 3.4 also only provides a local convergence rate. The authors argue BNGD could have a faster convergence. This does seem to again be a weaker result. So again, I think it would be very beneficial if the authors could clearly state the differences with previous work.
5) Saddles for neural nets
The authors claim they “have not encountered convergence to saddles” for the experiments with neural networks. How did you check whether the limit point reached by BNGD was not a saddle point? This requires computing all the eigenvalues of the Hessian which is typically expensive. How was this done exactly?
6) Extension of the analysis to deep neural networks
The analysis provided in this paper only applies to OLS while Kolher et al. also derived an analysis for neural networks. Can the authors comment on extending their own analysis to neural nets and how this would differ from the one derived in Kolher et al.?
7) Experiments
How would you estimate the range of suitable step sizes (for both a and w) for BNGD for a neural network?