Batch Normalization (BN) and its variants have seen widespread adoption in the deep learning community because they improve the training of deep neural networks. Discussions of why this normalization works so well remain unsettled. We make explicit the relationship between ordinary least squares and partial derivatives computed when back-propagating through BN. We recast the backpropagation of BN as a least squares fit, which zero-centers and decorrelates partial derivatives from normalized activations. This view, which we term gradientleast-squares, is an extensible and arithmetically accurate description of BN. To further explore this perspective, we motivate, interpret, and evaluate two adjustments to BN.
The primary technical contribution comes from Section 2, where it is demonstrated that the normalized back-propagated gradients obtained from a BN layer can be viewed as the residuals of the gradients obtained without BN regressed via a simple two-parameter model of the activations.  In some sense though this result is to be expected, since centering data (i.e., removing the mean as in BN) can be generically viewed as computing the residuals after a least squares fit of a single constant, and similarly for de-trending with respect to a single independent variable, in this case the activations.  So I'm not sure that Theorem 1 is really that much of an insightful breakthrough, even if it may be nice to work through the precise details in the specific case of a BN layer and the relationship to gradients.
But beyond this a larger issue is as follows:  This paper is framed as taking a step in explaining why batch normalization (BN) works so well.  For example, even the abstract mentions this as an unsettled issue in motivating the proposed analysis.  However, to me the interpretation of BN as introducing a form of least squares fit does not really extend our understanding of why it actually works better in practice, and this is the biggest disconnect of the paper.  The new perspective presented might be another way to interpret BN layers, but it unfortunately remains mostly unanswered exactly why this new perspective is relevant in actually explaining BN behavior.
The presented normalization theory is also used to motivate heuristic modifications to standard BN schemes.  For example, the paper proposed concatenating BN with a layer normalization layer, demonstrating some modest improvement on CIFAR-10 data.  But again, I don't see how viewing these normalization schemes as least-squares residuals motivates such concatenation any more than the merits of the original versions themselves.  Moreover, it is not even clear that BN+LN is in fact generally better since only a single data set is considered.  There are also no comparisons against competing BN modifications such as switch normalization (Luo et al. 2018) which also involves a hybrid method combining aspects of LN and BN.  Why not compare against approaches like this?
To conclude, in Section 6 the paper asks "Why do empirical improvements in neural networks with BN keep the gradient-least-squares residuals and drop the explained portion?"  But this question is not at all answered but rather deferred to future work.  For me this was a disappointment as this would seem to be an essential ingredient for actually developing a meaningful theory for why BN is helpful in practice.
Other comments:
* The analysis from Section 2, including Theorem 1, assume that the BN parameters c and b can be ignored (presumably this means fixing c = 1 and b = 0).  I did not carefully check the details, but do all the same derivations and conclusions still seamlessly go through when these parameters have general values that deviate from this standard initialization?  If not, then I don't really see what is the practical relevance, since once learning begins, both b and c will typically shift to arbitrary values.  Below eq. (1) it states that c and b are only ignored for clarity, but then later I did not see any subsequent discussion to handle the general case, which is what would be actually needed for explaining BN behavior in practice.
* Please run a speck-checker.  Example, "On some leve, the matrix gradient ..."
* The paper cites (Lipton and Steinhardt, 2018) in arguing that reasons for the effectiveness of BN are lacking.  Indeed (Lipton and Steinhardt, 2018) criticize the original BN paper for conflating speculation with explanation, or more precisely, framing speculation about why BN should be helpful as an actual true explanation without clear evidence.  But to me this submission is hovering somewhere in the same category, speculating that regressing away certain portions of the gradient could be useful but never really providing concrete evidence for why this should offer an improvement.