THEORETICAL ANALYSIS OF AUTO RATE-TUNING BY BATCH NORMALIZATION
Batch Normalization (BN) has become a cornerstone of deep learning across diverse architectures, appearing to help optimization as well as generalization. While the idea makes intuitive sense, theoretical analysis of its effectiveness has been lacking. Here theoretical support is provided for one of its conjectured properties, namely, the ability to allow gradient descent to succeed with less tuning of learning rates. It is shown that even if we fix the learning rate of scale-invariant parameters (e.g., weights of each layer with BN) to a constant (say, 0.3), gradient descent still approaches a stationary point (i.e., a solution where gradient is zero) in the rate of T −1/2 in T iterations, asymptotically matching the best bound for gradient descent with well-tuned learning rates. A similar result with convergence rate T −1/4 is also shown for stochastic gradient descent.
* Description
The work is motivated by the empirical performance of Batch Normalization and in particular the observed better robustness of the choice of the learning rate.  Authors analyze theoretically the asymptotic convergence rate for objectives involving normalization, not necessarily BN, and show that for scale-invariant groups of parameters (appearing as a result of normalization) the initial learning rate may be set arbitrary while still asymptotic convergence is guaranteed with the same rate as the best known in the general case. Offline gradient descent and stochastic gradient descent cases are considered.
* Strengths
The work addresses better theoretical understanding of successful heuristics in deep learning, namely batch normalization and other normalizations. The technical results obtained are non-trivial and detailed proofs are presented. Also I did not verify the proofs the paper appears technically correct and technically clear. The result may be interpreted in the following form: if one chooses to use BN or other normalization, the paper gives a recommendation that only the learning rate of scale-variant parameters need to be set, which may have some practical advantages. Perhaps more important than the rate of convergence, is the guarantee that the method will not diverge (and will not get stuck in a non-local minimum). 
* Criticism
This paper presents non-trivial theoretical results that are worth to be published but as I argue below its has a weak relevance to practice and the applicability of the obtained results is unclear.
-- Concerns regarding the clarity of presentation and interpretation of the results.
 
The properties of BN used as motivation for the study, are observed non-asymptotically with constant or empirically decreased learning rate schedules for a limited number of iterations. In contrast, the studied learning rates are asymptotic and there is a big discrepancy. SGD is observed to be significantly faster than batch gradient when far from convergence (experimental evidence), and this is with or without normalization. In practice, the training is stopped much before convergence, in the hope of finding solutions close to minimum with high probability. There is in fact no experimental evidence that the practical advantages of BN are relevant to the results proven. It makes a nice story that the theoretical properties justify the observations, but they may be as well completely unrelated. 
As seen from the formal construction, the theoretical results apply equally well to all normalization methods. It occludes the clarity that BN is emphasized amongst them. 
Considering theoretically, what advantages truly follow from the paper for optimizing a given function? Let’s consider the following cases.
1. For optimizing a general smooth function with all parameters forming a single scale-invariant vector. In this case, the paper proves that no careful selection of the learning rate is necessary. This result is beyond machine learning and unfortunately I cannot evaluate its merit. Is it known / not known in optimization?
2. The case of data-independent normalization (such as weight normalization).
Without normalization, we have to tune learning rate to achieve the optimal convergence. With normalization we still have to tune the learning rate (as scale-variant parameters remain or are reintroduced with each invariance to preserve the degrees of freedom), then we have to wait for the phase two of Lemma 3.2 so that the learning rate of scale-invariant parameters adapts, and from then on the optimal convergence rate can be guaranteed.
3. The case of Batch Normalization. Note that there is no direct correspondence between the loss of BN-normalized network (2) and the loss of the original network because of dependence of the normalization on the batches. In other words, there is no setting of parameters of the original network that would make its forward pass equivalent to that of BN network (2) for all batches. The theory tells the same as in case 2 above but with an additional price of optimizing a different function.
These points remain me puzzled regarding either practical or theoretical application of the result. It would be great if authors could elaborate. 
-- Difference from Wu et al. 2018
This works is cited as a source of inspiration in several places in the paper. As the submission is a theoretical result with no immediate applicability, it would be very helpful if the authors could detail the technical improvements over this related work. Note, ICLR policy says that arxiv preprints earlier than one month before submission are considered a prior art. Could the authors elaborate more on possible practical/theoretical applications?
 
* Side Notes (not affecting the review recommendation)
I believe that the claim that “BN reduces covariate shift” (actively discussed in the intro) was an imprecise statement in the original work. Instead, BN should be able to quickly adapt to the covariate shift when it occurs. It achieves this by using the parameterization in which the mean and variance statistics of neurons (the quantities whose change is called the covariate shift) depend on variables that are local to the layer (gamma, beta in (1)) rather than on the cumulative effect of all of the preceding layers.
* Revision
I took into account the discussion and the newly added experiments and increased the score. The experiments verify the proven effect and make the paper more substantial. Some additional comments about experiments follow.
Training loss plots would be more clear in the log scale.
Comparison to "SGD BN removed" is not fair because the initialization is different (application of BN re-initializes weight scales and biases). The same initialization can be achieved by performing one training pass with BN with 0 learning rate and then removing it, see e.g. Gitman, I. and Ginsburg, B. (2017). Comparison of batch normalization and weight normalization algorithms for the large-scale image classification.
The use of Glorot uniform initializer is somewhat subtle. Since BN is used, Glorot initialization has no effect for a forward pass. However, it affects the gradient norm. Is there a rationale in this setting or it is just a more tricky method to fix the weight norm to some constant, e.g. ||w||=1?