The goal of this paper is to study why typical neural networks train so fast, and how neural network architecture affects the speed of training. We introduce a simple concept called gradient confusion to help formally analyze this. When confusion is high, stochastic gradients produced by different data samples may be negatively correlated, slowing down convergence. But when gradient confusion is low, data samples interact harmoniously, and training proceeds quickly. Through novel theoretical and experimental results, we show how the neural net architecture affects gradient confusion, and thus the efficiency of training. We show that increasing the width of neural networks leads to lower gradient confusion, and thus easier model training. On the other hand, increasing the depth of neural networks has the opposite effect. Finally, we observe empirically that techniques like batch normalization and skip connections reduce gradient confusion, which helps reduce the training burden of very deep networks.
This paper introduces the concept of "gradient confusion" to explain why neural networks train fast with SGD. They also study the effects of width, depth on gradient confusion. 
- The theoretical results assume that the data is sampled from a sphere and do not really give much insight into the effect of width and depth. 
- There are some confounding factors in the experiments and there needs to be a better comparison to some related work. 
Detailed review below:
Section 1:
- Please clarify how "gradient confusion" relate to the interpolation conditon of Ma et al, 2017 and the strong growth condition of Vaswani et al? 
- If we run SGD with a constant step-size, it will bounce around the optimal point in a ball with radius that depends on the step-size. If I keep decreasing the step-size, this radius shrinks. How does gradient confusion relate to the step-size? Is it upper-bounded by a quantity that depends on the step-size and the batch-size?
Section 2:
- Definition 2.1: Why should this condition hold for "all" points w? Isn't it necessary only at w^* or in a small neighborhood around it? 
- The gradient confusion parameter \eta should depend on the batch-size. Please clarify this. 
- Figure 1: Previous work (Ma et al, Vaswani et al, Gunasekhar, 2017) all have shown that fast convergence can be obtained using SGD with a constant step-size with over-parametrized models and explained it using interpolation. What is the additional insight from gradient confusion?
- "Suppose that there is a Lipschitz constant for the Hessian" - This is a strong assumption and a vague argument, that is confusing rather than insightful. Please justify why this is a valid assumption for neural network models.
Section 3:
- If E_i || \nabla f_i(w)  ||^2 = O(\epsilon) => gradient confusion = O(\epsilon). Isn't the E_i || \nabla f_i(w)  ||^2 exactly the strong growth condition in Vaswani, et al. Can the gradient confusion results be directly derived from the results in that paper? Please compare. Also compare and cite "Stochastic Approximation of Smooth and Strongly Convex Functions:
Beyond the O(1/T ) Convergence Rate", COLT 2019. 
Section 4:
- Please compare against the previous results that assumed the data to be sampled from a sphere. 
- Thm 4.1: The theorem bounds the probability that gradient confusion holds for a given \eta. But the bounds of section 3 are vacuous even if the theorem holds with probability one, but for a large value of \eta. There needs to be an upper bound on \eta. Please clarify this. 
- Please compare against the results of this paper by Arora et al: "On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization"
- For the effect of layer width, the analysis is only for the initializated weights and does not consider the optimization, which is what the paper claimed in the introduction. Am I missing something? Please justify this. The gradient confusion can decrease as the optimization progresses? 
Section 5:
- "We reduce the learning rate by a factor of 10" But all the theory is for a constant step-size. Please explain this discrepancy. 
- in Figure 2, in the second figure, why is there a sharp full in the pairwise cosine similarities. 
- In all these experiments, explain why the batch-size and the step-size is not a confounding factor?