We show that in a variety of large-scale deep learning scenarios the gradient dynamically converges to a very small subspace after a short period of training. The subspace is spanned by a few top eigenvectors of the Hessian (equal to the number of classes in the dataset), and is mostly preserved over long periods of training. A simple argument then suggests that gradient descent may happen mostly in this subspace. We give an example of this effect in a solvable model of classification, and we comment on possible implications for optimization and learning.
This paper shows that gradient descent mostly happens in a tiny subspace which is spanned by the top eigenvectors of the Hessian. Empirical results are shown to support the claim. This finding is interesting and provides us some insights to design more efficient optimization algorithms. Overall, this paper is interesting and easy to follow. 
The experiments in section 2 do a decent job supporting the claim that gradient descent happens in a tiny subspace and the subspace is mostly preserved over long periods of training. However, I would like to add a couple more points to the discussion: 
- It's not surprising that the magnitude of gradient is larger in the high curvature directions, which means that the learning would always first happen in top subspace if the learning rate is small enough. It would be interesting to tune the parameter of learning rate to see if the phenomena would occur across different learning rate (especially large learning rate).
- The argument of gradient descent happening in a tiny space is quite obvious if the Hessian has only a few large eigenvalues. Therefore, it would be interesting to discuss the spectrum of the Hessian a little bit.
- Contrary to plain gradient descent, natural gradient is able to learn low curvature directions (small eigenvalues). It would be interesting to show some experiments with natural gradient methods.
Following section 2, the authors give a toy model to further backup their claims. However, I find the example is too restrictive and may not explain why the subspace would be preserved over the training. If I understand right, the loss function of the toy model is convex and Hessian is a constant over time. For this kind of toy model, the Hessian (or equivalently the Fisher matrix) only depends on the input distribution, so it's easy to see that the Hessian would be low-rank and preserved throughout the training. However, neural networks are highly non-convex, so it's unclear to me whether the implications of the toy model would generalize. I encourage the authors to analyze more complicated models. 
To summarize, I think this paper is interesting and well-written. However, it lacks convincing explanation why the subspace would preserve over the training (to me, it's more interesting than the point that gradient descent happens in tiny subspace). Anyway, it is not completely reasonable to expect all such possible discussions to take place at once.