A long-held conventional wisdom states that larger models train more slowly when using gradient descent. This work challenges this widely-held belief, showing that larger models can potentially train faster despite the increasing computational requirements of each training step. In particular, we study the effect of network structure (depth and width) on halting time and show that larger models-wider models in particular-take fewer training steps to converge. We design simple experiments to quantitatively characterize the effect of overparametrization on weight space traversal. Results show that halting time improves when growing model's width for three different applications, and the improvement comes from each factor: The distance from initialized weights to converged weights shrinks with a power-law-like relationship, the average step size grows with a power-law-like relationship, and gradient vectors become more aligned with each other during traversal.
Understanding the effects of over-parametrization in neural network training has been a major challenge, albeit a lot of progress has been made in the past few years. The present paper is another attempt in this direction, with a slightly different point of view: the work characterizes the impact of over-parametrization in the number of iterations it takes an algorithm to converge. Along the way, it also presents further empirical observations such as the distance between the initial point and the final point and the angle between the gradients and the line that connects the initial and final points. Even though the observations presented are very interesting, unfortunately, the paper doesn't have the level of rigor required that would make it a solid reference. 
The work presents its results somewhat clearly in the sense that one can simply reconstruct to probe in order to replicate the observations. This clarity is mainly due to the simplicity of the questions posed. There is nothing inherently wrong with simple questions, in fact, the kind of questions posed in the present paper are quite valuable, however, it lacks detailed study and rigor of a strong empirical work. Furthermore, the style of the exposition (anecdotal) and several obvious typos make the work look quite unfinished. 
Here are some flaws and suggestions that would improve the work substantially:
- A deeper literature review would help guide the reader put the paper in a better context. Especially, the related work section is quite poor, how exactly do those papers appear related to the present work? Do they support similar ideas or do they propose different perspectives?  
- The exposition should be made more to the point and concise (for instance 3rd paragraph of section 4.3 where it starts with Figure 5(a) What's meant by over-fitting regime, is it worse gen error, is it merely fitting tr data?.. How do we "know" from Figure 2, what's a strong evidence? Some concepts such as the capacity do not have precise and commonly agreed upon definitions, the paper uses those quite a bit and sometimes only later on the reader understands what it actually refers to... The misalignment section is also quite unclear.)
- The observations can be formalized and the curve fitting should be explained in further detail, the appendix touches upon simple cases but there is a strong literature behind those simple cases that could be quite useful for the purposes of the paper. 
- The authors have a lot of data available at no point the power law decay and exponent fitting are discussed. For a paper whose main point is this precise scaling, this looks like a major omission unless there is a specific reason for it (other than the hardness of fitting exponents to power laws). Merely showing the observables in a log-log plot weakens the support of the main claims.
- The theoretical argument provided is just an elementary observation whose assumptions and conditions are not discussed. It is not a straightforward task, for instance, a suggestion for a theoretical result on the distance between the initial and final weights is presented here: Lemma 1 A.3 https://arxiv.org/abs/1806.07572 (distance shrink as the number of parameters increase consistent with the observations of the present paper) (note that this is in addition to the several early-2018 mean field approximations to NNs whose solutions are found in the limit where the number of  parameters tend to infinity)
- All the figures from 5 to 8 are presented very quantitatively such as looking at different layers and observing the percentage reductions. The message one can gain from such presentations are extremely limited and not systematic. I encourage the authors to formulate solid observables that can and should be tested in further detail. 
Even though the paper is touching upon very interesting questions, at its current stage, it is not a good fit to be presented in a conference as it only presents anecdotal evidence. There is a lot of room to improve, but the good news is that most of the improvement should be straightforward.