To analyze deep ReLU network, we adopt a student-teacher setting in which an over-parameterized student network learns from the output of a fixed teacher network of the same depth, with Stochastic Gradient Descent (SGD). Our contributions are two-fold. First, we prove that when the gradient is small at every training sample, student node specializes to teacher nodes in the lowest layer under mild conditions. Second, analysis of noisy recovery and training dynamics in 2-layer network shows that strong teacher nodes (with large fan-out weights) are learned first and subtle teacher nodes are left unlearned until late stage of training. As a result, it could take a long time to converge into these small-gradient critical points. Our analysis shows that over-parameterization is a necessary condition for specialization to happen at the critical points, and helps student nodes cover more teacher nodes with fewer iterations. Both improve generalization. Different from Neural Tangent Kernel (Jacot et al., 2018) and statistical mechanics approach (Goldt et al., 2019), our approach operates on finite width, mild overparameterization (as long as there are more student nodes than teacher) and finite input dimension. Experiments justify our finding.
The paper studies the role of over-parametrization in the student-teacher multilayer ReLU networks. It presents a theoretical part about properties of SGD critical points for the teacher-student setting. And a heuristic and empirical part on dynamics of the SDG algorithm as a function of properties of the teacher networks. Overall, given previous literature, I do not find the presented results novel nor fundamentally very interesting and some parts are hard to understand due to missing details. I tend to vote for rejection at this point. More detailed questions, comments follow.
In related works:
** Paragraph on "Teacher-student/realizable setting": The recent line of works is interesting, but the authors should be clearer about this being a very classical setting dating back several decades. The first paper I know where the teacher student setting appeared is by Garder, Derrida'83 (model B, https://iopscience.iop.org/article/10.1088/0305-4470/22/12/004/pdf). In the classical textbook on neural networks Engel, Andreas, and Christian Van den Broeck. Statistical mechanics of learning. Cambridge University Press, 2001, there is a very detailed account of many results on the setting from 80s and 90s.  
** "A line of works (Saad & Solla, 1996; 1995; Goldt et al., 2019; Freeman & Saad, 1997; Mace & Coolen, 1998) studied the dynamics from a statistical mechanics point of view, focusing on local analysis near to some critical points." and "(Goldt et al., 2019) assumes Gaussian input and symmetric parameterization to analyze local structure around critical points," The statements that these works focus on local analysis is not correct. While some formal analysis in these works required an infinitesimally informed start toward the teacher the experiments (in particular all those in Goldt et al., 2019) are run from random initialization and these works show empirically that randomly initialized training converges exactly to the fixed points described in the analysis.
** "Local minima is Global" paragraph: This paragraph seems to neglect the empirically observed fact (e.g. https://arxiv.org/pdf/1906.02613.pdf) that there can be global minima that generalize bad. Hence being global does not ensure good generalization. 
Body of the paper:
** The authors cite: "Previous works (Ge et al., 2017; Livni et al., 2014) show that empirically SGD does not recover the parameters of a teacher network up to permutation." but they fail to mention that separate line of work, e.g.  (Saad & Solla, 1996; 1995; Goldt et al., 2019) observed empirically the opposite.The different exiting works have to be reconciles and understood and that may be beyond the scope of the present work. But presenting only one side of the results is not helping.
** The part on the dynamics with strong and weak directions reminds me on the results on so called "INCREMENTAL LEARNING" e.g. in the work:
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013.
also later: https://arxiv.org/pdf/1809.10374.pdf and others. 
It would be useful to understand what is the relation in more detail and comment on it. 
** The experimental part of the paper has numerous flaws that make it hard to be understood. For instance the authors do not specify the distribution of the input data. Some experiments are run with CIFAR and others with "random" data, but random in which sense? While generalization is the main focus of the paper the experimental results focus on the alignments of the teacher and students without really being clear how specifically the speed or the generalization error improves when neural networks are overparametrised. I found this information only in Fig. 8 for the test error. In Fig. 11 I do not know what are the different panels. What is the parameter p? So I do not know what to conclude from this figure .... in the first pannel the non-overparametrized loss (blue) decreases fastest. In the last pannel all curves are comparable. But this would suggest that over-parametrizatoin is not really helping which seems to go agains the rest of the conclusion in the paper.
** A side remark: I note that the paper is on 10 pages and hence according to the paper call higher standards should be applied in the review process.