Mean field models have provided a convenient framework for understanding the training dynamics for certain neural networks in the infinite width limit. The resulting mean field equation characterizes the evolution of the time-dependent empirical distribution of the network parameters. Following this line of work, this paper first focuses on the teacher-student setting. For the two-layer networks, we derive the necessary condition of the stationary distributions of the mean field equation and explain an empirical phenomenon concerning training speed differences using the Wasserstein flow description. Second, we apply this approach to two extended ResNet models and characterize the necessary condition of stationary distributions in the teacher-student setting.
This paper studies the mean fields limit of neural networks and extends the mean field limit treatment to resnets. The paper seems to be presenting non-trivial formal results but I could not really understand what are their specific consequences of interest. What are the concrete insights that the results of this paper bring? Even as a theoretician, I did not really understand what should I take from this paper. As such I cannot recommend its acceptance. But it is well plausible that with better explanations I will understand its value and change my mind. Some more concrete questions/issues follow: 
The introduction several times states: "First, for the two-layer networks in the teacher-student setting, we characterize for the first time the
necessary condition for a stationary distribution of the mean field PDE and show how it relates to the parameters of the teacher network. We also discuss how different activation functions influence the results." The outline states: "Section 2 concerns the mean field description of two-layer networks and provides necessary conditions for stationary distributions."
But where is this discussion of necessary condition? I cannot find a single mention of "necessary" in section 2 that only includes formal derivations and statements. Can the result of Section 2 be translated to somewhat more general audience? What is the take-home message there?  
Where is the discussion on the role of the activation function? I see the two examples stating the formal result for two different activations, but again I am unable to understand what should one take from this?
I could understand the result of section 2.2. about stronger weight-teachers being learned first. But this is completely intuitive in the same way as independent components corresponding to larger eigenvalues would get learned first in PCA. See also
similar conclusions in works on so-called "INCREMENTAL LEARNING" in Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013. And subsequent results. So this alone does not seem very novel result. 
As I had hard time understanding the section two on the simpler feedforward case, Section 3 was even less clear to me. How does what is done here compare to what is known on resnets and what should somebody interested in resnets (but not specifically in the mean field equations) take out from this. Some non-technical summary of the findings is seriously missing in this paper.
Comments that are less fundamental to the overall understanding:
** The mean field treatment was also extended to the multi-layer case in: https://arxiv.org/abs/1906.00193
** The paper present nice account on previous results involving the mean field limit. The manuscript should also discuss the long line of papers analyzing the teacher student setting on one-hidden-layer neural networks. After-all this seems to be the main object of study here so the results only make sence presented against what was previously known. Notably the case of eq. (1) where the 2nd layer weights are fixed to one is a model called the committee machine widely considered in previous literature (in the case of finitely wide one-hidden-layer network).
The (non-exhaustive) list of paper on the teacher-student setting is (more references are in those papers):
-- The teacher-student setting was introduced in Garder, Derrida'83 (model B, https://iopscience.iop.org/article/10.1088/0305-4470/22/12/004/pdf).
-- In the classical textbook on neural networks Engel, Andreas, and Christian Van den Broeck. Statistical mechanics of learning. Cambridge University Press, 2001, there is a rather detailed account of many results on the setting from 80s and 90s.
-- Notably the plain SGD was analyzed via ODEs for the teacher-student setting in classical papers: David Saad and Sara A Solla. On-line learning in soft committee machines. Physical Review E, 52 (4):4225, 1995.
David Saad and Sara A Solla. Dynamics of on-line gradient descent learning for multilayer neural networks. In Advances in neural information processing systems, pp. 302–308, 1996.
-- This line of work was recently extended with a focus on the overparametrized regime (but not infinitely wide) in: Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup, Sebastian Goldt, Madhu S. Advani, Andrew M. Saxe, Florent Krzakala, Lenka Zdeborová, NeurIPS'19.
-- There even in analysis with more than one hidden layer in Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized
neural networks, going beyond two layers. NeurIPS, 2019a.
It would be really interesting to see a discussion of what is similar and different between these works and results and the present work. And what is the added value of the present one about understanding the teacher-student setting.