The dynamics of DNNs during gradient descent is described by the so-called Neural Tangent Kernel (NTK). In this article, we show that the NTK allows one to gain precise insight into the Hessian of the cost of DNNs. When the NTK is fixed during training, we obtain a full characterization of the asymptotics of the spectrum of the Hessian, at initialization and during training. In the so-called mean-field limit, where the NTK is not fixed during training, we describe the first two moments of the Hessian at initialization.
This paper discusses the behavior of the Hessian of the loss of NN during training. There is no question that the subject is relevant, and the paper gives an interesting contribution to the problem.  I do support publication.
My main criticism is in the presentation of the paper: The title and the first pages are, I believe, really misleading. Unless I am missing an important point, all derivation of the papers are being made in the so-called NTK regime. In fact, this is explicitly stated in page 3 "In the limit we study in this paper, the NTK is fixed". While this is an interesting assumption, this is by no mean a trivial one that should be stated casually in the middle of the paper. This is an extremely strong assumption (essentially, one is either in the lazy Regime of [Chizat&Bach] with infinity small learning rate, or in the kernel regime of [Jacot&al] with astronomically large layers. This is a nice setting for mathematics, but by no means a natural one and this limits drastically the conclusion of the paper: what is studied is the behavior of the DNN in the "Kernel" regime, or the "Lazy" regime. Such a limitation should be explicitly mentioned, at least in the abstract and the introduction, if not in the title itself! It should, also, be mentioned in the conclusion ("We have given an explicit formula for the limiting moments of the Hessian of DNNs throughout training"). 
In this respect, with the connection to the Kernel, it make sense that the Hessian is dominated by, essentially, the equivalent of Kernel PCA! This is to be expected, since one is essentially doing a Kernel Ridge Regression with random features.
Other source of confusion (for me):
* "This is contrast to the results obtained in [Pennington&Bahri] etc...." Is the only difference the fact that the network is more than two layers? 
* "This gives theoretical confirmation of a number of observations about the Hessian"... I do not believe all these references are in the NTK regimes. (e.g. Chaudhari et al., for instance). 
That being said, the paper is a nice and interesting contribution to the NTK regime, that is the subject of intense studies currently. I do support acceptance if the paper clarify the range of applicability of the its results.