In this paper, we present some theoretical work to explain why simple gradient descent methods are so successful in solving non-convex optimization problems in learning large-scale neural networks (NN). After introducing a mathematical tool called canonical space, we have proved that the objective functions in learning NNs are convex in the canonical model space. We further elucidate that the gradients between the original NN model space and the canonical space are related by a pointwise linear transformation, which is represented by the so-called disparity matrix. Furthermore, we have proved that gradient descent methods surely converge to a global minimum of zero loss provided that the disparity matrices maintain full rank. If this full-rank condition holds, the learning of NNs behaves in the same way as normal convex optimization. At last, we have shown that the chance to have singular disparity matrices is extremely slim in large NNs. In particular, when over-parameterized NNs are randomly initialized, the gradient decent algorithms converge to a global minimum of zero loss in probability.
Summary:
This paper presents an argument that massively overparameterized neural networks trained with gradient descent on a supervised learning problem with convex loss function will converge. It argues that by mapping each model to a truncated Fourier series we can recover a canonical representation for the functions in the model space. Viewing this as a linear map at each point in parameter space and assuming that this matrix is always full rank, they claim convergence of gradient descent.
Decision:
This paper should be rejected because it lacks originality, the assumptions are often too strong, and the rigor of some claims is questionable.
Main argument:
Originality:
This is the main issue with the paper. Several papers, for example the ones from Allen-Zhu et al. and Du et al. cited in the paper, have shown that gradient descent on supervised learning provably converges for massively overparameterized neural networks. Not only that, but those works give rigorous proofs of their claims and even handle issues like step size, convergence rates, and precise conditions on the size and architecture of the neural network. This paper does not attempt to handle any of these details, claiming to simplify the proof. To me the proof is only simplified since the details are all omitted.
Assumptions:
The main way that details are avoided is by adding a strong assumption that the “disparity matrix” remains full rank over the course of training. This assumption is not supported in an empirical or formal way, but rather by appeal to some vague argument in section 3 that since the matrix is likely to be full rank at initialization, it is likely to remain so for large enough networks. 
Rigor:
As explained above, many details are omitted in favor of strong assumptions, but there are also some technical details that I think may be wrong. On page 2, it is claimed that the mapping from Lambda_M to L^1 is surjective, this is not true. Every function in L^1 can be approximated by some function in Lambda^M, but for finite M this map cannot be surjective. Another similar issue arises in the proof of Theorem 1 when it is claimed that there is a unique theta_epsilon corresponding to each f. This is again false since by truncating the Fourier series, infinitely many functions will get mapped to the same truncation (when they only differ in the higher coefficients). These issues make me question the validity as well as the originality of the arguments presented.
Additional feedback:
- I saw a few spelling and grammatical errors. For example: in the abstract the tenses alternate between present and past almost every sentence (“we have proved” should be “we prove”),  page 2 “parametarize”, page 3 “serie”, page 5 “distint”, page 7 “Liptschitz”, section 3.3 title should be “when does” not “when do”.