Despite their popularity and successes, deep neural networks are poorly understood theoretically and treated as 'black box' systems. Using a functional view of these networks gives us a useful new lens with which to understand them. Theoretical (in shallow) and experimentally probing properties of these networks reveals insights into the effect of standard initializations, the value of depth, the underlying loss surface, and the origins of generalization. One key result is that generalization results from smoothness of the functional approximation, combined with a flat initial approximation. This smoothness increases with number of units, explaining why massively overparamaterized networks continue to generalize well.
This paper proposes a functional characterization to understand the empirical success of deep neural networks. In particular, this paper focuses on the case of deep fully connected univariate ReLU networks, and show that the parameters will result in a Continuous Piecewise Linear (CPWL) approximation to the target function. Moreover, the authors derive the induced distributions of the function space parameters and show that increasing width can reduce the roughness of the initial function.
Besides, this paper analyzes the loss surface in the function space and reveals some relationship between the critical points in the function space and original NN parameter space. Furthermore, a type of gradient descent dynamic in the function space has also been derived.
Many experiments have been conducted to reveal how the expressiveness and optimization performance varies with the neural network width and depth.
Overall, the functional characterization is interesting can potentially help explain the generalization/expressiveness of deep neural networks. However, this paper is not well written and organized, and there are some “??'s” appearing on page 4. The authors should pay more attention to improving the writing and organization of this paper. 
Here are the detailed comments:
The statements of theorems are not clear. For example, Theorems 3 and 4 convey too much information, I believe the authors should simplify the statements to make them more concise.  Moreover, the purpose of these two lemmas is not clear. Do they imply something related to expressiveness?
There should be some discussion in the surrounding text of Theorem 5. Some notations are also missing. For example, what’s $\hat \epsilon(t)$?, what’s $a_i(t)$? Besides, what’s the purpose of Theorem 5. It seems that this Theorem is used in the main part of this paper.
One major drawback of this paper is that it only provides the theoretical analysis for two-layer networks, but the title and claimed contributions are related to deep ReLU networks. I wonder whether the theory can be extended to deep cases?
In Theorem 2, I think He initialization will give you $\sigma_w = \sqrt{2/(H+1)}$ and $\sigma_v = \sqrt{2/k}$, if the output dimension is k. In this way, I am curious whether Theorem 2 can still hold. Besides, what’s the meaning of the so-called “roughness“?
I don’t get the point in Implications of Corollary 1. For example, what do you mean “$f$ has significant curvature in the boundaries?”, why an initialization that allocates more breakpoints to the area where the curvature of $f$ lies can be faster to train? The authors should elaborate more on this.
After reading the authors' response.
Thanks for your response.  I still think the contribution of this paper is not enough as the theoretical analysis may not be able to be generalized to deep networks. Thus I would like to keep my score.