We study the training process of Deep Neural Networks (DNNs) from the Fourier analysis perspective. We demonstrate a very universal Frequency Principle (FPrinciple) -DNNs often fit target functions from low to high frequencies -on high-dimensional benchmark datasets such as MNIST/CIFAR10 and deep neural networks such as VGG16. This F-Principle of DNNs is opposite to the behavior of most conventional iterative numerical schemes (e.g., Jacobi method), which exhibit faster convergence for higher frequencies for various scientific computing problems. With theories under an idealized setting, we illustrate that this F-Principle results from the smoothness/regularity of the commonly used activation functions. The F-Principle implies an implicit bias that DNNs tend to fit training data by a low-frequency function. This understanding provides an explanation of good generalization of DNNs on most real datasets and bad generalization of DNNs on parity function or a randomized dataset.
The paper studies the training process of NNs through the lens of Fourier analysis. The authors argue that during the training process, NNs will first learn low frequencies part of the function first and then the high frequency part. To verify this claim empirically, the author propose two methods: 1. examine the convergence of different frequencies in a pre-selected direction in the frequency space during training; 2. examine the convergence rate of the 2-norm of low v.s. high frequencies during training.  Through the experimental results of these two methods, the authors conclude that NNs learn the low  frequency components before the high frequency components. The authors also discuss a potential application of this observation to solving high dimensional PDEs: coupling DNNs training (good at learning low frequency components) with the Jacobi method (good at learning high frequency components). Finally, the authors also provide some theoretical intuition (Thm 1., 2.) why low frequency components are learned faster and an explanation why NNs could generalize well on images but perform poorly on tasks like learning parity functions. 
Other comments: 
1. It seems the filtering method is a better (might be a sufficient) way to justify the F-Principle than the projection method, given the projection method examines only one direction (also appointed out in the paper).  
2. When talking about Fourier transform, would you specify what is the domain of the functions and how the functions are defined (section 3.1) The notation there is somewhat confusing (which makes the rest of the paper difficult to follow) since you are mentioning the Fourier transform of the set {(x_i, y_i)}. It will be helpful to define the function before defining its Fourier transform.  Please also mention what is the domain of the function, {x_i}_i or R^d? 
3. According to equation (4), it seems the domain of the functions is {x_i}_i, otherwise equation (4) should be a function of x\in R^d, not x_i.  
4. Could you elaborate why (4) is a good approximation of the low frequency energy rather than the L2 norm (over x\in R^d) of (4) with x_i replaced by x\in R^d. 
5. It might be useful to refine the related work section. It is not clear what are the previous contributions prior to this paper, and it seems [1] shares some similar results/observation with the this paper. 
Overall, I lean to a weak rejection. The key findings (and similar results, e.g. NNs learn simple functions first), i.e. F-Principle seems to have already appeared in previous works, e.g. [1] and the theoretical results of this paper are limited to an idealized setting (results of more general setting appear in another work, mentioned in the paper.)
[1]Nasim Rahaman, Devansh Arpit, Aristide Baratin, Felix Draxler, Min Lin, Fred A Hamprecht,
Yoshua Bengio, and Aaron Courville. On the spectral bias of deep neural networks. arXiv preprint
arXiv:1806.08734, 2018. 1, 8, A