ADAPTIVITY OF DEEP RELU NETWORK FOR LEARN- ING IN BESOV AND MIXED SMOOTH BESOV SPACES: OPTIMAL RATE AND CURSE OF DIMENSIONALITY
Deep learning has shown high performances in various types of tasks from visual recognition to natural language processing, which indicates superior flexibility and adaptivity of deep learning. To understand this phenomenon theoretically, we develop a new approximation and estimation error analysis of deep learning with the ReLU activation for functions in a Besov space and its variant with mixed smoothness. The Besov space is a considerably general function space including the Hölder space and Sobolev space, and especially can capture spatial inhomogeneity of smoothness. Throughout the analysis in the Besov space, it is shown that deep learning can achieve the minimax optimal rate and outperform any nonadaptive (linear) estimator such as kernel ridge regression, which shows that deep learning has higher adaptivity to the spatial inhomogeneity of the target function than other estimators such as linear ones. In addition to this, it is shown that deep learning can avoid the curse of dimensionality if the target function is in a mixed smooth Besov space. We also show that the dependency of the convergence rate on the dimensionality is tight due to its minimax optimality. These results support high adaptivity of deep learning and its superior ability as a feature extractor.
This paper describes approximation and estimation error bounds for functions in Besov spaces using estimators corresponding to deep ReLU networks. The general idea of connecting network parameters such as depth, width, and sparsity to classical function spaces is interesting and could lead to novel insights into how and why these networks work and under what settings. The authors carefully define Besov spaces and related literature, and overall the paper is clearly written. 
Despite these strengths, I'm left with several questions about the results. The most critical is this: piecewise polynomials are members of the Besov spaces of interest, and ReLU networks produce piecewise linear functions. How can piecewise linear approximations of piecewise polynomial functions lead to minimax optimal rates? The authors' analysis is based on cardinal B-spline approximations, which generally makes sense, but it seems like you would need more terms in a superposition of B-splines of order 2 (piecewise linear) than higher orders to approximate a piecewise polynomial to within a given accuracy. The larger number of terms should lead to worse estimation errors, which is contrary to the main result of the paper. I don't see how to reconcile these ideas. 
A second question is about the context of some broad claims, such as that the rates achieved by neural networks cannot be attained by any linear or nonadaptive method. Regarding linear methods, I agree with the author, but I feel like this aspect is given undue emphasis. The key paper cited for rates for linear methods is the Donoho and Johnstone Wavelet Shrinkage paper, in which they clearly show that nonlinear, nonadaptive wavelet shrinkage estimators do indeed achieve minimax rates (within a log factor) for Besov spaces. Given this, how should I interpret claims like "any linear/non-linear approximator
with fixed N -bases does not achieve the approximation error ... in some parameter settings such as 0 < p < 2 < r "?
Wavelets provide a fixed N-basis and achieve optimal rates for Besov spaces. Is the constraint on p and r a setting in which wavelet optimality breaks down? If not, then I don't think the claim is correct. If so, then it would be helpful to understand how relevant this regime for p and r is to practical settings (as opposed to being an edge case). 
The work on mixed Besov spaces (e.g. tensor product space of 1-d Besov spaces) is a fine result but not surprising.
A minor note: some of the references are strange, like citing a 2015 paper for minimax rates for Besov spaces that have been known for far longer or a 2003 paper that describes interpolation spaces that were beautifully described in DeVore '98. It would be appropriate to cite these earlier sources.