We show that finite-width deep ReLU neural networks yield rate-distortion optimal approximation (Bölcskei et al., 2018) of a wide class of functions, including polynomials, windowed sinusoidal functions, one-dimensional oscillatory textures, and the Weierstrass function, a fractal function which is continuous but nowhere differentiable. Together with the recently established universal approximation result for affine function systems (Bölcskei et al., 2018), this demonstrates that deep neural networks approximate vastly different signal structures generated by the affine group, the Weyl-Heisenberg group, or through warping, and even certain fractals, all with approximation error decaying exponentially in the number of neurons. We also prove that in the approximation of sufficiently smooth functions finite-width deep networks require strictly fewer neurons than finite-depth wide networks.
The focus of this paper is to show that finite-width deep neural networks with fully connected layers and ReLU activations are rate-distortion optimal approximators of certain classes of functions, meaning the approximation error decays exponentially in the number of neurons in the network. The function classes explored in this paper are: 1-d polynomials (on bounded intervals), 1-d sinusoidal functions (on bounded intervals), and other 1-d functions built from compositions or linear combinations of these, such as the so-called class of “oscillatory textures” and a class of continuous but nowhere differentiable functions known as Weierstrass functions. Finally, the paper also shows that as the desired approximation accuracy goes to zero finite-width deep ReLU networks require asymptotically fewer neurons than finite-depth wide ReLU networks in approximating a broad class of smooth functions.
The paper is well-written and the technical results are presented in a way that is easy to understand. The results are somewhat novel, although they do build off other recent works, namely Yarotsky (2016) and Telgarsky (2015). However, the authors were careful to cite when they reuse proof techniques from these and other works. The results in the main text appear to be technically sound. I did not check carefully all the proofs in the supplemental materials.
My major criticism is that the focus on certain specific function classes (oscillatory textures, Weierstrauss functions) seems arbitrary, and leaves open many questions. For example, there is existing work on the approximation ability of deep ReLU networks for functions in more general Holder and Sobolev spaces:
Hadrien Montanelli and Qiang Du. Deep ReLU networks lessen the curse of dimensionality. arXiv preprint arXiv:1712.08688, 2017.
J. Schmidt-Hieber. Nonparametric regression using deep neural networks with ReLU activation function. ArXiv e-prints, August 2017.
I was left wondering how the present results relate to these works, and what insight we get from understanding these particular function classes that we don't get from understanding Holder or Sobolev spaces.
Major comments
In Section 3, I found the progression of the results from approximation of x^2, to multiplication xy, and to general smooth functions to be very natural and well-motivated. However, sections 4 and 5 seem lack somewhat in motivation, since here the authors focus on very specific function classes (sinusoidal functions, oscillatory textures, and Weierstrass functions). While these results are still interesting, focusing on such specific functions is less satisfactory, since it raises questions about the true scope of the results (e.g., will similar approximation rates extend to other fractal functions, or just Weierstrauss functions?). Could the authors give further justification for why these function classes are interesting to focus on, or why they limit themselves in this way? Can the authors also put these results more into context with existing results on the approximation with ReLU networks?
The authors state multiple times that “all our results apply to the multivariate case” but that they restrict themselves to the univariate case for simplicity of presentation. While this is fine, some indication of how the results are altered in the multivariate case would be useful. For example, does the fixed-width M in multivariate generalizations of Prop 3.1--3.3 need to be bigger, smaller, or the same? What other constants are dimensionally dependent? Do the multivariate generalization of their results bear the "curse of dimensionality", i.e., does the number of neurons needed to reach epsilon accuracy depend geometrically on the dimension?
Minor comments
A conclusion or discussion section summarizing the overall technical contribution would be useful for the reader. Also, it would be useful to include some discussion on remaining open problems or future work.
On pg. 2, the authors state “the approximation results throughout the paper guarantee that the magnitude of the weights in the network does not grow faster than polynomially in the cardinality of of the domain over which the approximation takes place”. What does “cardinality of the domain” here mean? I think the authors mean the size D of the interval [-D,D] over which the approximation is valid.
On pg. 7, the authors say “We note that this result allows to show that local cosine bases (cite) can be approximated by deep ReLU networks with exponential error decay…”. I think the authors mean to say “...this result allows us to show…” or “this result allows one to show…”. Although it’s not clear to me whether this means it has been shown (it’s a direct corollary), or could possible be shown (it’s a corollary, but needs some non-trivial work). Also, one line to specify what a “local cosine basis” is would be helpful.