Approximation Rates for Shallow ReLU$^k$ Neural Networks on Sobolev Spaces via the Radon Transform
<|reference_start|>Approximation Rates for Shallow ReLU$^k$ Neural Networks on Sobolev Spaces via the Radon Transform: Let $\Omega\subset \mathbb{R}^d$ be a bounded domain. We consider the problem of how efficiently shallow neural networks with the ReLU$^k$ activation function can approximate functions from Sobolev spaces $W^s(L_p(\Omega))$ with error measured in the $L_q(\Omega)$-norm. Utilizing the Radon transform and recent results from discrepancy theory, we provide a simple proof of nearly optimal approximation rates in a variety of cases, including when $q\leq p$, $p\geq 2$, and $s \leq k + (d+1)/2$. The rates we derive are optimal up to logarithmic factors, and significantly generalize existing results. An interesting consequence is that the adaptivity of shallow ReLU$^k$ neural networks enables them to obtain optimal approximation rates for smoothness up to order $s = k + (d+1)/2$, even though they represent piecewise polynomials of fixed degree $k$.<|reference_end|>
@article{mao2024approximation,
    title={Approximation Rates for Shallow ReLU$^k$ Neural Networks on Sobolev
  Spaces via the Radon Transform},
    author={Tong Mao, Jonathan W. Siegel, Jinchao Xu},
    journal={arXiv preprint arXiv:2408.10996},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.10996},
    primaryClass={stat.ML cs.LG cs.NA math.NA}
}