Exponential Expressivity of ReLU$^k$ Neural Networks on Gevrey Classes with Point Singularities
<|reference_start|>Exponential Expressivity of ReLU$^k$ Neural Networks on Gevrey Classes with Point Singularities: We analyze deep Neural Network emulation rates of smooth functions with point singularities in bounded, polytopal domains $\mathrm{D} \subset \mathbb{R}^d$, $d=2,3$. We prove exponential emulation rates in Sobolev spaces in terms of the number of neurons and in terms of the number of nonzero coefficients for Gevrey-regular solution classes defined in terms of weighted Sobolev scales in $\mathrm{D}$, comprising the countably-normed spaces of I.M. Babu\v{s}ka and B.Q. Guo.   As intermediate result, we prove that continuous, piecewise polynomial high order (``$p$-version'') finite elements with elementwise polynomial degree $p\in\mathbb{N}$ on arbitrary, regular, simplicial partitions of polyhedral domains $\mathrm{D} \subset \mathbb{R}^d$, $d\geq 2$ can be exactly emulated by neural networks combining ReLU and ReLU$^2$ activations. On shape-regular, simplicial partitions of polytopal domains $\mathrm{D}$, both the number of neurons and the number of nonzero parameters are proportional to the number of degrees of freedom of the finite element space, in particular for the $hp$-Finite Element Method of I.M. Babu\v{s}ka and B.Q. Guo.<|reference_end|>
@article{opschoor2024exponential,
    title={Exponential Expressivity of ReLU$^k$ Neural Networks on Gevrey Classes
  with Point Singularities},
    author={Joost A. A. Opschoor, Christoph Schwab},
    journal={arXiv preprint arXiv:2403.02035},
    year={2024},
    doi={10.21136/AM.2024.0052-24},
    archivePrefix={arXiv},
    eprint={2403.02035},
    primaryClass={math.NA cs.LG cs.NA}
}