Interpolation with deep neural networks with non-polynomial activations: necessary and sufficient numbers of neurons
<|reference_start|>Interpolation with deep neural networks with non-polynomial activations: necessary and sufficient numbers of neurons: The minimal number of neurons required for a feedforward neural network to interpolate $n$ generic input-output pairs from $\mathbb{R}^d\times \mathbb{R}^{d'}$ is $\Theta(\sqrt{nd'})$. While previous results have shown that $\Theta(\sqrt{nd'})$ neurons are sufficient, they have been limited to sigmoid, Heaviside, and rectified linear unit (ReLU) as the activation function. Using a different approach, we prove that $\Theta(\sqrt{nd'})$ neurons are sufficient as long as the activation function is real analytic at a point and not a polynomial there. Thus, the only practical activation functions that our result does not apply to are piecewise polynomials. Importantly, this means that activation functions can be freely chosen in a problem-dependent manner without loss of interpolation power.<|reference_end|>
@article{madden2024interpolation,
    title={Interpolation with deep neural networks with non-polynomial activations:
  necessary and sufficient numbers of neurons},
    author={Liam Madden},
    journal={arXiv preprint arXiv:2405.13738},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.13738},
    primaryClass={cs.LG math.OC}
}