Bias Variance tradeoff in neural networks Large neural networks have low bias and high variance. Training on large datasets greatly reduces the variance allowing them to fit complicated functions. My question is why they seem to have much lower bias than other machine learning models, regardless of how flexible you make the other models. [EDIT thanks for comments - this framing is murky because it's not clear how much of the performance improvement could be attributed to lower variance due to the double descent phenomenon].
It's purported that if you plot amount of data vs performance, neural networks tend to not to plateau as quickly. One could argue that this has something to do with being a universal function approximator, but this apparently applies to other models like kernel methods as well.

(from https://www.researchgate.net/figure/Performance-Comparison-of-Deep-learning-based-algorithms-Vs-Traditional-Algorithms_fig1_338027948)
What about neural networks makes them especially good at working in regimes with large amounts of data compared to more traditional models which can also be made to be arbitrarily flexible?
I can only speculate it might have something to do with the following (but I can't find definitive sources):

*

*weaker inductive bias

*real-world data (distribution of images, language, etc.) being particularly amenable to NN operations

*having intermediate/hierarchical representations representations is somehow important (Yann LeCun talks about a Kuhnian [or Le-Kuhnian? ] paradigm shift on this point among researchers).


The existence of a bias-variance tradeoff has been assumed as inevitable (i.e., an axiom) in any model using data, including neural networks.
However, it has been observed since about 2018 that surprisingly, some cases of very large deep neural networks, trained with a correspondingly sufficiently large dataset, do not exhibit the classical bias-variance tradeoff. This means that these networks also generalize better. This phenomenon, termed "double descent", has been duplicated by other researchers. See for example:
"Reconciling modern machine-learning practice and the classical bias–variance trade-off", 2019, by Belkin, Hsu, Ma, Mandal, https://www.pnas.org/doi/10.1073/pnas.1903070116.
As of 2022, conclusively explaining this phenomenon is still an open research question, but there have recently been interesting inroads to answering it. For example, the following paper is a theoretical explanation to justify this mysterious phenomenon:
"A Universal Law of Robustness via Isoperimetry", 2021, by Bubeck and Sellke, https://arxiv.org/abs/2105.12806 (Outstanding Paper award at NeurIPS 2021)

*

*The analysis/explanation is based on a network having a small
Lipschitz constant (maximum value of the gradients), meaning the
function represented by the network is smooth.

*The paper also claims that in addition to good generalization, such a
phenomenon also implies better robustness to adversarial attacks.

*The analysis is not limited to neural networks, but is
general enough for many other function approximations (including Reproducing Kernel Hilbert Space).

*The paper gives specific guidance on the number of parameters vs. the
amount of data for this phenomenon to occur.

Emphasis: This "double descent" phenomenon does not occur in all deep neural networks trained with a correspondingly sufficiently large dataset. Rather, according to Bubeck and Sellke, it depends on the number of input data points, the effective dimension of the classification, the depth of the network (number of layers), and the overall number of parameters in the network.
Therefore, in other cases, neural networks, even deep ones, will still exhibit the bias-variance tradeoff.
In a sense, this guidance on parameter values in the Bubeck and Sellke paper can be regarded as a falsifiable prediction as to whether their analysis/explanation is (in)correct.
