Bias Variance tradeoff in neural networks Large neural networks have low bias and high variance. Training on large datasets greatly reduces the variance allowing them to fit complicated functions. My question is why they seem to have much lower bias than other machine learning models, regardless of how flexible you make the other models. [EDIT thanks for comments - this framing is murky because it's not clear how much of the performance improvement could be attributed to lower variance due to the double descent phenomenon].
It's purported that if you plot amount of data vs performance, neural networks tend to not to plateau as quickly. One could argue that this has something to do with being a universal function approximator, but this apparently applies to other models like kernel methods as well.

(from https://www.researchgate.net/figure/Performance-Comparison-of-Deep-learning-based-algorithms-Vs-Traditional-Algorithms_fig1_338027948)
What about neural networks makes them especially good at working in regimes with large amounts of data compared to more traditional models which can also be made to be arbitrarily flexible?
I can only speculate it might have something to do with the following (but I can't find definitive sources):

*

*weaker inductive bias

*real-world data (distribution of images, language, etc.) being particularly amenable to NN operations

*having intermediate/hierarchical representations representations is somehow important (Yann LeCun talks about a Kuhnian [or Le-Kuhnian? ] paradigm shift on this point among researchers).


Bias-variance trade-off is an old fashioned concept from classical statistics which fails to be useful in high-dimensional setting.
Here's an example of famous statistician being surprised that overfitting is reduced by increasing the number of parameters in linear regression.
A better way to explain good performance of neural networks is through the lens of statistical learning theory. One direction of work shows that if A) your learner is not very sensitive to small changes in your training set, and B) fits training data data it will also fit test data. See for instance, this paper by Bousquet.
Hastie's paper shows that adding parameters restricts final solution to a smaller L2-norm ball, hence improving stability A). At the same time, adding parameters can improve training fit, hence improving B).
B) is actually the harder part, much of modern progress in NN's has been achieved by coming up with clever ways of fitting the training data and ignoring generalization aspect.
