Bias Variance tradeoff in neural networks Large neural networks have low bias and high variance. Training on large datasets greatly reduces the variance allowing them to fit complicated functions. My question is why they seem to have much lower bias than other machine learning models, regardless of how flexible you make the other models. [EDIT thanks for comments - this framing is murky because it's not clear how much of the performance improvement could be attributed to lower variance due to the double descent phenomenon].
It's purported that if you plot amount of data vs performance, neural networks tend to not to plateau as quickly. One could argue that this has something to do with being a universal function approximator, but this apparently applies to other models like kernel methods as well.

(from https://www.researchgate.net/figure/Performance-Comparison-of-Deep-learning-based-algorithms-Vs-Traditional-Algorithms_fig1_338027948)
What about neural networks makes them especially good at working in regimes with large amounts of data compared to more traditional models which can also be made to be arbitrarily flexible?
I can only speculate it might have something to do with the following (but I can't find definitive sources):

*

*weaker inductive bias

*real-world data (distribution of images, language, etc.) being particularly amenable to NN operations

*having intermediate/hierarchical representations representations is somehow important (Yann LeCun talks about a Kuhnian [or Le-Kuhnian? ] paradigm shift on this point among researchers).


There are other theorems in mathematical analysis about converging to decent functions (the Stone–Weierstrass theorem about polynomials and Carleson's theorem about Fourier series come to mind). However, neural networks decrease the bias more than other regression models by taking to the extreme the idea of nonlinearity and interaction. A neural network with millions of parameters is routine. A generalized linear model with millions of parameters, due to nonlinear basis functions (e.g., polynomials or splines) and their interactions, is not as common. If you put in all of those nonlinear features and their interactions in a generalized linear model, taking it to a similar extreme as a neural network, I would expect similar issues of high variance and low bias.
In fact, there is a sense in which a neural network (at least some of them) involves a layer (or multiple layers) of feature extraction and then a generalized linear model on the extracted features. After all, if you draw out the usual "web" of a neural network and cover up everything before the final hidden layer, you wind up with something that looks like a generalized linear model.
