Bias Variance tradeoff in neural networks Large neural networks have low bias and high variance. Training on large datasets greatly reduces the variance allowing them to fit complicated functions. My question is why they seem to have much lower bias than other machine learning models, regardless of how flexible you make the other models. [EDIT thanks for comments - this framing is murky because it's not clear how much of the performance improvement could be attributed to lower variance due to the double descent phenomenon].
It's purported that if you plot amount of data vs performance, neural networks tend to not to plateau as quickly. One could argue that this has something to do with being a universal function approximator, but this apparently applies to other models like kernel methods as well.

(from https://www.researchgate.net/figure/Performance-Comparison-of-Deep-learning-based-algorithms-Vs-Traditional-Algorithms_fig1_338027948)
What about neural networks makes them especially good at working in regimes with large amounts of data compared to more traditional models which can also be made to be arbitrarily flexible?
I can only speculate it might have something to do with the following (but I can't find definitive sources):

*

*weaker inductive bias

*real-world data (distribution of images, language, etc.) being particularly amenable to NN operations

*having intermediate/hierarchical representations representations is somehow important (Yann LeCun talks about a Kuhnian [or Le-Kuhnian? ] paradigm shift on this point among researchers).


Sidenote: It depends on the situation
The question is a bit of a loaded question. It presupposes that neural networks are better. But, whether neural networks are better depends on the situation. If parametric models are applicable, then often these will work better. If the data generation doesn't follow a complex pattern (so nothing specific for a deep neural network to learn), then often the other shallow machine learning methods work better.
But still, the question is a fair question. The observation that over-parameterized models perform surprisingly good in particular settings, without overfitting (and whether this is only with neural networks or not is actually not relevant), that is not something unreal.
Flat vs Deep
The relationship between deep neural networks and other more shallow machine learning methods (shallow could be kernel methods or decision trees, but they can also be very complicated), is like the relationship between Copernicus model of the solar system and Ptolemy's model of the solar system.

*

*In some way, the kernel methods and decision trees are like glorified methods for smoothening or averaging of data. There is no strong connection with any simple underlying process or patterns that may create the observations. The methods are only superficially learning how to be able to describe the observations in a way that it can be interpolated or extrapolated with a sufficient accuracy. If you add more data the methods gain some precision for the area where the data has been added, but their learning capacity stagnates because the models never gain a 'higher level of understanding' of the patterns or some break down of the complexity of the observations into simple building blocks.


*On the other hand, the deep neural networks are sort of like applying Occam's razor and bring the complex gamut of observations down to a simplified underlying principle, which is captured by the organization of the network (and with every extra network layer the possibilities grow multiplicative increasing the potential complexity and rate of simplifying power). The neural network tries to learn the observation by learning a pattern behind it.
Double descent phenomenon
The over-parameterized networks are susceptible to fitting noise, but the simplest patterns are easier to learn and will become dominant. This is either due to some explicit regularization (obvious in ridge regression or Lasso) or due to some implicit regularization like limits on learning rates and stochastic decent. In this respect an example that more parameters work better is seen in this question: Is ridge regression useless in high dimensions ($n \ll p$)? How can OLS fail to overfit?.
Another effect could be that the multiple parameters work a bit like gradient boosting and are being blended together. When we spèeak about gradient boosting then there are also a lot of parameters fitted, more than the number of data points, but the method uses some average and we consider it not really as increasing the flexibility. This can happen in some similar way in deep neural networks and the first layers create several parallel pattern recognition models that are blended together in other layers.
