Theoretical Speculations as to Why Neural Networks have Replaced Kernel-Based Methods I have been reading about the history of statistical and machine learning algorithms, and am particularly interested in the reasons as to why neural networks have "replaced" kernel-based methods in many applications.
As I understand, kernel-based methods (e.g., SVM, Gaussian Process) were the dominant choice of machine learning algorithms for many years – even when neural networks existed. I tried to do some readings and find out the reasons :

*

*Why were kernel-based methods originally developed?

*Why do kernel-based methods "work"?

*Why were kernel-based methods able to perform successfully in real world applications?

Based on these readings, I have identified the following theoretical properties of Kernels which might in part answer the above questions:

*

*Cover's Theorem: Cover's Theorem states that patterns that are difficult to separate in lower dimensions might be better separable when projected into higher dimensions. Thus, kernel-based methods try to learn these projection/mapping functions.


*Mercer's Theorem: Supposedly, Mercer's Theorem guarantees the existence of this mapping function.


*Kernel Trick and the Representer Theorem: I do not fully understand what these are, but it seems that both of these serve to facilitate the computations required in kernel-based methods.


*Kernel-Based Methods as Non-Parametric Models: Non-Parametric Models can be considered to potentially have an infinite number of model parameters. Thus, with regards to the bias-variance tradeoff, when compared to parametric regression models with many terms; kernel-based methods may have the ability to better capture higher levels of complexity within the data without suffering as much from the pitfalls of high variance. Kernel-based methods as such do not have model parameters that need to be explicitly estimated and the machine learning process is about deciding optimal hyperparameter values.

On the other hand, the main theoretical result pertaining to neural networks seems to be the Universal Approximation Theorem. Different versions of the Universal Approximation Theorem exist (e.g., more recent statements of this theorem make claims multilayered and finite depth neural networks) – the original statement of this theorem states that a 2-layered neural network with infinite depth can approximate any function to an arbitrary level of precision:

This is said to be closely related to the Kolmogorov-Arnold Representation Theorem, which states that "every multivariate continuous function can be represented as a superposition of the 2-argument addition and continuous functions of one variable":

But at the end of the day, what lead to neural networks becoming the dominant class of machine learning algorithms and outperforming kernel-based methods in many real world applications? One could argue that this was solely based on empirical reasons and circumstantial evidence, e.g., neural networks just tend to outperform kernel-based methods - but are there any mathematical theorems which can attempt to better explain the downfall of kernel-based methods and the fast-paced rise of neural networks?

*

*First of all, not only neural networks are universal approximators. There is nothing special about them, they just proved to work quite well for a class of problems.

*Kernel based methods generally don't scale well. Neural networks gained popularity in the time when we (a) improved our computers, so we were able to run bigger neural networks and do this in reasonable time, (b) with Internet, we gained access to large amounts of data (images, text) to train them. Kernel methods don't scale for large datasets, while in many cases it is the data not the algorithms that lead to improvements (Halevy, Norvig, and Pereira, 2009). Neural networks do scale.

*Neural networks gained popularity because they flourished for tasks related to image classification and natural language processing, traditional machine learning, including kernel based methods, don't generally do well for such tasks. Those are also the tasks where neural networks do well today, while for example for tabular data traditional machine learning seems to be commonly used.

