Why Are Neural Networks Considered "Expensive" to Train? Recently, I was looking at the optimization functions required in training Kernel Based Methods compared to Neural Networks.
1) Kernel Methods:
For instance, I was looking at the optimization in Support Vector Machines:

And Gaussian Process Regression:

2) Neural Networks:

My Question: We often hear the reason that Neural Networks were initially less popular than Kernel Based Methods is because (deep) Neural Networks typically require significantly more computational resources to train compared to Kernel Based Methods.
I have informally heard that Gaussian Process Regression scales better to larger data sets (based on a choice of kernel function, the data can be directly entered into the structural form of the Gaussian Process), and I have also informally heard that training Neural Networks are generally considered to be extremely computationally expensive - but just by looking at the functions associated with each model that require to be optimized, how can we understand the differences in computational costs between Kernel Based Methods and Neural Networks?
Thanks!
References:

*

*https://jeremykun.com/2017/06/05/formulating-the-support-vector-machine-optimization-problem

*https://arxiv.org/pdf/2009.10862.pdf

*What is a compact vector equation expression the back-propagation algorithm for convolution neural networks?

Neural networks are not per se expensive to train, but as you can arbitrarily / without end stack layers and layers, you can end up with quite a large network.
A famous NLP network, GPT-3, is quite large:

GPT-3 comes in eight sizes, ranging from 125M to 175B parameters

Further, for achieving good results with NN you usually use huge amounts of training data, such as in this case

Notice GPT-2 1.5B is trained with 40GB of Internet text, which is roughly 10 Billion tokens (conversely assuming the average token size is 4 characters)

That even is a problem with storage, because

The 175 Billion parameters needs
175
×
4
=
700
G
B
memory to store in FP32 (each parameter needs 4 Bytes).

The point I want to make is that successful NN are often deep and trained with lots of data, that does not mean that the class of algorithms per se is expensive to train.
All quotes are from https://lambdalabs.com/blog/demystifying-gpt-3/
