Simulated annealing for deep learning: Why is gradient free statistical learning not in the main stream? In order to define what deep learning is, the learning portion is often listed with backpropagation as a requirement without alternatives in the main stream software libraries and in the literature. There are not many gradient free optimisations are mentioned in deep learning or in general statistical learning. Similarly, in "classical algorithms" (Nonlinear least squares) involves derivatives [1]. In general, gradient free learning in deep learning or classical algorithms are not in the main stream. One promising alternative is simulated annealing [2, 3], so-called 'nature-inspired optimization'.
Is there any inherent theoretical reason that why gradient free deep learning (statistical learning) is not in the main stream? (Or not preferred?)
Notes
[1] Such as Levenberg–Marquardt
[2] Simulated Annealing Algorithm for Deep Learning (2015)
[3] CoolMomentum: a method for stochastic optimization by Langevin dynamics with simulated annealing (2021) Though this is still not fully gradient-free, but does not require auto-differentiation.
Edit 1
Additional references using Ensemble Kalman Filter, showing a derivative free approach:

*

*Ensemble Kalman Inversion: A Derivative-Free Technique For Machine Learning Tasks arXiv:1808.03620.

*Ensemble Kalman Filter optimizing Deep Neural Networks: An alternative approach to non-performing Gradient Descent springer (manuscript-pdf)

Edit 2
As far as I gather, Yann LeCun does not consider gradient-free learning as part of deep learning ecosystem. "DL is constructing networks of parameterized functional modules & training them from examples using gradient-based optimization." tweet
Edit 3
Ben Bolker's comment on local geometry definitely deserves to be one of the answers.

In my understanding it's a consequence of the high number of variables that neural networks tends to require when tackling interesting problems. For simple tasks gradient-free methods work very well and are quite capable of beating gradient-based methods, as many of them deal with non-convex functions/local optima better than the grad-based methods and that tends to be the biggest issue for low dimensional problems.
However, as the number of dimensions/model variables increases, two things happen:

*

*Local optima cease to be optima and become saddles instead. To be, say, a local minimum a zero-gradient point must be a minimum with respect to every dimension. If you have a million of these, it is practically guaranteed that it won't be a minimum in at least one. Modern gradient-based methods deal with saddles reasonably well, so as models scale up the functions become effectively convex for them.

*A random perturbation of a solution candidate becomes increasingly unlikely to happen to have a direction similar to that of the gradient. That means that in grad-free methods that rely on such perturbations a lot of them will have to be made before the solution candidates move in the direction of the gradient, as opposed to just performing a random walk. Most grad-free methods fall into this category, and accordingly take a performance hit as models scale up.

The exception to that rule are the methods of the evolutionary strategies family. The main idea of these is to accumulate the information about the gradient from multiple perturbations and skew the distribution of subsequent perturbations in a way that makes them more likely to be aligned with the gradient. Those perform reasonably well on deep learning tasks [1]. They require roughly a few times more resources than the amount required by the gradient-based family to do the same job, but offer a superior performance on deceptive problems and improved horizontal scalability. I think the main reason why this approach never attracted mainstream attention is because the amount of parallel hardware required to get to the point where they compare favorably to grad-based is available to very few people in the world. It's been a while, but from what I recall the breakeven point is somewhere in the hundreds of GPUs region.
[1] https://eng.uber.com/deep-neuroevolution/
