Simulated annealing for deep learning: Why is gradient free statistical learning not in the main stream? In order to define what deep learning is, the learning portion is often listed with backpropagation as a requirement without alternatives in the main stream software libraries and in the literature. There are not many gradient free optimisations are mentioned in deep learning or in general statistical learning. Similarly, in "classical algorithms" (Nonlinear least squares) involves derivatives [1]. In general, gradient free learning in deep learning or classical algorithms are not in the main stream. One promising alternative is simulated annealing [2, 3], so-called 'nature-inspired optimization'.
Is there any inherent theoretical reason that why gradient free deep learning (statistical learning) is not in the main stream? (Or not preferred?)
Notes
[1] Such as Levenberg–Marquardt
[2] Simulated Annealing Algorithm for Deep Learning (2015)
[3] CoolMomentum: a method for stochastic optimization by Langevin dynamics with simulated annealing (2021) Though this is still not fully gradient-free, but does not require auto-differentiation.
Edit 1
Additional references using Ensemble Kalman Filter, showing a derivative free approach:

*

*Ensemble Kalman Inversion: A Derivative-Free Technique For Machine Learning Tasks arXiv:1808.03620.

*Ensemble Kalman Filter optimizing Deep Neural Networks: An alternative approach to non-performing Gradient Descent springer (manuscript-pdf)

Edit 2
As far as I gather, Yann LeCun does not consider gradient-free learning as part of deep learning ecosystem. "DL is constructing networks of parameterized functional modules & training them from examples using gradient-based optimization." tweet
Edit 3
Ben Bolker's comment on local geometry definitely deserves to be one of the answers.

Great question! To put it briefly, "Gradient Free Learning" (i.e. "metaheuristics", as pointed out by @user0123456789) is usually used when the "gradient" (i.e. derivative) of the loss function can not be evaluated. This can occur in instances such as :

*

*The derivative of the loss function does not exist (e.g. contains "indicator functions", piecewise functions)


*The derivative of the loss function exists, but is very costly to evaluate (e.g. I have heard talks in which gradient free optimization techniques were suggested for various problems involving reinforcement learning)


*Discrete Combinatorics/Optimization problems (this is kind of related to the first point, but imagine trying to optimize functions in which the inputs are a set of discrete objects and the output is a value associated with different inputs - for example: travelling salesman problem, knapsack optimization, scheduling, etc.)


*Gradient Free Optimization Techniques (e.g. Evolutionary Algorithms, Genetic Algorithm, Simulated Annealing, Particle Swarm, etc.) are sometimes preferred for certain types problems such as "games", in which optimal strategies are developed by mutating and combining random strategies according to their performance with respect to some target (e.g. https://en.wikipedia.org/wiki/Neuroevolution_of_augmenting_topologies , https://www.youtube.com/watch?v=OGHA-elMrxI)
The other note that I wanted to add was that in situations where the gradient of the loss function can be evaluated (e.g. classic MLP neural networks), I think there might be some theoretical results that guarantee the probabilistic convergence of Stochastic Gradient Descent (i.e. the opposite of gradient free learning) to a global maximum provided infinite iterations (I could be wrong about this) - with gradient free optimization techniques, as far I know there is no such guarantee  (over here, I myself asked a question about the "Schema Theorem", which uses Markov Chains to supposedly guarantee an improvement in results as the number of iterations in the Genetic Algorithm increases https://math.stackexchange.com/questions/4295279/does-the-following-computer-science-optimization-theorem-have-a-proof).
To sum everything up - chances are that if the derivative of your loss function "exists", try using classical gradient based optimization techniques. If the derivative does not exist, consider using Gradient Free based techniques.
For example, over here I asked a question about identifying "clusters" in a dataset such that the "proportion of zeros in all columns for a given cluster" is minimized. As far as I can think, there is no standard "gradient" in this problem, making it an ideal choice for "gradient free optimization techniques": https://or.stackexchange.com/questions/7488/mixed-integer-programming-optimization-using-the-genetic-algorithm
Hope this helps!
