FAIRie here. GA is essentially guess-and-check. SGD doesn&#x27;t require guess or even check; if your error function is differentiable, and you didn&#x27;t screw up the chain rule, you know what direction to head in to make it go down. For large models with lots of parameters, finding a setting for each parameter that makes the model as a whole go down by random choice has complexity n * f(n) for some monotonically increasing f, while SGD really finds you your next model, with decent guarantees it will be better, in O(n) time.<p>The theoretical hand-wringing about SGD for neural nets is that their loss surface isn&#x27;t convex. It turns out this doesn&#x27;t matter. The loss surface is a high-dimensional egg-carton, and you need to get winning-the-lottery-while-struck-by-lightning unlucky to find a significantly shallow local minimum. There <i>are</i> lots of saddle points, and you need to do something to drive out of those, but stochasticity seems sufficient in practice.