I wonder why gradient descent (gating all compute&#x2F;memory operations with parameters, gradient updates to parameter vectors along sample loss gradient) ended up dominant over genetic programming (algorithms as parse trees, mutate&#x2F;recombine to minimize sample loss). Both seem equally theoretically unjustified, although I guess SGD is more friendly for gpus.<p>Also I wonder if all programming should just be done by showing a sgd-solver example i&#x2F;o. Of course this notion has been around since the 80s with prolog but apparently representing &quot;concepts&quot; as 1000 dimension feature vectors is more robust than single boolean variables.