My super-hand-waving explanation of why machine learning works:<p>MNIST is a handwritten digit database.  Each 784-pixel image (28x28) corresponds to a digit from 0 to 9.  As a pure mathematical construction, there are at most 2^784 inputs possible, and a small number of possible outputs.<p>So if you have 784 completely different ways of analyzing the image, and you combine them in the right ways, you will get roughly an approximation of an answer.  This is a tautology if the 784 ways are &quot;the value of each pixel&quot; and the combinations are &quot;magic&quot;, but if you have more &quot;intelligent&quot; combinations you should have combinations that are less magic.  And in this case, since humans can generally determine the digit value from a 7-light display, it seems reasonable that there exists some way to have &quot;intelligent&quot; combinations such that they combine to form a neural network that solves the problem of digit identification.<p>And that (still hand-wavy) explanation can also plausibly describe how a human would describe identifying a number.  If I ask you &quot;why is this a 1 and not a 3&quot;, you might say &quot;because it&#x27;s straight&quot; or &quot;because it&#x27;s narrow&quot; or &quot;because it doesn&#x27;t have a point in the middle&quot; or any number of other descriptions of the object.  So you can envision a 2-layer network where the middle layer calculates this (and due to the structure of images, in practice it might better be a 3 or 4-layer network. but the important point is that the search algorithms don&#x27;t rely on it or you knowing what these middle layers are ahead of time)<p>Which only leaves the question of how &quot;neural network learning&quot; is supposed to find this.  And there are a few heuristics which combine to (in practice) be a very effective search.  We have back-propagation (which is much easier with automatic differentiation), so we can adjust the entirety of the network based on the output.  (and it&#x27;s an axiom that if you have a lot of things, they will be similar in the ways they are the same, and different [hopefully in some regular way] in the ways they are not the same).  We have drop-off, where we attempt to prune connections that are irrelevant.  We can add new connections to see if they are relevant.  We can do any number of hill-climbing algorithms on the output of the fitness function.  And, as a valid search algorithm, it tends to converge to a valid result.<p>Obviously none of this is at all rigorous, but if you know the math here enough I don&#x27;t think you&#x27;re asking the questions in this article.