In addition to &quot;We need a cluster for deep learning&quot;, the second most popular mostly untrue thing I hear is &quot;We have no idea how neural networks learn&quot;.<p>However, there are many papers that explore various ways to make a network learn, and they keep improving on performance, suggesting they&#x27;re on to <i>something</i>. There are also many papers that discuss possible theoretical implications of experimental results.<p>But what does Knowing Why The Network Works mean exactly? &quot;It works because universal approximation and gradient descent&quot;, but that&#x27;s not a very satisfactory answer. &quot;It works because it starts at a general solution and, over the course of many iterations, takes many small steps in an ever changing direction defined by a gradient approximation generated by looking at the difference between an average error and a target output (which should trend towards 0)&quot;.<p>What would a satisfactory &quot;why&quot; even look like exactly? As in, what form might it take compared to some other scientific discipline where we <i>do</i> know what&#x27;s going on?<p>Personally, I think the whole thing is a red herring -- people in the field have <i>some idea</i> of how neural nets work, and there are many disciplines considered by many to be mature sciences that are far from settled on a grand theoretical scale.<p>That said, the theory I&#x27;m most interested in is recent attempts to connect a memory module to neural networks so they can &quot;learn&quot; to store important&#x2F;complex&#x2F;distributed information that can be recalled with high accuracy later. That will make it easier to do things like ask a neural network to remember your name, or where you left your keys, or whatever.