Deep Neural Networks learn layers of linear functions - represented by adjustable &#x27;learning&#x27; weighted connections.<p>Each layer of neuron activations is a new representation of the data - produced by the weighted connections.<p>Function compositions of linear functions are still only a linear function.<p>Each neuron in a layer sums it&#x27;s weighted inputs this summation is a non-linearity that allows layers to be composed - function composition.<p>This is famously expressed in Minsky and Papert&#x27;s 1968 &#x27;Perceptron&#x27;: a single layer of network weights is incapable of learning XOR.<p>One analysis is that Neural nets transform the shape of the data until a single line on a twisty high dimensional manifold produces the desired distinction. 
<a href="http:&#x2F;&#x2F;colah.github.io&#x2F;posts&#x2F;2014-03-NN-Manifolds-Topology&#x2F;" rel="nofollow">http:&#x2F;&#x2F;colah.github.io&#x2F;posts&#x2F;2014-03-NN-Manifolds-Topology&#x2F;</a><p>A single layer network is a universal approximator and a net can be trained or distilled from another net - but deep nets are overwhelmingly better at the initial learning and discovery.<p>Neural nets have been related to Kadanof&#x27;s spin glasses, suggesting learning is alike to phase transitions or sand pile collapse where small local changes can produce profound global changes.
<a href="https:&#x2F;&#x2F;charlesmartin14.wordpress.com&#x2F;2015&#x2F;04&#x2F;01&#x2F;why-deep-learning-works-ii-the-renormalization-group&#x2F;" rel="nofollow">https:&#x2F;&#x2F;charlesmartin14.wordpress.com&#x2F;2015&#x2F;04&#x2F;01&#x2F;why-deep-le...</a><p>Generally when training nets the learning initially learns a lot very quickly, most of the error vanishes at the start.<p>Word2Vec demonstrates that nets learn very powerful representations, that word2vec vector algebra is semantic points to unexpectedly powerful representations.<p>Similar semantic vector math can be performed on images and the same vectors can translate modalities, e.g. text to images.
<a href="https:&#x2F;&#x2F;github.com&#x2F;Newmu&#x2F;dcgan_code#arithmetic-on-faces" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;Newmu&#x2F;dcgan_code#arithmetic-on-faces</a><p>Natural Evolution produces efficient solutions.<p>I propose the successes of deep learning so far are partially explicable because they are working within human culture and perception, relearning our very efficient solutions - akin to distilling a deepnet into a shallow one.<p>This hypothesis will be tested if embodied deep neural nets using re-inforcement learning discover their own efficient solutions to performing tasks in the real world - robotics.<p>IMHO Peter Abeel&#x27;s deep net, learning to robustly output robot motor torques directly from camera pixels will show if embodied deep nets can do discovery rather than relearning what we know. <a href="http:&#x2F;&#x2F;www.cs.berkeley.edu&#x2F;~pabbeel&#x2F;research_rll.html" rel="nofollow">http:&#x2F;&#x2F;www.cs.berkeley.edu&#x2F;~pabbeel&#x2F;research_rll.html</a>