Regardless of whether this post is right or wrong, it is a showcase for the kind of out-of-the-box thinking that can force people within and from outside the field to see deep neural nets from a different angle. For that alone, I think this is a fantastic essay.<p>Some comments:<p>The author writes: &quot;using multiple copies of a neuron in different places is the neural network equivalent of using functions. Because there is less to learn, the model learns more quickly and learns a better model. This technique – the technical name for it is &#x27;weight tying&#x27; – is essential to the phenomenal results we’ve recently seen from deep learning. Of course, one can&#x27;t just arbitrarily put copies of neurons all over the place. For the model to work, you need to do it in a principled way, exploiting some structure in your data. In practice, there are a handful of patterns that are widely used, such as recurrent layers and convolutional layers.&quot;<p>Recurrent and convolutional layers are not just two examples of widely used &quot;weight tying&quot; patterns; they are THE TWO MAJOR WAYS in which virtually everyone &quot;ties&quot; weights -- across time or pixel space, usually.  With the exception of tiny fully-connected models used to solve relatively small problems, every successful deep neural net model of meaningful scale I know of is either a convnet or an RNN!<p>If anyone knows of a counter example, I&#x27;d love to hear about it!<p>I would also have pointed out that &quot;weight tying&quot; not only allows the deep neural net to learn more quickly; it also <i>massively</i> reduces the &quot;true&quot; number of parameters required to specify the model, thereby reducing by a similar proportion the number of samples necessary to have reasonable PAC learning guarantees.