Answers.Body: <p>Deep Learning is not a generalization of Hopfield networks. Deep Learning is a &quot;generalization&quot; of the neural networks/connectionism field started by Rumelhart and McClelland.</p>
  <p>There are two kinds of neural networks:</p>
  <ul>
  <li>Directed (Perceptron, MLP, ConvNets, RNNs, etc.)</li>
  <li>Undirected (Hopfield Nets, Boltzmann Machines, Energy-based models, etc.)</li>
  </ul>
  <p>Any of these can be made deep. As you said, Boltzmann machines are the probabilistic version of Hopfield Networks, and there has been a lot more work on deepifying these models than Hopfield nets: Deep Boltzmann machines, Deep Belief Networks, and deep energy models. Hinton is really the guy you want to read to learn about these models, but you can have a look at <a href="http://www.icml-2011.org/papers/557_icmlpaper.pdf" rel="nofollow noreferrer">this paper</a> which compares the three models.</p>
  <p>Not sure about the Gestalt organisation. I guess I'll leave that to your interpretation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-02T15:17:29.617Z
Answers.Text: My question wasn't probably clear enough. I was asking about the emergence of the ability to categorise (Gestalt) in NN.
Answers.UserDisplayName: Mario Alemi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-04T09:48:47.563Z
Answers.Text: @MarioAlemi This is an old question, but let me comment because your question needs to be improved. You don't seem to be asking about the ability to categorize in neural networks. You probably already know (even back then when you asked this) that neural networks are able to classify given multiple features of the input. Wouldn't that be the Gestalt? If that was your question, what does the Hopfield network have to do with this? Please, edit your post to clarify what your question really is. Indeed, to me, it's not clear what you're asking, especially after your comment above.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-04T09:48:51.047Z
Answers.Text: @MarioAlemi If you're asking "are today's neural networks a generalisation of Hopfield networks?" (which is what you're literally asking), then this answer seems to answer that question (in the first sentence!).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-27T06:39:56.280Z
Answers.LastActivityDate: 2019-04-27T06:39:56.280Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: samlaf
Body: <p>Hopfield networks are able to store a vector and retrieve it starting from a noisy version of it. They do so setting weights in order to minimize the energy function when all neurons are set equal to the vector values, and retrieve the vector using the noisy version of it as input and allowing the net to settle to an energy minimum. </p>
  
  <p>Leaving aside problems like the fact that there is no guarantee that the net will settle in the nearest minimum etc – problems eventually solved with Boltzmann machines and eventually with back-propagation – the breakthrough was they are a starting point for having abstract representations. Two versions of the same document would recall the same state, they would be represented, in the network, by the same state. </p>
  
  <p>As Hopfield himself wrote in his 1982 paper <a href="https://bi.snu.ac.kr/Courses/g-ai09-2/hopfield82.pdf" rel="nofollow noreferrer">Neural networks and physical systems with emergent collective computational abilities</a></p>
  
  <blockquote>
    <p>The present modeling might then be related to how an entity or Gestalt is remembered or categorized on the basis of inputs representing a collection of its features.</p>
  </blockquote>
  
  <p>On the other side, the breakthrough of deep learning was the ability to build multiple, hierarchical representations of the input, eventually leading to making AI-practitioners' life easier, simplifying feature engineering. (see e.g. <a href="https://arxiv.org/pdf/1206.5538.pdf" rel="nofollow noreferrer">Representation Learning: A Review and New Perspectives</a>, Bengio, Courville, Vincent).</p>
  
  <p>From a conceptual point of view, I believe one can see deep learning as a generalization of Hopfield nets: from one single representation to a hierarchy of representation.</p>
  
  <p><em>Is that true from a computational/topological point of view as well? Not considering how "simple" Hopfield networks were (2-state neurons, undirected, energy function), can one see each layer of a network as a Hopfield network and the whole process as a sequential extraction of previously memorized Gestalt, and a reorganization of these Gestalt?</em></p>
  
Tags: <neural-networks><deep-learning><deep-neural-networks><topology><hopfield-network>
Title: Can layers of deep neural networks be seen as Hopfield networks?
UserDisplayName: Mario Alemi
ThreadText: # Can layers of deep neural networks be seen as Hopfield networks?
  
  (Asked by: username_0 on 2018-09-03)
  
  Hopfield networks are able to store a vector and retrieve it starting from a noisy version of it. They do so setting weights in order to minimize the energy function when all neurons are set equal to the vector values, and retrieve the vector using the noisy version of it as input and allowing the net to settle to an energy minimum.
  
  Leaving aside problems like the fact that there is no guarantee that the net will settle in the nearest minimum etc – problems eventually solved with Boltzmann machines and eventually with back-propagation – the breakthrough was they are a starting point for having abstract representations. Two versions of the same document would recall the same state, they would be represented, in the network, by the same state.
  
  As Hopfield himself wrote in his 1982 paper [Neural networks and physical systems with emergent collective computational abilities](https://bi.snu.ac.kr/Courses/g-ai09-2/hopfield82.pdf)
  
  > The present modeling might then be related to how an entity or Gestalt is remembered or categorized on the basis of inputs representing a collection of its features.
  
  On the other side, the breakthrough of deep learning was the ability to build multiple, hierarchical representations of the input, eventually leading to making AI-practitioners' life easier, simplifying feature engineering. (see e.g. [Representation Learning: A Review and New Perspectives](https://arxiv.org/pdf/1206.5538.pdf), Bengio, Courville, Vincent).
  
  From a conceptual point of view, I believe one can see deep learning as a generalization of Hopfield nets: from one single representation to a hierarchy of representation.
  
  *Is that true from a computational/topological point of view as well? Not considering how "simple" Hopfield networks were (2-state neurons, undirected, energy function), can one see each layer of a network as a Hopfield network and the whole process as a sequential extraction of previously memorized Gestalt, and a reorganization of these Gestalt?*
  
  **Tags:** neural-networks, deep-learning, deep-neural-networks, topology, hopfield-network
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-04-27. Score: 3
  
  Deep Learning is not a generalization of Hopfield networks. Deep Learning is a "generalization" of the neural networks/connectionism field started by Rumelhart and McClelland.
  
  There are two kinds of neural networks:
  
  * Directed (Perceptron, MLP, ConvNets, RNNs, etc.)
  * Undirected (Hopfield Nets, Boltzmann Machines, Energy-based models, etc.)
  
  Any of these can be made deep. As you said, Boltzmann machines are the probabilistic version of Hopfield Networks, and there has been a lot more work on deepifying these models than Hopfield nets: Deep Boltzmann machines, Deep Belief Networks, and deep energy models. Hinton is really the guy you want to read to learn about these models, but you can have a look at [this paper](http://www.icml-2011.org/papers/557_icmlpaper.pdf) which compares the three models.
  
  Not sure about the Gestalt organisation. I guess I'll leave that to your interpretation.
  
  > **Comments:**
  > **username_0**: My question wasn't probably clear enough. I was asking about the emergence of the ability to categorise (Gestalt) in NN.  
  > **username_2**: @MarioAlemi This is an old question, but let me comment because your question needs to be improved. You don't seem to be asking about the ability to categorize in neural networks. You probably already know (even back then when you asked this) that neural networks are able to classify given multiple features of the input. Wouldn't that be the Gestalt? If that was your question, what does the Hopfield network have to do with this? Please, edit your post to clarify what your question really is. Indeed, to me, it's not clear what you're asking, especially after your comment above.  
  > **username_2**: @MarioAlemi If you're asking "are today's neural networks a generalisation of Hopfield networks?" (which is what you're literally asking), then this answer seems to answer that question (in the first sentence!).  
  
  ---
  
  