Answers.Body: <p>Will try to formulate my understanding of the ideas in this paper, mention my own concerns that I see are relevant to your question, and see if we can identify any confusions along the way that might clarify the issue</p>
  <p>On eq(6) of <a href="https://ml-jku.github.io/hopfield-layers/" rel="nofollow noreferrer">the relevant blog post</a>, they identify the weight matrix of a discrete, binary Hopfield network as</p>
  <p><span class="math-container">$$ \boldsymbol{W} = \sum_i^N x_i x_i^T $$</span></p>
  <p>with N raw stored entries, that are retrieved by iterating the initial guess <span class="math-container">$\xi$</span> with the following update rule</p>
  <p><span class="math-container">$$ \xi_{t+1} = \text{sgn}( \boldsymbol{W} \xi_{t} - b ) $$</span></p>
  <p>Now to the paper in question, the update rule for the generalization they propose for continuous states that would be used is (eq 22 in OP):</p>
  <p><span class="math-container">$$ \xi_{t+1} = \boldsymbol{X} \text{softmax}( \beta \boldsymbol{X}^T \xi_{t} ) $$</span></p>
  <p>Where <span class="math-container">$\boldsymbol{X} = (x_0, x_1, \dots , x_N ) $</span></p>
  <p>The first substantial difference I see is that, while on the case of binary entries, all the weights of the network are encoded in the matrix <span class="math-container">$\boldsymbol{W}$</span>, hence the storage is constant regardless of how many actual patterns are stored in it. In contrast, in this continuous case generalized rule, the <span class="math-container">$\boldsymbol{X}$</span> matrix seems to grow linearly with the size of entries, in fact it keeps all the stored entries directly. By their update rule, it doesn't seem to be a way around storing and keeping around the entire entries, and the update rule seems to only find a &quot;best fit&quot; among the entries, using the scalar dot product of the attention mechanism. I still think I might be missing something important here</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-02T18:41:11.527Z
Answers.LastActivityDate: 2021-02-02T18:46:16.707Z
Answers.LastEditDate: 2021-02-02T18:46:16.707Z
Answers.UserDisplayName: lurscher
Body: <p>In the paper <a href="https://arxiv.org/pdf/2008.02217.pdf" rel="nofollow noreferrer">Hopfield networks is all you need</a>, the authors mention that their modern Hopfield network layers are a good replacement for pooling, GRU, LSTM, and attention layers, and tend to outperform them in various tasks.</p>
  <p>I understand that they show that the layers can store an exponential amount of vectors, but that should still be worse than attention layers that can focus parts of an arbitrary length input sequence.</p>
  <p>Also, in their paper, they briefly allude to Neural Turing Machine and related memory augmentation architectures, but do not comment on the comparison between them.</p>
  <p>Has someone studied how these layers help improve the performance over pooling and attention layers, and is there any comparison between replacing layers with Hopfield layers vs augmenting networks with external memory like Neural Turing Machines?</p>
  <p><strong>Edit 29 Jan 2020</strong>
  I believe my intuition that attention mechanism should outperform hopfield layers was wrong, as I was comparing the hopfield layer that uses an input vector for query <span class="math-container">$R (\approx Q)$</span> and stored patterns <span class="math-container">$Y$</span> for both Key <span class="math-container">$K$</span> and Values <span class="math-container">$V$</span>. In this case my assumption was that hopfield layer would be limited by its storage capacity while attention mechanism does not have such constraints.</p>
  <p>However the authors do mention that the input <span class="math-container">$Y$</span> may be modified to ingest two extra input vectors for Key and Value. I believe in this case it would perform hopfield network mapping instead of attention and I do not know how the 2 compare.
  <a href="https://i.sstatic.net/CIz3W.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/CIz3W.png" alt="Hopfield Layer - Attention analogy" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-28T12:09:32.917Z
Comments.Text: The paper that you're mentioning "Hopfield networks is all you need" is from 2020 (so very recent) and is apparently only a pre-print (i.e. it's not yet been published in any journal or conference proceedings). There's [this blog post](https://ml-jku.github.io/hopfield-layers/) and the [code](https://github.com/ml-jku/hopfield-layers). Have you already looked at them? Also, why are you claiming this "but that should still be worse than attention layers that can focus parts of an arbitrary length input sequence."? Can you clarify that? Please, edit your post to clarify that.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-28T12:11:09.310Z
Comments.Text: Moreover, please, put your main **specific** question in the title, otherwise, this post looks like you're asking us to review a 95-papes paper, which is of course off-topic here, as it's too broad.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-29T13:10:02.487Z
Comments.Text: @nbro I understand that the question is broader than anyone (and I) would like it to be, but since my question is if someone knows the reasoning behind the performance claims of the network, how would you re-phrase it?
  I am updating my question regarding my intuition of attention vs hopfield as I realized an error in my assumption from previous reading.
Comments.UserDisplayName: Rijul Gupta
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-29T20:51:15.293Z
Comments.Text: I suggest that you re-write your post as if there was no edit because I don't think that anyone is already writing an answer to your question(s). While reviewing your post, try to be more specific and now that you have a clearer idea of what you want to ask, ask it _explicitly_. Moreover, if they are showing the results of the experiments that show that their model outperforms other models, what else do you want to get as an answer?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-29T20:53:02.750Z
Comments.Text: For example, when you ask "Has someone studied how these layers help improve the performance over pooling and attention layers", but, in the paper, aren't they showing, with experiments, that this is the case? Or are they just claiming that? It may be a good idea to quote the parts of the paper where they claim (without proving) what you want to be proved.
Comments.UserDisplayName: nbro
Tags: <neural-networks><attention><neural-turing-machine><hopfield-network>
Title: Reasoning behind performance improvement with hopfield networks
UserDisplayName: Rijul Gupta
ThreadText: # Reasoning behind performance improvement with hopfield networks
  
  (Asked by: username_0 on 2021-01-28)
  
  In the paper [Hopfield networks is all you need](https://arxiv.org/pdf/2008.02217.pdf), the authors mention that their modern Hopfield network layers are a good replacement for pooling, GRU, LSTM, and attention layers, and tend to outperform them in various tasks.
  
  I understand that they show that the layers can store an exponential amount of vectors, but that should still be worse than attention layers that can focus parts of an arbitrary length input sequence.
  
  Also, in their paper, they briefly allude to Neural Turing Machine and related memory augmentation architectures, but do not comment on the comparison between them.
  
  Has someone studied how these layers help improve the performance over pooling and attention layers, and is there any comparison between replacing layers with Hopfield layers vs augmenting networks with external memory like Neural Turing Machines?
  
  **Edit 29 Jan 2020**
  I believe my intuition that attention mechanism should outperform hopfield layers was wrong, as I was comparing the hopfield layer that uses an input vector for query $R (\approx Q)$ and stored patterns $Y$ for both Key $K$ and Values $V$. In this case my assumption was that hopfield layer would be limited by its storage capacity while attention mechanism does not have such constraints.
  
  However the authors do mention that the input $Y$ may be modified to ingest two extra input vectors for Key and Value. I believe in this case it would perform hopfield network mapping instead of attention and I do not know how the 2 compare.
  [![Hopfield Layer - Attention analogy](https://i.sstatic.net/CIz3W.png)](https://i.sstatic.net/CIz3W.png)
  
  **Tags:** neural-networks, attention, neural-turing-machine, hopfield-network
  
  > **Comments:**
  > **username_1**: The paper that you're mentioning "Hopfield networks is all you need" is from 2020 (so very recent) and is apparently only a pre-print (i.e. it's not yet been published in any journal or conference proceedings). There's [this blog post](https://ml-jku.github.io/hopfield-layers/) and the [code](https://github.com/ml-jku/hopfield-layers). Have you already looked at them? Also, why are you claiming this "but that should still be worse than attention layers that can focus parts of an arbitrary length input sequence."? Can you clarify that? Please, edit your post to clarify that.  
  > **username_1**: Moreover, please, put your main **specific** question in the title, otherwise, this post looks like you're asking us to review a 95-papes paper, which is of course off-topic here, as it's too broad.  
  > **username_0**: @username_1 I understand that the question is broader than anyone (and I) would like it to be, but since my question is if someone knows the reasoning behind the performance claims of the network, how would you re-phrase it? I am updating my question regarding my intuition of attention vs hopfield as I realized an error in my assumption from previous reading.  
  > **username_1**: I suggest that you re-write your post as if there was no edit because I don't think that anyone is already writing an answer to your question(s). While reviewing your post, try to be more specific and now that you have a clearer idea of what you want to ask, ask it _explicitly_. Moreover, if they are showing the results of the experiments that show that their model outperforms other models, what else do you want to get as an answer?  
  > **username_1**: For example, when you ask "Has someone studied how these layers help improve the performance over pooling and attention layers", but, in the paper, aren't they showing, with experiments, that this is the case? Or are they just claiming that? It may be a good idea to quote the parts of the paper where they claim (without proving) what you want to be proved.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-02-02. Score: 1
  
  Will try to formulate my understanding of the ideas in this paper, mention my own concerns that I see are relevant to your question, and see if we can identify any confusions along the way that might clarify the issue
  
  On eq(6) of [the relevant blog post](https://ml-jku.github.io/hopfield-layers/), they identify the weight matrix of a discrete, binary Hopfield network as
  
  $$ \boldsymbol{W} = \sum\_i^N x\_i x\_i^T $$
  
  with N raw stored entries, that are retrieved by iterating the initial guess $\xi$ with the following update rule
  
  $$ \xi\_{t+1} = \text{sgn}( \boldsymbol{W} \xi\_{t} - b ) $$
  
  Now to the paper in question, the update rule for the generalization they propose for continuous states that would be used is (eq 22 in OP):
  
  $$ \xi\_{t+1} = \boldsymbol{X} \text{softmax}( \beta \boldsymbol{X}^T \xi\_{t} ) $$
  
  Where $\boldsymbol{X} = (x\_0, x\_1, \dots , x\_N ) $
  
  The first substantial difference I see is that, while on the case of binary entries, all the weights of the network are encoded in the matrix $\boldsymbol{W}$, hence the storage is constant regardless of how many actual patterns are stored in it. In contrast, in this continuous case generalized rule, the $\boldsymbol{X}$ matrix seems to grow linearly with the size of entries, in fact it keeps all the stored entries directly. By their update rule, it doesn't seem to be a way around storing and keeping around the entire entries, and the update rule seems to only find a "best fit" among the entries, using the scalar dot product of the attention mechanism. I still think I might be missing something important here
  
  
  ---
  
  