Answers.Body: <blockquote>
  <ol>
  <li>Is this really the self attention mentioned throughout &quot;attention is all you need&quot;?</li>
  </ol>
  </blockquote>
  <p>I think that the crucial point that you are missing is the difference between the Transformer architecture and recurrent neural network (RNN) architecture.</p>
  <p>In the RNNs the context information is stored in the hidden state <span class="math-container">$h_t$</span> which is modified and passed onto the next step of the sequence. In its basic implementation the recurrent approach suffers from the <a href="https://en.wikipedia.org/wiki/Vanishing_gradient_problem#Recurrent_network_model" rel="nofollow noreferrer">vanishing gradient problem</a>. Different approaches was first introduced to mitigate this problem: <a href="https://en.wikipedia.org/wiki/Long_short-term_memory" rel="nofollow noreferrer">LSTM</a>, <a href="https://en.wikipedia.org/wiki/Gated_recurrent_unit" rel="nofollow noreferrer">GRU</a> and other flavors of the same general idea - have a multiplicative weight that learns the relative importance of the context data and controls the flow of information between different steps. &quot;Attention&quot; was just another name given to the similar idea.</p>
  <p>The &quot;all you need&quot; part of the <a href="https://arxiv.org/abs/1706.03762" rel="nofollow noreferrer">Attention is all you need</a> paper is about dropping the RNN architecture altogether. Quoting the paper: &quot;<em>In this work we propose the Transformer, a model architecture eschewing recurrence and instead
  relying entirely on an attention mechanism to draw global dependencies between input and output.</em>&quot;</p>
  <blockquote>
  <ol start="2">
  <li>Did I understand this paper correctly, and indeed other than the LSTM, there are no relationships between the different time stamps</li>
  </ol>
  </blockquote>
  <p>I'd say that yes, you did understand the paper correctly - most of the researchers was relying on the RNN architecture to extract the &quot;interaction&quot; information between different tokens. It looks obvious now, but at that time the transformers paper was revolutionary.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-03T12:06:18.677Z
Answers.Text: Thanks. Regarding 1. I actually didn't miss the point, but noticed that this is called self attention, and saw light after the attention paper, which led me to believe it is somehow similar. I am still missing what do they mean in "all you need" when they refer to attention everywhere, since I now see it is nothing more than a global MLP.
Answers.UserDisplayName: Gulzar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-03T12:26:57.653Z
Answers.Text: @Gulsar Can I suggest you to read the "1.Introduction" section of the "Attention Is All You Need" paper?
Answers.UserDisplayName: Kostya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-03T12:30:28.070Z
Answers.Text: This is the 1st thing I did, what exactly are you referring to?
Answers.UserDisplayName: Gulzar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-03T19:20:53.727Z
Answers.Text: @Gulzar I thought that the first section of the paper is answering the question « what do they mean by ‘all you need’ » pretty unambiguously
Answers.UserDisplayName: Kostya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-04T15:42:28.027Z
Answers.Text: Since there isn't a single reference there to self-attention, I ask again for the quote you are refering to.
Answers.UserDisplayName: Gulzar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-03T10:29:42.457Z
Answers.LastActivityDate: 2023-04-03T10:29:42.457Z
Answers.UserDisplayName: Kostya
Body: <p>During trying to understand transformers by reading <a href="https://arxiv.org/abs/1706.03762" rel="nofollow noreferrer">Attention is all you need</a>, I noticed the authors constantly refer to &quot;self attention&quot; without explaining it.</p>
  <p>The original attention mechanism is introduced in <a href="https://arxiv.org/abs/1409.0473" rel="nofollow noreferrer">NEURAL MACHINE TRANSLATION
  BY JOINTLY LEARNING TO ALIGN AND TRANSLATE</a>, in which a the relationship between each output and each input is encoded into a matrix whose size is dependent on input and output sizes.</p>
  <p>It seems the paper which first introduced self-attention is <a href="https://arxiv.org/abs/1703.03130" rel="nofollow noreferrer">A STRUCTURED SELF-ATTENTIVE
  SENTENCE EMBEDDING</a>.</p>
  <p>I am not finding similar ideas in this paper, and would like an explanation.</p>
  <p>In this paper, self-attention is defined like so</p>
  <p>Given <span class="math-container">$H=(h_1, h_2, ... h_n)$</span>, a sequence of hidden bidirectional LSTM states, each of dimension <span class="math-container">$2u$</span>, they try to encode the sentence into a fixed length vector. They do so by weighting the hidden states according to the probabilities (weights) vector <span class="math-container">$a$</span> defined as</p>
  <p><span class="math-container">$a = softmax(W_{s_2}tanh(W_{s_1}H^T))$</span></p>
  <p>This can be read as a multi layer perceptron, <span class="math-container">$A$</span>, operating on vectors of size <span class="math-container">$2u$</span>, with a single layer of activation, with activation function <span class="math-container">$tanh$</span>, and output size of 1. <span class="math-container">$A$</span>'s output units are log probabilities, meaning &quot;how likely is timestamp <span class="math-container">$i$</span> to be important for the sentence embedding&quot;.
  <span class="math-container">$A$</span> is then used on the <span class="math-container">$n$</span> hidden states, <span class="math-container">$H$</span>, to obtain <span class="math-container">$n$</span> weights, one for each hidden state <span class="math-container">$h_i$</span>.</p>
  <p>Then they go on to extend this from dimension <span class="math-container">$1$</span> to <span class="math-container">$r$</span>.</p>
  <p>This is all well, but I fail to see what good this <span class="math-container">$A$</span> actually does - <strong>it does not use any relationship between two different states <span class="math-container">$h_i, h_j, i \ne j$</span> other than that already found by the LSTM, thus (in my mind,) defeating the purpose</strong>.</p>
  <p>I expected something like the original attention - creating a <span class="math-container">$nxn$</span> matrix, and have some MLP <span class="math-container">$a(h_i, h_j)$</span> calculate each cell, thus encoding probability of the relationship between the cells. This was not done here.</p>
  <hr />
  <p>My questions</p>
  <ol>
  <li>Is this really the self attention mentioned throughout &quot;attention is all you need&quot;? if not, please refer me to the correct paper</li>
  <li>Did I understand this paper correctly, and indeed other than the LSTM, there are no relationships between the different time stamps, thus defeating the purpose of encoding the sentence better than the LSTM? [we could just add another layer into the LSTM, isn't it the same or better?]</li>
  <li>Please explain the point of this paper. Adding a global MLP on all LSTM states? That's it?</li>
  <li>Why only a single layer? why <span class="math-container">$tanh$</span>? why not just a single vector of
  size <span class="math-container">$1x2h$</span> and no activation at all?</li>
  </ol>
  
Tags: <papers><transformer><attention><sequence-modeling>
Title: Understanding self attention - How come there is no connection between different states?
UserDisplayName: Gulzar
ThreadText: # Understanding self attention - How come there is no connection between different states?
  
  (Asked by: username_0 on 2023-04-03)
  
  During trying to understand transformers by reading [Attention is all you need](https://arxiv.org/abs/1706.03762), I noticed the authors constantly refer to "self attention" without explaining it.
  
  The original attention mechanism is introduced in [NEURAL MACHINE TRANSLATION
  BY JOINTLY LEARNING TO ALIGN AND TRANSLATE](https://arxiv.org/abs/1409.0473), in which a the relationship between each output and each input is encoded into a matrix whose size is dependent on input and output sizes.
  
  It seems the paper which first introduced self-attention is [A STRUCTURED SELF-ATTENTIVE
  SENTENCE EMBEDDING](https://arxiv.org/abs/1703.03130).
  
  I am not finding similar ideas in this paper, and would like an explanation.
  
  In this paper, self-attention is defined like so
  
  Given $H=(h\_1, h\_2, ... h\_n)$, a sequence of hidden bidirectional LSTM states, each of dimension $2u$, they try to encode the sentence into a fixed length vector. They do so by weighting the hidden states according to the probabilities (weights) vector $a$ defined as
  
  $a = softmax(W\_{s\_2}tanh(W\_{s\_1}H^T))$
  
  This can be read as a multi layer perceptron, $A$, operating on vectors of size $2u$, with a single layer of activation, with activation function $tanh$, and output size of 1. $A$'s output units are log probabilities, meaning "how likely is timestamp $i$ to be important for the sentence embedding".
  $A$ is then used on the $n$ hidden states, $H$, to obtain $n$ weights, one for each hidden state $h\_i$.
  
  Then they go on to extend this from dimension $1$ to $r$.
  
  This is all well, but I fail to see what good this $A$ actually does - **it does not use any relationship between two different states $h\_i, h\_j, i \ne j$ other than that already found by the LSTM, thus (in my mind,) defeating the purpose**.
  
  I expected something like the original attention - creating a $nxn$ matrix, and have some MLP $a(h\_i, h\_j)$ calculate each cell, thus encoding probability of the relationship between the cells. This was not done here.
  
  ---
  
  My questions
  
  1. Is this really the self attention mentioned throughout "attention is all you need"? if not, please refer me to the correct paper
  2. Did I understand this paper correctly, and indeed other than the LSTM, there are no relationships between the different time stamps, thus defeating the purpose of encoding the sentence better than the LSTM? [we could just add another layer into the LSTM, isn't it the same or better?]
  3. Please explain the point of this paper. Adding a global MLP on all LSTM states? That's it?
  4. Why only a single layer? why $tanh$? why not just a single vector of
     size $1x2h$ and no activation at all?
  
  **Tags:** papers, transformer, attention, sequence-modeling
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-04-03. Score: 1
  
  > 1. Is this really the self attention mentioned throughout "attention is all you need"?
  
  I think that the crucial point that you are missing is the difference between the Transformer architecture and recurrent neural network (RNN) architecture.
  
  In the RNNs the context information is stored in the hidden state $h\_t$ which is modified and passed onto the next step of the sequence. In its basic implementation the recurrent approach suffers from the [vanishing gradient problem](https://en.wikipedia.org/wiki/Vanishing_gradient_problem#Recurrent_network_model). Different approaches was first introduced to mitigate this problem: [LSTM](https://en.wikipedia.org/wiki/Long_short-term_memory), [GRU](https://en.wikipedia.org/wiki/Gated_recurrent_unit) and other flavors of the same general idea - have a multiplicative weight that learns the relative importance of the context data and controls the flow of information between different steps. "Attention" was just another name given to the similar idea.
  
  The "all you need" part of the [Attention is all you need](https://arxiv.org/abs/1706.03762) paper is about dropping the RNN architecture altogether. Quoting the paper: "*In this work we propose the Transformer, a model architecture eschewing recurrence and instead
  relying entirely on an attention mechanism to draw global dependencies between input and output.*"
  
  > 2. Did I understand this paper correctly, and indeed other than the LSTM, there are no relationships between the different time stamps
  
  I'd say that yes, you did understand the paper correctly - most of the researchers was relying on the RNN architecture to extract the "interaction" information between different tokens. It looks obvious now, but at that time the transformers paper was revolutionary.
  
  > **Comments:**
  > **username_0**: Thanks. Regarding 1. I actually didn't miss the point, but noticed that this is called self attention, and saw light after the attention paper, which led me to believe it is somehow similar. I am still missing what do they mean in "all you need" when they refer to attention everywhere, since I now see it is nothing more than a global MLP.  
  > **username_1**: @Gulsar Can I suggest you to read the "1.Introduction" section of the "Attention Is All You Need" paper?  
  > **username_0**: This is the 1st thing I did, what exactly are you referring to?  
  > **username_1**: @Gulzar I thought that the first section of the paper is answering the question « what do they mean by ‘all you need’ » pretty unambiguously  
  > **username_0**: Since there isn't a single reference there to self-attention, I ask again for the quote you are refering to.  
  
  ---
  
  