Answers.Body: <p>The key difference between the attention mechanisms used in &quot;Learning To Align And Translate&quot; and &quot;Attention Is All You Need&quot; lies in the way that the similarity between the query and key vectors is calculated.</p>
  <h3>&quot;Learning To Align And Translate&quot;</h3>
  <p>The attention score is calculated by a learned function using a feed-forward neural network that takes in the query and key vectors and outputs a (log) probability of correspondence between them. This approach requires the model to learn a mapping from the input and output spaces to a joint space where the vectors can be compared against each other.</p>
  <h3>&quot;Attention Is All You Need&quot;</h3>
  <p>Here the attention is calculated as the similarity between the query and key vectors by taking their dot product and scaling it by the square root of their dimensionality. This approach does not require the model to learn a mapping to a joint space, but instead relies on the inherent structure of the vector space.</p>
  <h2>Pros/Cons</h2>
  <p>One advantage of the approach used in &quot;Attention Is All You Need&quot; is that it is computationally more efficient than the method used in &quot;Learning To Align And Translate&quot;, especially for long sequences. Specifically, the scaled dot product attention is faster compared to &quot;general/Bahdanau attention&quot; in the sense that the latter is a learnt via a usually shallow feedforward neural network. In that sense, overhead space and time complexity is added while traversing the computational graph of the model as part of training.</p>
  <p>That being said, there have been studies that have explored the use of different attention mechanisms in Transformers, including variants of the dot product and learned similarity functions. While the dot product attention used in &quot;Attention Is All You Need&quot; has shown to be effective in many cases, other mechanisms may be more appropriate for certain tasks or data types.</p>
  <p>I copy below recent studies relevant to transformer variations and its attention mechanism.</p>
  <ul>
  <li>Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with
  sparse transformers. CoRR, abs/1904.10509, 2019.</li>
  <li>Nikita Kitaev, L. Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. ArXiv,
  abs/2001.04451, 2020.</li>
  <li>Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, and T. Lillicrap. Compressive transformers for long-range sequence modelling. ArXiv, abs/1911.05507, 2020.</li>
  <li>Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention
  with linear complexity. ArXiv, abs/2006.04768, 2020.</li>
  <li>A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret. Transformers are rnns: Fast autoregressive
  transformers with linear attention. In Proceedings of the International Conference on Machine
  Learning (ICML), 2020.</li>
  <li>Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer:
  Rethinking self-attention in transformer models, 2020.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-20T07:55:00.960Z
Answers.Text: Can you please cite these other studies? I was looking for a more in-depth answer
Answers.UserDisplayName: Gulzar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-20T08:04:33.673Z
Answers.Text: It is also not clear why "all you need"'s mechanism is faster. One can easily parallelize the "Learning To Align And Translate" method as there are no dependencies. Thus, I am not following why long sequences have any effect. Please notice we are disregarding LSTMs here, just the attention mechanism itself.
Answers.UserDisplayName: Gulzar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-20T10:10:15.773Z
Answers.Text: The scaled dot product attention is faster compared to "general/Bahdanau attention" in the sense that the latter is a learnt via a usually shallow feedforward neural network. In that sense, overhead space and time complexity is added while traversing the computational graph of the model as part of training. I will  update my post.
Answers.UserDisplayName: hH1sG0n3
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-21T00:10:40.420Z
Answers.Text: Wow so many references! Thank you that is so much appreciated. If you don't mind, how did you search for these?
Answers.UserDisplayName: Gulzar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-08T12:27:34.980Z
Answers.Text: thank you very much for your input, I was considering this effect. This is very insightful. However I would like to add that dot product attention puts more responsibility on the quality of the embedding than joint attention. Also I haven't yet looked at the reference I think that we could explore deeper network for the joint attention other than a one-layer ffn
Answers.UserDisplayName: Sheed
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-19T11:13:57.253Z
Answers.LastActivityDate: 2023-04-20T10:22:00.390Z
Answers.LastEditDate: 2023-04-20T10:22:00.390Z
Answers.UserDisplayName: hH1sG0n3
Body: <p>As far as I know, attention was first introduced in <a href="https://arxiv.org/abs/1409.0473" rel="nofollow noreferrer">Learning To Align And Translate</a>.</p>
  <p>There, the core mechanism which is able to disregard the sequence length, is a dynamically-built matrix, of shape output_size X input_size, in which every position <span class="math-container">$(o, i)$</span> holds the (log) probability that output <span class="math-container">$o$</span> should attend to input <span class="math-container">$i$</span>.</p>
  <p>That (log) probability is obtained by operating a learned function <span class="math-container">$a(h, s)$</span>, where <span class="math-container">$h$</span> is a hidden state of the input, and <span class="math-container">$s$</span> is a cell state of the output.</p>
  <p>Please let's disregard the fact that these inputs are RNN-based, and only look at the attention mechanism itself - a dynamic matrix of (log) probabilities is built, each slot is built by a function taking in two vectors, and outputting their &quot;correspondence&quot;.</p>
  <hr />
  <p>Jump forward to the iconic <a href="https://arxiv.org/abs/1706.03762" rel="nofollow noreferrer">Attention Is All You Need</a>.</p>
  <p>Please disregard the fact that in this paper, <span class="math-container">$K$</span> was separated from <span class="math-container">$V$</span>, unlike in the previous one.<br />
  I just want to look at the mechanism itself.</p>
  <p>Let's look only at Multi-Head Attention, and in it, let's look only at the part actually doing the attention: <span class="math-container">$ QK^T $</span></p>
  <p>Let's assume <span class="math-container">$Q$</span> and <span class="math-container">$K$</span> are vectors and not matrices, for simplicity. Their attention score is their dot product.</p>
  <hr />
  <p>Let's compare the core attention mechanisms of &quot;align and translate&quot; against &quot;all you need&quot;.</p>
  <p>In &quot;align and translate&quot;, the function learns how two vectors correspond to one another</p>
  <p>In &quot;all you need, the function learns to project embeddings into a continuous space, where they can be compared against other such projections by their dot-product.</p>
  <p>One could easily implement multi-head-attention with the dynamic matrix method, by a function <span class="math-container">$b(k, q)$</span> yielding the (log) probability that the two correspond, and putting that into a dynamic-size matrix.</p>
  <hr />
  <p><strong>My question is what in the &quot;all you need&quot; core attention method makes it better than the &quot;align and translate&quot; core attention method?</strong></p>
  <p>Are there ablation studies for this point?</p>
  <p>My intuition tells me it would be easier for a network to learn how to correspond vectors, rather than to learn an entire continuous space.</p>
  <hr />
  <p>Again, please disregard the other contributions in &quot;all you need&quot;, such as self-attention, separation of key from value, normalization, Transformer, ect.</p>
  
Tags: <papers><transformer><attention><sequence-modeling>
Title: Difference between dot product attention and "matrix attention"
UserDisplayName: Gulzar
ThreadText: # Difference between dot product attention and "matrix attention"
  
  (Asked by: username_0 on 2023-04-16)
  
  As far as I know, attention was first introduced in [Learning To Align And Translate](https://arxiv.org/abs/1409.0473).
  
  There, the core mechanism which is able to disregard the sequence length, is a dynamically-built matrix, of shape output\_size X input\_size, in which every position $(o, i)$ holds the (log) probability that output $o$ should attend to input $i$.
  
  That (log) probability is obtained by operating a learned function $a(h, s)$, where $h$ is a hidden state of the input, and $s$ is a cell state of the output.
  
  Please let's disregard the fact that these inputs are RNN-based, and only look at the attention mechanism itself - a dynamic matrix of (log) probabilities is built, each slot is built by a function taking in two vectors, and outputting their "correspondence".
  
  ---
  
  Jump forward to the iconic [Attention Is All You Need](https://arxiv.org/abs/1706.03762).
  
  Please disregard the fact that in this paper, $K$ was separated from $V$, unlike in the previous one.  
  I just want to look at the mechanism itself.
  
  Let's look only at Multi-Head Attention, and in it, let's look only at the part actually doing the attention: $ QK^T $
  
  Let's assume $Q$ and $K$ are vectors and not matrices, for simplicity. Their attention score is their dot product.
  
  ---
  
  Let's compare the core attention mechanisms of "align and translate" against "all you need".
  
  In "align and translate", the function learns how two vectors correspond to one another
  
  In "all you need, the function learns to project embeddings into a continuous space, where they can be compared against other such projections by their dot-product.
  
  One could easily implement multi-head-attention with the dynamic matrix method, by a function $b(k, q)$ yielding the (log) probability that the two correspond, and putting that into a dynamic-size matrix.
  
  ---
  
  **My question is what in the "all you need" core attention method makes it better than the "align and translate" core attention method?**
  
  Are there ablation studies for this point?
  
  My intuition tells me it would be easier for a network to learn how to correspond vectors, rather than to learn an entire continuous space.
  
  ---
  
  Again, please disregard the other contributions in "all you need", such as self-attention, separation of key from value, normalization, Transformer, ect.
  
  **Tags:** papers, transformer, attention, sequence-modeling
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-04-19. Score: 2 | Accepted answer
  
  The key difference between the attention mechanisms used in "Learning To Align And Translate" and "Attention Is All You Need" lies in the way that the similarity between the query and key vectors is calculated.
  
  ### "Learning To Align And Translate"
  
  The attention score is calculated by a learned function using a feed-forward neural network that takes in the query and key vectors and outputs a (log) probability of correspondence between them. This approach requires the model to learn a mapping from the input and output spaces to a joint space where the vectors can be compared against each other.
  
  ### "Attention Is All You Need"
  
  Here the attention is calculated as the similarity between the query and key vectors by taking their dot product and scaling it by the square root of their dimensionality. This approach does not require the model to learn a mapping to a joint space, but instead relies on the inherent structure of the vector space.
  
  ## Pros/Cons
  
  One advantage of the approach used in "Attention Is All You Need" is that it is computationally more efficient than the method used in "Learning To Align And Translate", especially for long sequences. Specifically, the scaled dot product attention is faster compared to "general/Bahdanau attention" in the sense that the latter is a learnt via a usually shallow feedforward neural network. In that sense, overhead space and time complexity is added while traversing the computational graph of the model as part of training.
  
  That being said, there have been studies that have explored the use of different attention mechanisms in Transformers, including variants of the dot product and learned similarity functions. While the dot product attention used in "Attention Is All You Need" has shown to be effective in many cases, other mechanisms may be more appropriate for certain tasks or data types.
  
  I copy below recent studies relevant to transformer variations and its attention mechanism.
  
  * Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with
    sparse transformers. CoRR, abs/1904.10509, 2019.
  * Nikita Kitaev, L. Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. ArXiv,
    abs/2001.04451, 2020.
  * Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, and T. Lillicrap. Compressive transformers for long-range sequence modelling. ArXiv, abs/1911.05507, 2020.
  * Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention
    with linear complexity. ArXiv, abs/2006.04768, 2020.
  * A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret. Transformers are rnns: Fast autoregressive
    transformers with linear attention. In Proceedings of the International Conference on Machine
    Learning (ICML), 2020.
  * Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer:
    Rethinking self-attention in transformer models, 2020.
  
  > **Comments:**
  > **username_0**: Can you please cite these other studies? I was looking for a more in-depth answer  
  > **username_0**: It is also not clear why "all you need"'s mechanism is faster. One can easily parallelize the "Learning To Align And Translate" method as there are no dependencies. Thus, I am not following why long sequences have any effect. Please notice we are disregarding LSTMs here, just the attention mechanism itself.  
  > **username_1**: The scaled dot product attention is faster compared to "general/Bahdanau attention" in the sense that the latter is a learnt via a usually shallow feedforward neural network. In that sense, overhead space and time complexity is added while traversing the computational graph of the model as part of training. I will  update my post.  
  > **username_0**: Wow so many references! Thank you that is so much appreciated. If you don't mind, how did you search for these?  
  > **username_2**: thank you very much for your input, I was considering this effect. This is very insightful. However I would like to add that dot product attention puts more responsibility on the quality of the embedding than joint attention. Also I haven't yet looked at the reference I think that we could explore deeper network for the joint attention other than a one-layer ffn  
  
  ---
  
  