Answers.Body: <p>The <a href="https://arxiv.org/abs/1706.03762" rel="nofollow noreferrer">Attention is All you Need</a> has this footnote at the passage motivating the introduction of the <span class="math-container">$1/\sqrt{d_k}$</span> factor:</p>
  <blockquote>
  <ol start="4">
  <li>To illustrate why the dot products get large, assume that the components of <span class="math-container">$q$</span> and <span class="math-container">$k$</span> are independent random variables with mean 0 and variance 1. Then their dot product, <span class="math-container">$q \cdot k = \sum^{d_k}_{i=1}q_ik_i$</span> has mean 0 and variance <span class="math-container">$d_k$</span>.</li>
  </ol>
  </blockquote>
  <p>I suspect that it hints on the cosine-vs-dot difference intuition. The cosine similarity ignores magnitudes of the input vectors - you can scale <span class="math-container">$h^{enc}$</span> and <span class="math-container">$h^{dec}$</span> by arbitrary factors and still get the same value of the cosine distance.</p>
  <p>The footnote talks about vectors with normally distributed components, clearly implying that their magnitudes are important. This suggests that the dot product attention is preferable, since it takes into account magnitudes of input vectors.  And the magnitude might contain some useful information about the &quot;absolute relevance&quot; of the <span class="math-container">$Q$</span> and <span class="math-container">$K$</span> embeddings.</p>
  <p><strong>Edit after more digging:</strong> Note that transformer architecture has the <code>Add &amp; Norm</code> blocks after each
  attention and FF block. At first I thought that it settles your question: since
  every input vector is normalized then cosine distance should be equal to the
  dot product. That's incorrect though - the &quot;Norm&quot; here means <a href="https://arxiv.org/abs/1607.06450" rel="nofollow noreferrer">Layer
  Normalization</a> - analogously to batch normalization it has trainable mean and
  scale parameters, so my point above about the vector norms still holds.</p>
  <p>Finally, since apparently <a href="https://ai.stackexchange.com/q/27260/20538">we don't really know why the BatchNorm works</a>
  same thing holds for the LayerNorm. Interestingly, it seems like (1) BatchNorm
  (2) LayerNorm and (3) your question about normalization in the attention
  mechanism - all of it look like different ways at looking at the same, yet
  undiscovered and clearly stated thing.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-18T21:42:45.567Z
Answers.Text: I think it's a helpful point. I'll leave this open till the bounty ends in case any one else has input. Thanks.
Answers.UserDisplayName: Alexander Soare
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-18T23:25:18.267Z
Answers.Text: @AlexanderSoare Thank you (also for great question). I've spent some more time digging deeper into it - check my edit.
Answers.UserDisplayName: Kostya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-19T09:04:06.633Z
Answers.Text: Thanks for sharing more of your thoughts. I never thought to related it to the LayerNorm as there's a softmax and dot product with $V$ in between so things rapidly get more complicated when trying to look at it from a bottom up perspective. I think my main takeaways from your answer are a) cosine distance doesn't take scale into account, b) they divide by $sqrt(d_k)$ but it could have been something else and might have worked and we don't really know why
Answers.UserDisplayName: Alexander Soare
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-19T09:04:35.643Z
Answers.Text: By the way, re layer norm vs batch norm I also have [this question](https://ai.stackexchange.com/questions/27309/what-are-the-consequences-of-layer-norm-vs-batch-norm) if you're interested in following it
Answers.UserDisplayName: Alexander Soare
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-14T11:57:11.137Z
Answers.Text: https://github.com/neqkir/attention-mechanism
Answers.UserDisplayName: kiriloff
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-17T23:51:24.440Z
Answers.LastActivityDate: 2021-04-19T08:55:49.723Z
Answers.LastEditDate: 2021-04-19T08:55:49.723Z
Answers.UserDisplayName: Kostya
Body: <p>I'm following this <a href="https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html" rel="noreferrer">blog post</a> which enumerates the various types of attention.</p>
  <p>It mentions content-based attention where the alignment scoring function for the <span class="math-container">$j$</span>th encoder hidden state with respect to the <span class="math-container">$i$</span>th context vector is the cosine distance:</p>
  <p><span class="math-container">$$
  e_{ij} = \frac{\mathbf{h}^{enc}_{j}\cdot\mathbf{h}^{dec}_{i}}{||\mathbf{h}^{enc}_{j}||\cdot||\mathbf{h}^{dec}_{i}||}
  $$</span></p>
  <p>It also mentions dot-product attention:</p>
  <p><span class="math-container">$$
  e_{ij} = \mathbf{h}^{enc}_{j}\cdot\mathbf{h}^{dec}_{i}
  $$</span></p>
  <p>To me, it seems like these are only different by a factor. If we fix <span class="math-container">$i$</span> such that we are focusing on only one time step in the decoder, then that factor is only dependent on <span class="math-container">$j$</span>. Specifically, it's <span class="math-container">$1/\mathbf{h}^{enc}_{j}$</span>.</p>
  <p>So we could state: &quot;the only adjustment content-based attention makes to dot-product attention, is that it scales each alignment score inversely with the norm of the corresponding encoder hidden state before softmax is applied.&quot;</p>
  <p>What's the motivation behind making such a minor adjustment? What are the consequences?</p>
  <hr />
  <p>Follow up question:</p>
  <p>What's more, is that in <a href="https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf" rel="noreferrer">Attention is All you Need</a> they introduce the scaled dot product where they divide by a constant factor (square root of size of encoder hidden vector) to avoid vanishing gradients in the softmax. Any reason they don't just use cosine distance?</p>
  
Tags: <neural-networks><attention><seq2seq>
Title: What's the difference between content-based attention and dot-product attention?
UserDisplayName: Alexander Soare
ThreadText: # What's the difference between content-based attention and dot-product attention?
  
  (Asked by: username_0 on 2020-12-30)
  
  I'm following this [blog post](https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html) which enumerates the various types of attention.
  
  It mentions content-based attention where the alignment scoring function for the $j$th encoder hidden state with respect to the $i$th context vector is the cosine distance:
  
  $$
  e\_{ij} = \frac{\mathbf{h}^{enc}\_{j}\cdot\mathbf{h}^{dec}\_{i}}{||\mathbf{h}^{enc}\_{j}||\cdot||\mathbf{h}^{dec}\_{i}||}
  $$
  
  It also mentions dot-product attention:
  
  $$
  e\_{ij} = \mathbf{h}^{enc}\_{j}\cdot\mathbf{h}^{dec}\_{i}
  $$
  
  To me, it seems like these are only different by a factor. If we fix $i$ such that we are focusing on only one time step in the decoder, then that factor is only dependent on $j$. Specifically, it's $1/\mathbf{h}^{enc}\_{j}$.
  
  So we could state: "the only adjustment content-based attention makes to dot-product attention, is that it scales each alignment score inversely with the norm of the corresponding encoder hidden state before softmax is applied."
  
  What's the motivation behind making such a minor adjustment? What are the consequences?
  
  ---
  
  Follow up question:
  
  What's more, is that in [Attention is All you Need](https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf) they introduce the scaled dot product where they divide by a constant factor (square root of size of encoder hidden vector) to avoid vanishing gradients in the softmax. Any reason they don't just use cosine distance?
  
  **Tags:** neural-networks, attention, seq2seq
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-04-17. Score: 5
  
  The [Attention is All you Need](https://arxiv.org/abs/1706.03762) has this footnote at the passage motivating the introduction of the $1/\sqrt{d\_k}$ factor:
  
  > 4. To illustrate why the dot products get large, assume that the components of $q$ and $k$ are independent random variables with mean 0 and variance 1. Then their dot product, $q \cdot k = \sum^{d\_k}\_{i=1}q\_ik\_i$ has mean 0 and variance $d\_k$.
  
  I suspect that it hints on the cosine-vs-dot difference intuition. The cosine similarity ignores magnitudes of the input vectors - you can scale $h^{enc}$ and $h^{dec}$ by arbitrary factors and still get the same value of the cosine distance.
  
  The footnote talks about vectors with normally distributed components, clearly implying that their magnitudes are important. This suggests that the dot product attention is preferable, since it takes into account magnitudes of input vectors. And the magnitude might contain some useful information about the "absolute relevance" of the $Q$ and $K$ embeddings.
  
  **Edit after more digging:** Note that transformer architecture has the `Add & Norm` blocks after each
  attention and FF block. At first I thought that it settles your question: since
  every input vector is normalized then cosine distance should be equal to the
  dot product. That's incorrect though - the "Norm" here means [Layer
  Normalization](https://arxiv.org/abs/1607.06450) - analogously to batch normalization it has trainable mean and
  scale parameters, so my point above about the vector norms still holds.
  
  Finally, since apparently [we don't really know why the BatchNorm works](https://ai.stackexchange.com/q/27260/20538)
  same thing holds for the LayerNorm. Interestingly, it seems like (1) BatchNorm
  (2) LayerNorm and (3) your question about normalization in the attention
  mechanism - all of it look like different ways at looking at the same, yet
  undiscovered and clearly stated thing.
  
  > **Comments:**
  > **username_0**: I think it's a helpful point. I'll leave this open till the bounty ends in case any one else has input. Thanks.  
  > **username_1**: @username_0 Thank you (also for great question). I've spent some more time digging deeper into it - check my edit.  
  > **username_0**: Thanks for sharing more of your thoughts. I never thought to related it to the LayerNorm as there's a softmax and dot product with $V$ in between so things rapidly get more complicated when trying to look at it from a bottom up perspective. I think my main takeaways from your answer are a) cosine distance doesn't take scale into account, b) they divide by $sqrt(d_k)$ but it could have been something else and might have worked and we don't really know why  
  > **username_0**: By the way, re layer norm vs batch norm I also have [this question](https://ai.stackexchange.com/questions/27309/what-are-the-consequences-of-layer-norm-vs-batch-norm) if you're interested in following it  
  > **username_2**: https://github.com/neqkir/attention-mechanism  
  
  ---
  
  