Answers.Body: <p>Because it's a good fit, pretty much</p>
  <p>Yes, you can use any &quot;inverse distance measure&quot;, however, scalar product paired with softmax is a out-of-the-box good fit, as two vectors the less they agree (direction-wise), the &quot;more negative&quot; the scalar product gets, so <span class="math-container">$e^x$</span> with <span class="math-container">$x$</span> being the scalar product gets to 0, and the more the agree (in the direction), the &quot;more positive&quot; they get, so <span class="math-container">$e^x$</span> gets bigger and bigger</p>
  <p>Bu sure, you can define other attention metrics, just pay attention to make them well behaved (so, no <span class="math-container">$\frac{1}{k^Tq}$</span>)</p>
  <p>Edit:<br />
  If you still want to use softmax to calculate attentions at the end, the logits has to be inversely proportional to the distance, because you want &quot;high positive&quot; values if they agree, and &quot;low&quot; values if they disagree</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-13T11:51:27.537Z
Answers.LastActivityDate: 2023-09-13T11:51:27.537Z
Answers.UserDisplayName: Alberto
Answers.Body: <p>Dot-product attention and Euclidean distance-based attention are not necessarily similar, differing only in terms of computational cost. I wrote a recent paper to report that using the Euclidean distance can produce substantially different parameters and different behavior. This is due to the fact that, &quot;while there exist order-preserving transformations between the Euclidean distance and the inner product, these transformations are not trivial; for each direction, the transformation involves adding one additional dimension [Bachrach et al., 2014].&quot; Notably, &quot;because Euclidean distance measures dissimilarity, it needs to be massaged into use as a similarity score.&quot; And I presented results which suggest that the negative-log of Euclidean distance is more promising than the negative Euclidean distance, and that the negative-log interestingly recovers &quot;the inverse distance weighting function employed by Shepard [1968] for numerical interpolation of irregularly-spaced points.&quot;</p>
  <p>C McCarter, “<a href="https://arxiv.org/abs/2310.18805" rel="nofollow noreferrer">Inverse distance weighting attention</a>.” Associative Memory &amp; Hopfield Networks Workshop @ NeurIPS, 2023.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-05T22:24:13.797Z
Answers.LastActivityDate: 2023-11-05T22:24:13.797Z
Answers.UserDisplayName: calmcc
Body: <p>In models using attention (eg Transformer architectures) we used scaled dot-product to measure similarity rather than (negative or inverse) Euclidean distance. Why is this the case?</p>
  <p>Does Layer Normalization make these basically identical anyways? Or are there statistical, geometric, or training dynamics problems caused by treating embeddings as locations rather than directions? Or is this choice due to the extra computational overhead of using Euclidean distance?</p>
  
Tags: <transformer><deep-neural-networks><attention>
Title: Why is dot-product and not Euclidean distance used for attention?
UserDisplayName: Betterthan Kwora
ThreadText: # Why is dot-product and not Euclidean distance used for attention?
  
  (Asked by: username_0 on 2023-09-13)
  
  In models using attention (eg Transformer architectures) we used scaled dot-product to measure similarity rather than (negative or inverse) Euclidean distance. Why is this the case?
  
  Does Layer Normalization make these basically identical anyways? Or are there statistical, geometric, or training dynamics problems caused by treating embeddings as locations rather than directions? Or is this choice due to the extra computational overhead of using Euclidean distance?
  
  **Tags:** transformer, deep-neural-networks, attention
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-09-13. Score: 3
  
  Because it's a good fit, pretty much
  
  Yes, you can use any "inverse distance measure", however, scalar product paired with softmax is a out-of-the-box good fit, as two vectors the less they agree (direction-wise), the "more negative" the scalar product gets, so $e^x$ with $x$ being the scalar product gets to 0, and the more the agree (in the direction), the "more positive" they get, so $e^x$ gets bigger and bigger
  
  Bu sure, you can define other attention metrics, just pay attention to make them well behaved (so, no $\frac{1}{k^Tq}$)
  
  Edit:  
  If you still want to use softmax to calculate attentions at the end, the logits has to be inversely proportional to the distance, because you want "high positive" values if they agree, and "low" values if they disagree
  
  
  ---
  
  ### Answer by username_2 on 2023-11-05. Score: 2
  
  Dot-product attention and Euclidean distance-based attention are not necessarily similar, differing only in terms of computational cost. I wrote a recent paper to report that using the Euclidean distance can produce substantially different parameters and different behavior. This is due to the fact that, "while there exist order-preserving transformations between the Euclidean distance and the inner product, these transformations are not trivial; for each direction, the transformation involves adding one additional dimension [Bachrach et al., 2014]." Notably, "because Euclidean distance measures dissimilarity, it needs to be massaged into use as a similarity score." And I presented results which suggest that the negative-log of Euclidean distance is more promising than the negative Euclidean distance, and that the negative-log interestingly recovers "the inverse distance weighting function employed by Shepard [1968] for numerical interpolation of irregularly-spaced points."
  
  C McCarter, “[Inverse distance weighting attention](https://arxiv.org/abs/2310.18805).” Associative Memory & Hopfield Networks Workshop @ NeurIPS, 2023.
  
  
  ---
  
  