Answers.Body: <p>The answer by @pi-tau is upto mark. Here is a TL-DR version of it.</p>
  <p>Assuming the keys and queries have mean of <span class="math-container">$0$</span> and standard deviation of <span class="math-container">$1$</span>, the variance of the attention score (the sum of the dot products) is <span class="math-container">$d$</span>, and the standard deviation is <span class="math-container">$\sqrt{d}$</span>. To ensure stable gradient flow, you want to normalize your attention scores, i.e., make them have mean of <span class="math-container">$0$</span> and standard deviation of <span class="math-container">$1$</span>. Since we already assumed that the keys and queries have <span class="math-container">$0$</span> mean and unit standard deviation, the only thing you need to do to achieve this is to scale the attention scores with <span class="math-container">$\sqrt{d}$</span>.</p>
  <p>This scaling helps to maintain stable gradients and prevents the softmax function from saturating and becoming unresponsive to small changes in its input. Hence, it's a crucial part of the self-attention mechanism in Transformer models.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-27T05:25:21.163Z
Answers.LastActivityDate: 2023-12-27T05:25:21.163Z
Answers.UserDisplayName: randomuser94786
Answers.Body: <p>From the <a href="https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf" rel="nofollow noreferrer">attention is all you need</a> paper (section 3.2.1):</p>
  <blockquote>
  <p>Dot-product attention is identical to our algorithm, except for the scaling factor of <span class="math-container">$\frac{1}{\sqrt{d_k}}$</span>. [...] While for small values of <span class="math-container">$d_k$</span> the two mechanisms perform similarly, additive attention outperforms dot product attention without scaling [...]</p>
  </blockquote>
  <p>Then</p>
  <blockquote>
  <p>We suspect that for large values of <span class="math-container">$d_k$</span> the dot product grow large in magnitude, pushing the softmax function into region where it has <em>extremely small gradients</em>. To counteract this effect, we scale the dot products by <span class="math-container">$\frac{1}{\sqrt{d_k}}$</span>.</p>
  </blockquote>
  <p>As you can read the scaling is to fight vanishing gradients, in fact you can interpret it as a sort of <em>temperature</em> that smooths the logits <span class="math-container">$QK^\top$</span> making them more uniformly distributed (since a peaked distribution would cause gradient instabilities), therefore increasing the gradient magnitude.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-24T16:34:17.460Z
Answers.Text: The motivation of this particular expression and not another one (like the product of the norm $\frac{1}{|Q| |K|}$ or even $1/d$) is not very well explained
Answers.UserDisplayName: Sheed
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-24T14:02:30.777Z
Answers.LastActivityDate: 2023-08-24T14:02:30.777Z
Answers.UserDisplayName: Luca Anzalone
Answers.Body: <p>In the <a href="https://arxiv.org/abs/1706.03762" rel="noreferrer">official paper</a> the justification for this scaling is given in footnote <span class="math-container">$[4]$</span> on page 4.</p>
  <p>Here is a more lengthy explanation of the same thing:<br />
  <strong>Let’s assume that</strong> the keys <span class="math-container">$K$</span> and queries <span class="math-container">$Q$</span> have zero mean and unit std. The attention score between some query <span class="math-container">$i$</span> and some key <span class="math-container">$j$</span> is:
  <span class="math-container">$$ \alpha = q_i k_j^T = \sum_{n=1}^{d_k} q_{in}k_{jn} $$</span>
  We know that <a href="https://en.wikipedia.org/wiki/Bienaym%C3%A9%27s_identity" rel="noreferrer">the variance of a sum of random variables</a> equals the sum of the variances of those variables. Thus, for the variance of the attention score we get:
  <span class="math-container">$$ \text{Var}(\alpha) = d_k, $$</span>
  since <span class="math-container">$ \text{Var}(q_{in}k_{jn}) = 1 \quad \forall n.$</span></p>
  <p>The standard deviation is:
  <span class="math-container">$$ \text{std}(\alpha) = \sqrt{d_k} $$</span></p>
  <p>After you compute the attention scores you need to apply softmax in order to obtain the attention probabilities. The problem is that the softmax function saturates very quickly. This means that if one of the attention scores is slightly bigger than the others, then the softmax function will put almost all of the weight on that element. See for example this:</p>
  <pre><code>x = torch.FloatTensor([1, 1, 1, 5])
  y = torch.softmax(x, dim=-1)
  # y: [0.0174, 0.0174, 0.0174, 0.9479]
  </code></pre>
  <p>Since the variance of the attention scores is <span class="math-container">$d_k$</span> then almost certainly one <strong>random</strong> score will be much higher than the others. The softmax will output attention probability of <span class="math-container">$1.$</span> for that element and zeros for the others and you will get no gradient propagation.</p>
  <p>What you want to do to ensure stable gradient flow is normalize your attention scores. That is, make them with zero mean and unit std. We already assumed that the keys and queries have zero mean and unit std, so the only thing you need to do in order to achieve this is to scale the attention scores with <span class="math-container">$\sqrt{d_k}$</span>.</p>
  <p><strong>But is it safe to assume that our keys and queries have unit variance?</strong><br />
  Well, yes! The architecture of the Transformer includes LayerNorm layers that are used exactly for this reason: normalizing the inputs before applying the self-attention layer. In addition the weights of the self-attention layer are initialized using <strong>Xavier initialization</strong> and so the computed keys, queries and values will also have unit std. In my opinion this scaled dot-product works so well only when combined with LayerNorm and Xavier initialization. I have written a very detailed blog post about the Transformer architecture, if you want to read it here is a link:<br />
  <a href="https://pi-tau.github.io/posts/transformer" rel="noreferrer">https://pi-tau.github.io/posts/transformer</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-24T22:28:33.310Z
Answers.Text: "you will get no gradient propagation." I think this is not entirely correct. You will get no backprop only for the element with $p \to 0$.
Answers.UserDisplayName: Antonios Sarikas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-26T07:38:40.887Z
Answers.Text: @adosar If the std is too large, then the softmax will output `p=1` for **one element** and `p=0` for all the rest.
Answers.UserDisplayName: pi-tau
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-26T11:07:34.743Z
Answers.Text: But where these 1's and 0's appear during backprop? If we can't answer this mathematically, we can't state there will be no gradient during backprop.
Answers.UserDisplayName: Antonios Sarikas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-01T08:28:02.087Z
Answers.Text: During the **forward pass** (not the backward) we will get a score of `1` for **one random entry** and **all the others** will be `0`. This happens because the standard deviation of the computed attention scores is quite large ($sqrt(d_k)$). This is the mathematical explanation. The simple explanation is that the attention scores will vary by a lot. Now, since the softmax func involves an exponentiation operation, then calculating the attention prob for the largest number will be close to `1.` and the other probs will be close to `0`. This is just fact you can see the example with [1, 1, 1, 5]
Answers.UserDisplayName: pi-tau
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-01T08:29:33.447Z
Answers.Text: Now, since the forward pass for the self-attention layer produces `0`s everywhere and `1` at a random position, then when you compute the derivative you will get a gradient of zero. Thus, no gradient flow.
Answers.UserDisplayName: pi-tau
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-07T08:50:09.530Z
Answers.Text: Thanks for the elaboration. With regards to the gradient flow: the derivative will be zero since the derivative of softmax is 0 when the output is 1 or 0 (i.e. we are in the saturating regime)?
Answers.UserDisplayName: Antonios Sarikas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-08T10:22:15.707Z
Answers.Text: Yes, exactly. You can see this from the graph of the softmax function. It is almost horizontal for inputs that produce output of 0 or 1. This means that the slope of the graph (i.e. the derivative) is 0.
Answers.UserDisplayName: pi-tau
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-25T12:11:43.127Z
Answers.LastActivityDate: 2023-09-25T12:11:43.127Z
Answers.UserDisplayName: pi-tau
Body: <p>In attention settings, typically when the both <em>Query Q</em> and <em>Key K</em> are of the same dimension <em>d</em> we can compute their attention score in the following manner:</p>
  <p><span class="math-container">$$\frac{Q^T K}{\sqrt{d}}$$</span></p>
  <p>The justification being that the dot product tends to blow up in higher dimension, and <span class="math-container">$\sqrt{d}$</span> helps to &quot;keep things reasonable&quot;. I also read that it serves to reduce the standard deviation to 1, however chatGPT contradicts that statement, and I can't find resources that support this claim.</p>
  <p>Therefore what is the mathematical justification behind <span class="math-container">$\sqrt{d}$</span> and not another form such as <span class="math-container">$\log d$</span>, <span class="math-container">$d^5$</span>, <span class="math-container">$e^{\pi d}$</span> or simply <span class="math-container">$|Q||V|$</span> (a cosine similarity!) if we want to keep things &quot;bounded&quot;?</p>
  
Tags: <attention>
Title: Why use a "square root" in the scaled dot product
UserDisplayName: Sheed
ThreadText: # Why use a "square root" in the scaled dot product
  
  (Asked by: username_0 on 2023-08-24)
  
  In attention settings, typically when the both *Query Q* and *Key K* are of the same dimension *d* we can compute their attention score in the following manner:
  
  $$\frac{Q^T K}{\sqrt{d}}$$
  
  The justification being that the dot product tends to blow up in higher dimension, and $\sqrt{d}$ helps to "keep things reasonable". I also read that it serves to reduce the standard deviation to 1, however chatGPT contradicts that statement, and I can't find resources that support this claim.
  
  Therefore what is the mathematical justification behind $\sqrt{d}$ and not another form such as $\log d$, $d^5$, $e^{\pi d}$ or simply $|Q||V|$ (a cosine similarity!) if we want to keep things "bounded"?
  
  **Tags:** attention
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2023-09-25. Score: 9 | Accepted answer
  
  In the official paper the justification for this scaling is given in footnote $[4]$ on page 4.
  Here is a more lengthy explanation of the same thing:
  Let’s assume that the keys $K$ and queries $Q$ have zero mean and unit std. The attention score between some query $i$ and some key $j$ is:
  $$ \alpha = q_i k_j^T = \sum_{n=1}^{d_k} q_{in}k_{jn} $$
  We know that the variance of a sum of random variables equals the sum of the variances of those variables. Thus, for the variance of the attention score we get:
  $$ \text{Var}(\alpha) = d_k, $$
  since $ \text{Var}(q_{in}k_{jn}) = 1 \quad \forall n.$
  The standard deviation is:
  $$ \text{std}(\alpha) = \sqrt{d_k} $$
  After you compute the attention scores you need to apply softmax in order to obtain the attention probabilities. The problem is that the softmax function saturates very quickly. This means that if one of the attention scores is slightly bigger than the others, then the softmax function will put almost all of the weight on that element. See for example this:
  x = torch.FloatTensor([1, 1, 1, 5])
  y = torch.softmax(x, dim=-1)
  # y: [0.0174, 0.0174, 0.0174, 0.9479]
  
  Since the variance of the attention scores is $d_k$ then almost certainly one random score will be much higher than the others. The softmax will output attention probability of $1.$ for that element and zeros for the others and you will get no gradient propagation.
  What you want to do to ensure stable gradient flow is normalize your attention scores. That is, make them with zero mean and unit std. We already assumed that the keys and queries have zero mean and unit std, so the only thing you need to do in order to achieve this is to scale the attention scores with $\sqrt{d_k}$.
  But is it safe to assume that our keys and queries have unit variance?
  Well, yes! The architecture of the Transformer includes LayerNorm layers that are used exactly for this reason: normalizing the inputs before applying the self-attention layer. In addition the weights of the self-attention layer are initialized using Xavier initialization and so the computed keys, queries and values will also have unit std. In my opinion this scaled dot-product works so well only when combined with LayerNorm and Xavier initialization. I have written a very detailed blog post about the Transformer architecture, if you want to read it here is a link:
  https://pi-tau.github.io/posts/transformer
  
  > **Comments:**
  > **username_4**: "you will get no gradient propagation." I think this is not entirely correct. You will get no backprop only for the element with $p \to 0$.  
  > **username_3**: @adosar If the std is too large, then the softmax will output `p=1` for **one element** and `p=0` for all the rest.  
  > **username_4**: But where these 1's and 0's appear during backprop? If we can't answer this mathematically, we can't state there will be no gradient during backprop.  
  > **username_3**: During the **forward pass** (not the backward) we will get a score of `1` for **one random entry** and **all the others** will be `0`. This happens because the standard deviation of the computed attention scores is quite large ($sqrt(d_k)$). This is the mathematical explanation. The simple explanation is that the attention scores will vary by a lot. Now, since the softmax func involves an exponentiation operation, then calculating the attention prob for the largest number will be close to `1.` and the other probs will be close to `0`. This is just fact you can see the example with [1, 1, 1, 5]  
  > **username_3**: Now, since the forward pass for the self-attention layer produces `0`s everywhere and `1` at a random position, then when you compute the derivative you will get a gradient of zero. Thus, no gradient flow.  
  > **username_4**: Thanks for the elaboration. With regards to the gradient flow: the derivative will be zero since the derivative of softmax is 0 when the output is 1 or 0 (i.e. we are in the saturating regime)?  
  > **username_3**: Yes, exactly. You can see this from the graph of the softmax function. It is almost horizontal for inputs that produce output of 0 or 1. This means that the slope of the graph (i.e. the derivative) is 0.  
  
  ---
  
  ### Answer by username_1 on 2023-12-27. Score: 3
  
  The answer by @pi-tau is upto mark. Here is a TL-DR version of it.
  
  Assuming the keys and queries have mean of $0$ and standard deviation of $1$, the variance of the attention score (the sum of the dot products) is $d$, and the standard deviation is $\sqrt{d}$. To ensure stable gradient flow, you want to normalize your attention scores, i.e., make them have mean of $0$ and standard deviation of $1$. Since we already assumed that the keys and queries have $0$ mean and unit standard deviation, the only thing you need to do to achieve this is to scale the attention scores with $\sqrt{d}$.
  
  This scaling helps to maintain stable gradients and prevents the softmax function from saturating and becoming unresponsive to small changes in its input. Hence, it's a crucial part of the self-attention mechanism in Transformer models.
  
  
  ---
  
  ### Answer by username_2 on 2023-08-24. Score: 0
  
  From the [attention is all you need](https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf) paper (section 3.2.1):
  
  > Dot-product attention is identical to our algorithm, except for the scaling factor of $\frac{1}{\sqrt{d\_k}}$. [...] While for small values of $d\_k$ the two mechanisms perform similarly, additive attention outperforms dot product attention without scaling [...]
  
  Then
  
  > We suspect that for large values of $d\_k$ the dot product grow large in magnitude, pushing the softmax function into region where it has *extremely small gradients*. To counteract this effect, we scale the dot products by $\frac{1}{\sqrt{d\_k}}$.
  
  As you can read the scaling is to fight vanishing gradients, in fact you can interpret it as a sort of *temperature* that smooths the logits $QK^\top$ making them more uniformly distributed (since a peaked distribution would cause gradient instabilities), therefore increasing the gradient magnitude.
  
  > **Comments:**
  > **username_0**: The motivation of this particular expression and not another one (like the product of the norm $\frac{1}{|Q| |K|}$ or even $1/d$) is not very well explained  
  
  ---
  
  