Answers.Body: <p>Assume that the query embeddings <span class="math-container">$Q$</span> and key embeddings <span class="math-container">$K$</span> have zero mean and unit std. Then for the variance of the attention score between any query and key we get:</p>
  <p><span class="math-container">$$ \alpha = q_i k_j^T = \sum_{n=1}^{d_k} q_{in} k_{jn} $$</span>
  <span class="math-container">$$ \text{Var}(\alpha) = d_k $$</span>
  <span class="math-container">$$ \text{std}(\alpha) = \sqrt{d_k} $$</span></p>
  <p>But is it safe to assume that our query embeddings and key embeddings have unit variance?</p>
  <p>The answer is yes. The embeddings are computed by multiplying the input with the query and key matrices (<span class="math-container">$W_Q$</span> and <span class="math-container">$W_K$</span>). We can assume that the input already has unit variance by using a normalizing layer (e.g. LayerNorm), and the weights of the <span class="math-container">$W_Q$</span> and <span class="math-container">$W_K$</span> matrices should be initialized so that variance is preserved.</p>
  <p><em><strong>But why do we want unit variance at all?</strong></em></p>
  <p>Applying softmax on the attention scores with such high variance will result in all of the weight being placed on one random element, while all the other elements will have a weight of zero. Thus, in order to have the attention scores with unit std, we scale by <span class="math-container">$\sqrt{d_k}$</span>.</p>
  <p>For more checkout <a href="https://pi-tau.github.io/posts/transformer/#multi-head-attention-layer" rel="nofollow noreferrer">this</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-29T20:54:56.273Z
Answers.Text: Very good answer! For anyone wondering, we can approximate unit variance for out weights matrix using Xavier-Glorot:
Answers.UserDisplayName: Dude156
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-29T14:58:47.697Z
Answers.LastActivityDate: 2023-07-29T14:58:47.697Z
Answers.UserDisplayName: pi-tau
Answers.Body: <p>The mathematical explanation is already explained in the another answer. But I feel bothered why OP's code get an output of 1.6 variance, hence I run my own experiment to prove it:</p>
  <pre><code>import numpy as np
  a = np.random.randn(1111,111)
  print('variance of matrix A =', a.var())
  b = np.random.randn(111,1111)
  print('variance of matrix B =', b.var())
  c = a @ b
  print('variance of dot-product =', c.var())
  d = c / np.sqrt(111)
  print('variance of scaled (normalized) dot-product =', d.var())
  </code></pre>
  <p>The output:</p>
  <pre><code>variance of matrix A = 0.9998308190814476
  variance of matrix B = 0.9957289239999045
  variance of dot-product = 110.5603610976103
  variance of scaled (normalized) dot-product = 0.9960392891676604
  </code></pre>
  <p>As shown above, the variance is becoming (close to) 1 after dividing by <span class="math-container">$\sqrt({d_k})$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-07T10:31:15.847Z
Answers.LastActivityDate: 2023-04-07T10:31:15.847Z
Answers.UserDisplayName: Thariq Nugrohotomo
Answers.Body: <p>It might help to take two small matrices that match the assumptions (mean of zero and variance of one) and just do the matrix multiplication.  The dimensionality of K scales Q in the multiplication, scaling the variance simultaneously.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-21T01:09:56.240Z
Answers.Text: Hey, I did do this but it does not come out as expected. I have edited the post
Answers.UserDisplayName: Jacob B
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-18T10:03:21.550Z
Answers.LastActivityDate: 2020-05-18T10:03:21.550Z
Answers.UserDisplayName: David Hoelzer
Answers.Body: <p>In statistics, if <span class="math-container">$X$</span> and <span class="math-container">$Y$</span> are independent and randomly distributed variables:</p>
  <p><span class="math-container">$\mathbb{E}[X + Y] = \mathbb{E}[X] + \mathbb{E}[Y] \\
  Var(X + Y) = Var(X) + Var(Y) \\
  \mathbb{E}[XY] = \mathbb{E}[X]\mathbb{E}[Y] \\
  Var(XY) = (Var(X) + \mathbb{E}[X]^2)(Var(Y) + \mathbb{E}[Y]^2) - \mathbb{E}[X]^2\mathbb{E}[Y]^2$</span></p>
  <p>Let <span class="math-container">$Q$</span> and <span class="math-container">$K$</span> be random <span class="math-container">$d_k$</span> x <span class="math-container">$d_k$</span> matrices, where each entry is a some random distribution with <span class="math-container">$0$</span> mean and <span class="math-container">$1$</span> variance. Every entry is independent from each other.</p>
  <p>Since each entry of <span class="math-container">$Q$</span> and <span class="math-container">$K$</span> have identical distribution, we can focus only on the top-left-most element of <span class="math-container">$QK$</span> without loss of generality. The same applies to every other element.</p>
  <p>The top-left-most element of <span class="math-container">$QK$</span> is <span class="math-container">$\displaystyle \sum_{i=0}^{d_k} Q_{1,i} K_{i, 1}$</span>.</p>
  <p>Since <span class="math-container">$Q$</span> and <span class="math-container">$K$</span> are independent:</p>
  <p><span class="math-container">$\mathbb{E}[Q_{1, i} K_{i, 1}] = \mathbb{E}[Q_{1, i}] \mathbb{E}[K_{i, 1}] = 0 \\
  Var(Q_{1, i} K_{i, 1}) = (Var(Q_{1, i}) + \mathbb{E}[Q_{1, i}]^2)(Var(K_{i, 1}) + \mathbb{E}[K_{i, 1}]^2) - \mathbb{E}[Q_{1, i}]^2\mathbb{E}[K_{i, 1}]^2 = 1$</span></p>
  <p>And so summing up <span class="math-container">$d_k$</span> of them:</p>
  <p><span class="math-container">$\mathbb{E} \left[\displaystyle \sum_{i=0}^{d_k} Q_{1,i} K_{i, 1} \right] = \displaystyle \sum_{i=0}^{d_k} \mathbb{E} \left[ Q_{1,i} K_{i, 1}  \right] = 0 \\
  Var\left(\displaystyle \sum_{i=0}^{d_k} Q_{1,i} K_{i, 1} \right) = \displaystyle \sum_{i=0}^{d_k} Var\left( Q_{1,i} K_{i, 1}  \right) = d_k$</span></p>
  <p>For your code block, you are computing the dot product of matrices <span class="math-container">$a$</span> and <span class="math-container">$b$</span>, when you should be doing a matrix multiplication (the attention function multiplies <span class="math-container">$Q$</span> by <span class="math-container">$K$</span> after all, which is the vectorized form of dot-product -- it doesn't actually do dot product). It should work out to unit variance.</p>
  <p><em>edit: the last paragraph is incorrect, as dot is the same as matrix multiply in the above case</em></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-08T20:40:53.053Z
Answers.Text: Awesome, thank you this is a great answer. I am still confused on the code part though looking at the docs for numpy the np.dot function should be equivalent to np.matmul in 2 dimensions.
Answers.UserDisplayName: Jacob B
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-08T20:53:27.697Z
Answers.Text: Ah, you are right! dot is the same as matmul. I'm not sure how to explain the code part in that case. Maybe dimension=2 is too small? The sample variance will itself have a variance that reduces with larger dimension size.
Answers.UserDisplayName: user3667125
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-08T22:23:29.720Z
Answers.Text: Hmmm I see I will try with larger shapes, in any case your answer provided great insight to my initial question so thanks for taking the time!
Answers.UserDisplayName: Jacob B
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-29T21:34:48.007Z
Answers.Text: I appreciate it if you answer this related question https://ai.stackexchange.com/questions/40244/in-the-attention-mechanism-why-dont-we-normalize-after-multiplying-values
Answers.UserDisplayName: Peyman
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-08T02:07:02.287Z
Answers.LastActivityDate: 2020-12-08T20:51:10.567Z
Answers.LastEditDate: 2020-12-08T20:51:10.567Z
Answers.UserDisplayName: user3667125
Body: <p>In scaled dot product attention, we scale our outputs by dividing the dot product by the square root of the dimensionality of the matrix:</p>
  <p><a href="https://i.sstatic.net/wLI4m.png" rel="noreferrer"><img src="https://i.sstatic.net/wLI4m.png" alt="enter image description here" /></a></p>
  <p>The reason why is stated that this constrains the distribution of the weights of the output to have a standard deviation of 1.</p>
  <p>Quoted from <a href="https://www.tensorflow.org/tutorials/text/transformer" rel="noreferrer">Transformer model for language understanding | TensorFlow</a>:</p>
  <blockquote>
  <p>For example, consider that <span class="math-container">$Q$</span> and <span class="math-container">$K$</span> have a mean of 0 and variance of 1. Their matrix multiplication will have a mean of 0 and variance of <span class="math-container">$d_k$</span>. Hence, square root of <span class="math-container">$d_k$</span> is used for scaling (and not any other number) because the matmul of <span class="math-container">$Q$</span> and <span class="math-container">$K$</span> should have a mean of 0 and variance of 1, and you get a gentler softmax.</p>
  </blockquote>
  <p>Why does this multiplication have a variance of <span class="math-container">$d_k$</span>?</p>
  <p>If I understand this, I will then understand why dividing by <span class="math-container">$\sqrt({d_k})$</span> would normalize to 1.</p>
  <p>Trying this experiment on 2x2 arrays I get an output of 1.6 variance:</p>
  <p><a href="https://i.sstatic.net/GCe3t.png" rel="noreferrer"><img src="https://i.sstatic.net/GCe3t.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-16T23:20:20.040Z
Comments.Text: Even though here programming questions are generally off-topic, in case the code is necessary, it's preferable to provide it rather than an actual screenshot. So, please, if you can still provide the actual code, edit your post to include it.
Comments.UserDisplayName: nbro
Tags: <neural-networks><machine-learning><natural-language-processing><transformer><attention>
Title: Why does this multiplication of $Q$ and $K$ have a variance of $d_k$, in scaled dot product attention?
UserDisplayName: Jacob B
ThreadText: # Why does this multiplication of $Q$ and $K$ have a variance of $d_k$, in scaled dot product attention?
  
  (Asked by: username_0 on 2020-05-18)
  
  In scaled dot product attention, we scale our outputs by dividing the dot product by the square root of the dimensionality of the matrix:
  
  [![enter image description here](https://i.sstatic.net/wLI4m.png)](https://i.sstatic.net/wLI4m.png)
  
  The reason why is stated that this constrains the distribution of the weights of the output to have a standard deviation of 1.
  
  Quoted from [Transformer model for language understanding | TensorFlow](https://www.tensorflow.org/tutorials/text/transformer):
  
  > For example, consider that $Q$ and $K$ have a mean of 0 and variance of 1. Their matrix multiplication will have a mean of 0 and variance of $d\_k$. Hence, square root of $d\_k$ is used for scaling (and not any other number) because the matmul of $Q$ and $K$ should have a mean of 0 and variance of 1, and you get a gentler softmax.
  
  Why does this multiplication have a variance of $d\_k$?
  
  If I understand this, I will then understand why dividing by $\sqrt({d\_k})$ would normalize to 1.
  
  Trying this experiment on 2x2 arrays I get an output of 1.6 variance:
  
  [![enter image description here](https://i.sstatic.net/GCe3t.png)](https://i.sstatic.net/GCe3t.png)
  
  **Tags:** neural-networks, machine-learning, natural-language-processing, transformer, attention
  > **Comments:**
  > **username_1**: Even though here programming questions are generally off-topic, in case the code is necessary, it's preferable to provide it rather than an actual screenshot. So, please, if you can still provide the actual code, edit your post to include it.  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_6 on 2020-12-08. Score: 12 | Accepted answer
  
  In statistics, if $X$ and $Y$ are independent and randomly distributed variables:
  
  $\mathbb{E}[X + Y] = \mathbb{E}[X] + \mathbb{E}[Y] \\
  Var(X + Y) = Var(X) + Var(Y) \\
  \mathbb{E}[XY] = \mathbb{E}[X]\mathbb{E}[Y] \\
  Var(XY) = (Var(X) + \mathbb{E}[X]^2)(Var(Y) + \mathbb{E}[Y]^2) - \mathbb{E}[X]^2\mathbb{E}[Y]^2$
  
  Let $Q$ and $K$ be random $d\_k$ x $d\_k$ matrices, where each entry is a some random distribution with $0$ mean and $1$ variance. Every entry is independent from each other.
  
  Since each entry of $Q$ and $K$ have identical distribution, we can focus only on the top-left-most element of $QK$ without loss of generality. The same applies to every other element.
  
  The top-left-most element of $QK$ is $\displaystyle \sum\_{i=0}^{d\_k} Q\_{1,i} K\_{i, 1}$.
  
  Since $Q$ and $K$ are independent:
  
  $\mathbb{E}[Q\_{1, i} K\_{i, 1}] = \mathbb{E}[Q\_{1, i}] \mathbb{E}[K\_{i, 1}] = 0 \\
  Var(Q\_{1, i} K\_{i, 1}) = (Var(Q\_{1, i}) + \mathbb{E}[Q\_{1, i}]^2)(Var(K\_{i, 1}) + \mathbb{E}[K\_{i, 1}]^2) - \mathbb{E}[Q\_{1, i}]^2\mathbb{E}[K\_{i, 1}]^2 = 1$
  
  And so summing up $d\_k$ of them:
  
  $\mathbb{E} \left[\displaystyle \sum\_{i=0}^{d\_k} Q\_{1,i} K\_{i, 1} \right] = \displaystyle \sum\_{i=0}^{d\_k} \mathbb{E} \left[ Q\_{1,i} K\_{i, 1} \right] = 0 \\
  Var\left(\displaystyle \sum\_{i=0}^{d\_k} Q\_{1,i} K\_{i, 1} \right) = \displaystyle \sum\_{i=0}^{d\_k} Var\left( Q\_{1,i} K\_{i, 1} \right) = d\_k$
  
  For your code block, you are computing the dot product of matrices $a$ and $b$, when you should be doing a matrix multiplication (the attention function multiplies $Q$ by $K$ after all, which is the vectorized form of dot-product -- it doesn't actually do dot product). It should work out to unit variance.
  
  *edit: the last paragraph is incorrect, as dot is the same as matrix multiply in the above case*
  
  > **Comments:**
  > **username_0**: Awesome, thank you this is a great answer. I am still confused on the code part though looking at the docs for numpy the np.dot function should be equivalent to np.matmul in 2 dimensions.  
  > **username_6**: Ah, you are right! dot is the same as matmul. I'm not sure how to explain the code part in that case. Maybe dimension=2 is too small? The sample variance will itself have a variance that reduces with larger dimension size.  
  > **username_0**: Hmmm I see I will try with larger shapes, in any case your answer provided great insight to my initial question so thanks for taking the time!  
  > **username_7**: I appreciate it if you answer this related question https://ai.stackexchange.com/questions/40244/in-the-attention-mechanism-why-dont-we-normalize-after-multiplying-values  
  
  ---
  
  ### Answer by username_2 on 2023-07-29. Score: 4
  
  Assume that the query embeddings $Q$ and key embeddings $K$ have zero mean and unit std. Then for the variance of the attention score between any query and key we get:
  
  $$ \alpha = q\_i k\_j^T = \sum\_{n=1}^{d\_k} q\_{in} k\_{jn} $$
  $$ \text{Var}(\alpha) = d\_k $$
  $$ \text{std}(\alpha) = \sqrt{d\_k} $$
  
  But is it safe to assume that our query embeddings and key embeddings have unit variance?
  
  The answer is yes. The embeddings are computed by multiplying the input with the query and key matrices ($W\_Q$ and $W\_K$). We can assume that the input already has unit variance by using a normalizing layer (e.g. LayerNorm), and the weights of the $W\_Q$ and $W\_K$ matrices should be initialized so that variance is preserved.
  
  ***But why do we want unit variance at all?***
  
  Applying softmax on the attention scores with such high variance will result in all of the weight being placed on one random element, while all the other elements will have a weight of zero. Thus, in order to have the attention scores with unit std, we scale by $\sqrt{d\_k}$.
  
  For more checkout [this](https://pi-tau.github.io/posts/transformer/#multi-head-attention-layer).
  
  > **Comments:**
  > **username_3**: Very good answer! For anyone wondering, we can approximate unit variance for out weights matrix using Xavier-Glorot:  
  
  ---
  
  ### Answer by username_4 on 2023-04-07. Score: 1
  
  The mathematical explanation is already explained in the another answer. But I feel bothered why OP's code get an output of 1.6 variance, hence I run my own experiment to prove it:
  import numpy as np
  a = np.random.randn(1111,111)
  print('variance of matrix A =', a.var())
  b = np.random.randn(111,1111)
  print('variance of matrix B =', b.var())
  c = a @ b
  print('variance of dot-product =', c.var())
  d = c / np.sqrt(111)
  print('variance of scaled (normalized) dot-product =', d.var())
  
  The output:
  variance of matrix A = 0.9998308190814476
  variance of matrix B = 0.9957289239999045
  variance of dot-product = 110.5603610976103
  variance of scaled (normalized) dot-product = 0.9960392891676604
  
  As shown above, the variance is becoming (close to) 1 after dividing by $\sqrt({d_k})$.
  
  
  ---
  
  ### Answer by username_5 on 2020-05-18. Score: 0
  
  It might help to take two small matrices that match the assumptions (mean of zero and variance of one) and just do the matrix multiplication. The dimensionality of K scales Q in the multiplication, scaling the variance simultaneously.
  
  > **Comments:**
  > **username_0**: Hey, I did do this but it does not come out as expected. I have edited the post  
  
  ---
  
  