Answers.Body: <p>You first compute the <em>query embeddings</em> <span class="math-container">$Q$</span> and the <em>key embeddings</em> <span class="math-container">$K$</span> by multiplying your input sequence <span class="math-container">$x$</span> with the query matrix <span class="math-container">$W_Q$</span> and the key matrix <span class="math-container">$W_K$</span> respectively. Then you calculate the attention scores and scale them by dividing by the dimensionality of the key embedding space.</p>
  <p><span class="math-container">$$ A = \frac{xW_QW_K^Tx^T}{\sqrt{d_k}} = \frac{QK^T}{\sqrt{d_k}} $$</span></p>
  <p>After that you apply softmax to get the attention weights (probabilities). But to see why we scale by <span class="math-container">$\frac{1}{\sqrt{d_k}}$</span> let’s assume that the query embeddings and key embeddings have zero mean and unit std. Then for the variance of the attention score between any query and key we get:</p>
  <p><span class="math-container">$$ \alpha = q_i k_j^T = \sum_{n=1}^{d_k} q_{in} k_{jn} $$</span>
  <span class="math-container">$$ \text{Var}(\alpha) = d_k $$</span>
  <span class="math-container">$$ \text{std}(\alpha) = \sqrt{d_k} $$</span></p>
  <p>Applying softmax on the attention scores with such high variance will result in all of the weight being placed on one random element, while all the other elements will have a weight of zero. Thus, in order to have the attention scores with unit std, we scale by <span class="math-container">$\sqrt{d_k}$</span>.</p>
  <p>But is it safe to assume that our query embeddings and key embeddings have unit variance?</p>
  <p>The answer is yes. The embeddings are computed by multiplying the input with the query and key matrices (<span class="math-container">$W_Q$</span> and <span class="math-container">$W_K$</span>). We can assume that the input already has unit variance by using a normalizing layer (e.g. LayerNorm), and the weights of the <span class="math-container">$W_Q$</span> and <span class="math-container">$W_K$</span> matrices should be initialized so that variance is preserved.</p>
  <p>For more checkout <a href="https://pi-tau.github.io/posts/transformer/#multi-head-attention-layer" rel="nofollow noreferrer">this</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-29T14:53:44.337Z
Answers.LastActivityDate: 2023-07-29T14:53:44.337Z
Answers.UserDisplayName: pi-tau
Body: <p>In Cross-attention, divide the similarity between <span class="math-container">$Q$</span> and <span class="math-container">$K$</span> by <span class="math-container">$\sqrt{d_k}$</span>. <br>
  Here <span class="math-container">$\sqrt{d_k}$</span> is the diagonal of the hypercube, is this a coincidence or famous theorem?
  <span class="math-container">$$ \frac{QK^T}{\sqrt{d_k}} $$</span>
  <a href="https://i.sstatic.net/OiFF9.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/OiFF9.png" alt="enter image description here" /></a></p>
  
Tags: <machine-learning><transformer><attention>
Title: diagonal of the hypercube in cross-attention
UserDisplayName: diffusion stable
ThreadText: # diagonal of the hypercube in cross-attention
  
  (Asked by: username_0 on 2023-07-29)
  
  In Cross-attention, divide the similarity between $Q$ and $K$ by $\sqrt{d\_k}$.   
  Here $\sqrt{d\_k}$ is the diagonal of the hypercube, is this a coincidence or famous theorem?
  $$ \frac{QK^T}{\sqrt{d\_k}} $$
  [![enter image description here](https://i.sstatic.net/OiFF9.png)](https://i.sstatic.net/OiFF9.png)
  
  **Tags:** machine-learning, transformer, attention
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-07-29. Score: 1
  
  You first compute the *query embeddings* $Q$ and the *key embeddings* $K$ by multiplying your input sequence $x$ with the query matrix $W\_Q$ and the key matrix $W\_K$ respectively. Then you calculate the attention scores and scale them by dividing by the dimensionality of the key embedding space.
  
  $$ A = \frac{xW\_QW\_K^Tx^T}{\sqrt{d\_k}} = \frac{QK^T}{\sqrt{d\_k}} $$
  
  After that you apply softmax to get the attention weights (probabilities). But to see why we scale by $\frac{1}{\sqrt{d\_k}}$ let’s assume that the query embeddings and key embeddings have zero mean and unit std. Then for the variance of the attention score between any query and key we get:
  
  $$ \alpha = q\_i k\_j^T = \sum\_{n=1}^{d\_k} q\_{in} k\_{jn} $$
  $$ \text{Var}(\alpha) = d\_k $$
  $$ \text{std}(\alpha) = \sqrt{d\_k} $$
  
  Applying softmax on the attention scores with such high variance will result in all of the weight being placed on one random element, while all the other elements will have a weight of zero. Thus, in order to have the attention scores with unit std, we scale by $\sqrt{d\_k}$.
  
  But is it safe to assume that our query embeddings and key embeddings have unit variance?
  
  The answer is yes. The embeddings are computed by multiplying the input with the query and key matrices ($W\_Q$ and $W\_K$). We can assume that the input already has unit variance by using a normalizing layer (e.g. LayerNorm), and the weights of the $W\_Q$ and $W\_K$ matrices should be initialized so that variance is preserved.
  
  For more checkout [this](https://pi-tau.github.io/posts/transformer/#multi-head-attention-layer).
  
  
  ---
  
  