Answers.Body: <p>Your calculations are correct, though training a whole <span class="math-container">$W$</span> is not always equivalent.</p>
  <p>We can agree that <span class="math-container">$W_q, W_k \in R^{n\times m}$</span>, thus <span class="math-container">$W_{qk} = W_qW_k^T \in R^{n\times n}$</span><br />
  Now, check the parameter count of <span class="math-container">$W_{qk}$</span> parametrized like this, and fully parametrized: in the former case, you have <span class="math-container">$2\cdot(n\cdot m)$</span> parameters, and in the latter <span class="math-container">$n^2$</span>. Already here you can see that there might be a difference if <span class="math-container">$m &lt; \frac{n}{2}$</span>.</p>
  <p>in other words, in this case the resulting matrix (the fully parametrized) might be full rank, where instead the factorized one has lower rank</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-27T16:08:44.987Z
Answers.Text: Interestin, I hadn't thought of that. Would it be fair to say that in the special case where  $m \ge \frac{n}{2}$ it's more parameter efficient to  use the combined weights? Is that uncommon in the wild?
Answers.UserDisplayName: fakedrake
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-27T16:54:31.903Z
Answers.Text: @fakedrake yeah i’d say that is more reasonable, tough i’ve not enough experience to say if that’s done usually or not
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-26T11:07:17.453Z
Answers.LastActivityDate: 2024-05-26T11:07:17.453Z
Answers.UserDisplayName: Alberto
Body: <p>If I understand correctly the attention layer is represented as
  <span class="math-container">$$
  \begin{align}
  &amp;softmax(\frac{Q K^T}{\sqrt{d_k}}) V  \\
  = &amp;softmax(\frac{(s W_q) (s W_k)^T}{\sqrt{d_k}}) V \\
  = &amp;softmax(\frac{(s W_q) (W_k^T s^T)}{\sqrt{d_k}}) V \\
  = &amp;softmax(\frac{s (W_q W_k^T) s^T}{\sqrt{d_k}}) V \\
  \end{align}
  $$</span></p>
  <p>Wouldn't it be better to take <span class="math-container">$W_q W_k^T$</span> as a single parameter?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-26T10:56:37.910Z
Comments.Text: dw, you got some "{}" missplaced, fixed for you
Comments.UserDisplayName: Alberto
Tags: <transformer><attention><weights><linear-algebra>
Title: Why are the Q and K matrices two separate matrices in attention?
UserDisplayName: fakedrake
ThreadText: # Why are the Q and K matrices two separate matrices in attention?
  
  (Asked by: username_0 on 2024-05-25)
  
  If I understand correctly the attention layer is represented as
  $$
  \begin{align}
  &softmax(\frac{Q K^T}{\sqrt{d\_k}}) V \\
  = &softmax(\frac{(s W\_q) (s W\_k)^T}{\sqrt{d\_k}}) V \\
  = &softmax(\frac{(s W\_q) (W\_k^T s^T)}{\sqrt{d\_k}}) V \\
  = &softmax(\frac{s (W\_q W\_k^T) s^T}{\sqrt{d\_k}}) V \\
  \end{align}
  $$
  
  Wouldn't it be better to take $W\_q W\_k^T$ as a single parameter?
  
  **Tags:** transformer, attention, weights, linear-algebra
  
  > **Comments:**
  > **username_1**: dw, you got some "{}" missplaced, fixed for you  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-05-26. Score: 1 | Accepted answer
  
  Your calculations are correct, though training a whole $W$ is not always equivalent.
  
  We can agree that $W\_q, W\_k \in R^{n\times m}$, thus $W\_{qk} = W\_qW\_k^T \in R^{n\times n}$  
  Now, check the parameter count of $W\_{qk}$ parametrized like this, and fully parametrized: in the former case, you have $2\cdot(n\cdot m)$ parameters, and in the latter $n^2$. Already here you can see that there might be a difference if $m < \frac{n}{2}$.
  
  in other words, in this case the resulting matrix (the fully parametrized) might be full rank, where instead the factorized one has lower rank
  
  > **Comments:**
  > **username_0**: Interestin, I hadn't thought of that. Would it be fair to say that in the special case where  $m \ge \frac{n}{2}$ it's more parameter efficient to  use the combined weights? Is that uncommon in the wild?  
  > **username_1**: @username_0 yeah i’d say that is more reasonable, tough i’ve not enough experience to say if that’s done usually or not  
  
  ---
  
  