<p>Your calculations are correct, though training a whole <span class="math-container">$W$</span> is not always equivalent.</p>
<p>We can agree that <span class="math-container">$W_q, W_k \in R^{n\times m}$</span>, thus <span class="math-container">$W_{qk} = W_qW_k^T \in R^{n\times n}$</span><br />
Now, check the parameter count of <span class="math-container">$W_{qk}$</span> parametrized like this, and fully parametrized: in the former case, you have <span class="math-container">$2\cdot(n\cdot m)$</span> parameters, and in the latter <span class="math-container">$n^2$</span>. Already here you can see that there might be a difference if <span class="math-container">$m &lt; \frac{n}{2}$</span>.</p>
<p>in other words, in this case the resulting matrix (the fully parametrized) might be full rank, where instead the factorized one has lower rank</p>

CC BY-SA 4.0
2024-05-27T16:08:44.987Z
Interestin, I hadn't thought of that. Would it be fair to say that in the special case where  $m \ge \frac{n}{2}$ it's more parameter efficient to  use the combined weights? Is that uncommon in the wild?
fakedrake
CC BY-SA 4.0
2024-05-27T16:54:31.903Z
@fakedrake yeah i’d say that is more reasonable, tough i’ve not enough experience to say if that’s done usually or not
Alberto
CC BY-SA 4.0
2024-05-26T11:07:17.453Z
2024-05-26T11:07:17.453Z
Alberto
<p>If I understand correctly the attention layer is represented as
<span class="math-container">$$
\begin{align}
&amp;softmax(\frac{Q K^T}{\sqrt{d_k}}) V  \\
= &amp;softmax(\frac{(s W_q) (s W_k)^T}{\sqrt{d_k}}) V \\
= &amp;softmax(\frac{(s W_q) (W_k^T s^T)}{\sqrt{d_k}}) V \\
= &amp;softmax(\frac{s (W_q W_k^T) s^T}{\sqrt{d_k}}) V \\
\end{align}
$$</span></p>
<p>Wouldn't it be better to take <span class="math-container">$W_q W_k^T$</span> as a single parameter?</p>

CC BY-SA 4.0
2024-05-26T10:56:37.910Z
dw, you got some "{}" missplaced, fixed for you
Alberto
<transformer><attention><weights><linear-algebra>
Why are the Q and K matrices two separate matrices in attention?
fakedrake
# Why are the Q and K matrices two separate matrices in attention?

(Asked by: username_0 on 2024-05-25)

If I understand correctly the attention layer is represented as
$$
\begin{align}
&softmax(\frac{Q K^T}{\sqrt{d\_k}}) V \\
= &softmax(\frac{(s W\_q) (s W\_k)^T}{\sqrt{d\_k}}) V \\
= &softmax(\frac{(s W\_q) (W\_k^T s^T)}{\sqrt{d\_k}}) V \\
= &softmax(\frac{s (W\_q W\_k^T) s^T}{\sqrt{d\_k}}) V \\
\end{align}
$$

Wouldn't it be better to take $W\_q W\_k^T$ as a single parameter?

**Tags:** transformer, attention, weights, linear-algebra

> **Comments:**
> **username_1**: dw, you got some "{}" missplaced, fixed for you  

---

## 1 answers

### Answer by username_1 on 2024-05-26. Score: 1 | Accepted answer

Your calculations are correct, though training a whole $W$ is not always equivalent.

We can agree that $W\_q, W\_k \in R^{n\times m}$, thus $W\_{qk} = W\_qW\_k^T \in R^{n\times n}$  
Now, check the parameter count of $W\_{qk}$ parametrized like this, and fully parametrized: in the former case, you have $2\cdot(n\cdot m)$ parameters, and in the latter $n^2$. Already here you can see that there might be a difference if $m < \frac{n}{2}$.

in other words, in this case the resulting matrix (the fully parametrized) might be full rank, where instead the factorized one has lower rank

> **Comments:**
> **username_0**: Interestin, I hadn't thought of that. Would it be fair to say that in the special case where  $m \ge \frac{n}{2}$ it's more parameter efficient to  use the combined weights? Is that uncommon in the wild?  
> **username_1**: @username_0 yeah i’d say that is more reasonable, tough i’ve not enough experience to say if that’s done usually or not  

---

