Answers.Body: <p>Let me come up with a possibly too naive (and too high-level) answer: Since the weights of the word embedding layer (i.e. the word embeddings) are not trained in advance (via cooccurrence) and independently from the training of the transformer weights (but happens in the same training runs), the word embeddings don't necessarily indicate semantic similarity when taking the dot product (as they do when using Word2Vec or so). So this doesn't have to be expected neither when choosing the same matrix <span class="math-container">$W = W^Q =W^K= W^V$</span> in each attention head: the dot product doesn't give scores to be interpreted as similarities but just of some (hard to interpret and only sometimes symmetric) relation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-25T20:30:36.703Z
Answers.Text: Doesn't this contradicts the role of the embedding layer? I mean **embedding followed by projection** is just a **linear layer**.
Answers.UserDisplayName: Antonios Sarikas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-15T09:47:22.173Z
Answers.LastActivityDate: 2023-06-15T10:00:09.437Z
Answers.LastEditDate: 2023-06-15T10:00:09.437Z
Answers.UserDisplayName: Hans-Peter Stricker
Answers.Body: <p>Actually a multi-head attention (MHA) layer has multiple &quot;heads&quot;, where each head is made by three *projection matrices like: <span class="math-container">$H_i = (W_i^Q, W_i^K, W_i^V)$</span>, for the <span class="math-container">$i$</span>-th head. A typical number of heads value is eight, for example: each head is responsible of learning to attend to different parts of the sequence, thus you can (partially) <em>interpret</em> transformer-based models by inspecting their <em>attention masks</em>.</p>
  <p>A word embedding <span class="math-container">$x_i$</span> is not directly fed to MHA. I mean MHA accepts three matrices <span class="math-container">$Q$</span> (query), <span class="math-container">$K$</span> (key) and <span class="math-container">$V$</span> (value), which got projected by <span class="math-container">$W^Q$</span>, <span class="math-container">$W^K$</span> and <span class="math-container">$W^V$</span>, respectively, when computing the attention scores (i.e. the scaled dot-product attention.) According to the <a href="https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf" rel="nofollow noreferrer">paper</a> (see section 3.2.1) and <a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/MultiHeadAttention" rel="nofollow noreferrer">here</a>, keys <span class="math-container">$d_k$</span> and value <span class="math-container">$d_k$</span> dimensions can be different, simply because these can be different sequences of (word) embeddings. For example, in a Question-Answer model <span class="math-container">$Q$</span> can be the embeddings from the question, while <span class="math-container">$K$</span> or <span class="math-container">$V$</span> of the expected answer. So learning different weight matrices, <span class="math-container">$W^Q$</span>, <span class="math-container">$W^K$</span>, and <span class="math-container">$W^V$</span>, allows
  to capture different relevant aspects for each input. Even when <span class="math-container">$Q=K=V$</span>, i.e. the so called <em>self-attention</em>, the matrices are still fundamental.</p>
  <p>In general, consider that queries attend to keys to yield the <em>attention probabilities</em> (via dot-product and softmax) which are used to weight the values. So they basically say which timestep (and so which work embedding in the sequence) is relevant in values (the learned representation by projection.) In other words, the general meaning is which words (in <span class="math-container">$V$</span>) are relevant to attend at time <span class="math-container">$t$</span>: so something that goes beyond a simple similarity (which may occur in the self-attention case.)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-15T17:02:34.780Z
Answers.Text: Thanks, but I think you are wrong in calling the case Q = K = V self-attention. Self-attention means to compare the tokens of one sequence with its own, not of another sequence. Furthermore, my question has nothing to do with multi-head attention, I was asking for the three matrices inside one attention head.
Answers.UserDisplayName: Hans-Peter Stricker
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-16T15:50:24.170Z
Answers.Text: @Hans-PeterStricker I thought you were confusing the three matrices and multi-head attention, sorry. Anyway, the tensorflow's docs say "If $Q$, $K$, $V$ are the same, then this is self-attention." That's what I meant by $Q=K=V$: I should have written that all are the same sequence, indeed.
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-16T15:59:30.060Z
Answers.Text: @Hans-PeterStricker The authors say "instead of perfoming a single attention function [...] we found beneficial to linearly project the queries, keys and values $h$ times with different learned projections [...]" (see beginning of section 3.2.2). So, I think they validated such design with prior experiments. Probably the linear projections "rotate" the representation of $Q$, $K$, and $V$, such that learning is easier.
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-17T07:43:35.207Z
Answers.Text: Thanks a lot, Luca. To be honest, I find the quote from tensorflow's docs a bit misleading and I dare to doubt that this is a definition of "self-attention". I've always read that self-attention is defined as paying attention of the tokens of a sequence to the tokens of the same sequence.
Answers.UserDisplayName: Hans-Peter Stricker
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-17T07:48:39.643Z
Answers.Text: From [tf.keras.layers.Attention](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Attention): "Inputs are query tensor of shape [batch_size, Tq, dim], value tensor of shape [batch_size, Tv, dim] and key tensor of shape [batch_size, Tv, dim]." It states that Tk = Tv, which doesn't make sense. It must be Tk=Tq, and that's how it's stated in the Attention is all you need paper: "learned linear projections to dk, dk and dv dimensions". So the tensorflow documentation is faulty sometimes, and not too reliable a source?
Answers.UserDisplayName: Hans-Peter Stricker
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-17T14:24:05.420Z
Answers.Text: @Hans-PeterStricker Also PyTorch's [docs](https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html) states the same. Moreover in TF's doc it says that the key is optional, and if not provided will use value for both K and V. [This](https://sebastianraschka.com/blog/2023/self-attention-from-scratch.html) explanation, defines the self-attention inputs by projecting the embeddings $x$ three times to $Q,K$ and $V$: this is also my intuition, which should also match the fact that you want to pay attention to tokens of the same input sequence $x$ (although projected.)
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-15T16:55:01.413Z
Answers.LastActivityDate: 2023-06-16T15:54:21.080Z
Answers.LastEditDate: 2023-06-16T15:54:21.080Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>My question is why the attention head matrices <span class="math-container">$W^Q$</span>, <span class="math-container">$W^K$</span>, <span class="math-container">$W^V$</span> should not be the same <span class="math-container">$W = W^Q =W^K= W^V$</span>. In my understanding of transformer-based language models  <strong>one</strong> attention head is responsible for <strong>one</strong> syntactic or semantic relation between any two words in the context. One might think that such a relation is represented by <strong>one</strong> matrix <span class="math-container">$W$</span> that projects the full word embeddings <span class="math-container">$x_i$</span> from their full semantic space to a semantic subspace responsible for this relation. Here we could - in principle - calculate scores <span class="math-container">$\sigma_{ij}$</span> as &quot;similiarities&quot; between two projected words <span class="math-container">$Wx_i$</span> and <span class="math-container">$Wx_j$</span> and then calculate the weighted sum of the projected tokens <span class="math-container">$Wx_k$</span>.</p>
  <p>I wonder why this would not work, and why we need three different matrices.</p>
  <p>Another way around: What does it mean to calculate the score as the dot-product of two vectors from two different semantic subspaces? Is this still some kind of similiarity (which lies at the heart of word embeddings)? And doesn't it sound like comparing apples and pears?</p>
  <p>Or viewed differently: How similar are the three matrices of an attention head in practice, e.g. when considering some 100<span class="math-container">$\times$</span>100 attention heads of a large transformer model like ChatGPT?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-17T14:30:35.913Z
Comments.Text: I found [this](https://ai.stackexchange.com/a/25225/69468) answer that provides a motivation for using multiple projection matrices. Basically, is also a way to save model parameters. Since the embedding dimension of $x$ is usually larger than the dimensions used for the projections of $Q$, $K$, and $V$.
Comments.UserDisplayName: Luca Anzalone
Tags: <natural-language-processing><transformer><attention><explainable-ai><vector-semantics>
Title: Why shouldn't the attention matrices $W^Q$, $W^K$, $W^V$ be the same?
UserDisplayName: Hans-Peter Stricker
ThreadText: # Why shouldn't the attention matrices $W^Q$, $W^K$, $W^V$ be the same?
  
  (Asked by: username_0 on 2023-06-15)
  
  My question is why the attention head matrices $W^Q$, $W^K$, $W^V$ should not be the same $W = W^Q =W^K= W^V$. In my understanding of transformer-based language models **one** attention head is responsible for **one** syntactic or semantic relation between any two words in the context. One might think that such a relation is represented by **one** matrix $W$ that projects the full word embeddings $x\_i$ from their full semantic space to a semantic subspace responsible for this relation. Here we could - in principle - calculate scores $\sigma\_{ij}$ as "similiarities" between two projected words $Wx\_i$ and $Wx\_j$ and then calculate the weighted sum of the projected tokens $Wx\_k$.
  
  I wonder why this would not work, and why we need three different matrices.
  
  Another way around: What does it mean to calculate the score as the dot-product of two vectors from two different semantic subspaces? Is this still some kind of similiarity (which lies at the heart of word embeddings)? And doesn't it sound like comparing apples and pears?
  
  Or viewed differently: How similar are the three matrices of an attention head in practice, e.g. when considering some 100$\times$100 attention heads of a large transformer model like ChatGPT?
  
  **Tags:** natural-language-processing, transformer, attention, explainable-ai, vector-semantics
  > **Comments:**
  > **username_1**: I found [this](https://ai.stackexchange.com/a/25225/69468) answer that provides a motivation for using multiple projection matrices. Basically, is also a way to save model parameters. Since the embedding dimension of $x$ is usually larger than the dimensions used for the projections of $Q$, $K$, and $V$.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2023-06-15. Score: 1
  
  Let me come up with a possibly too naive (and too high-level) answer: Since the weights of the word embedding layer (i.e. the word embeddings) are not trained in advance (via cooccurrence) and independently from the training of the transformer weights (but happens in the same training runs), the word embeddings don't necessarily indicate semantic similarity when taking the dot product (as they do when using Word2Vec or so). So this doesn't have to be expected neither when choosing the same matrix $W = W^Q =W^K= W^V$ in each attention head: the dot product doesn't give scores to be interpreted as similarities but just of some (hard to interpret and only sometimes symmetric) relation.
  
  > **Comments:**
  > **username_2**: Doesn't this contradicts the role of the embedding layer? I mean **embedding followed by projection** is just a **linear layer**.  
  
  ---
  
  ### Answer by username_1 on 2023-06-15. Score: 0
  
  Actually a multi-head attention (MHA) layer has multiple "heads", where each head is made by three \*projection matrices like: $H\_i = (W\_i^Q, W\_i^K, W\_i^V)$, for the $i$-th head. A typical number of heads value is eight, for example: each head is responsible of learning to attend to different parts of the sequence, thus you can (partially) *interpret* transformer-based models by inspecting their *attention masks*.
  
  A word embedding $x\_i$ is not directly fed to MHA. I mean MHA accepts three matrices $Q$ (query), $K$ (key) and $V$ (value), which got projected by $W^Q$, $W^K$ and $W^V$, respectively, when computing the attention scores (i.e. the scaled dot-product attention.) According to the [paper](https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf) (see section 3.2.1) and [here](https://www.tensorflow.org/api_docs/python/tf/keras/layers/MultiHeadAttention), keys $d\_k$ and value $d\_k$ dimensions can be different, simply because these can be different sequences of (word) embeddings. For example, in a Question-Answer model $Q$ can be the embeddings from the question, while $K$ or $V$ of the expected answer. So learning different weight matrices, $W^Q$, $W^K$, and $W^V$, allows
  to capture different relevant aspects for each input. Even when $Q=K=V$, i.e. the so called *self-attention*, the matrices are still fundamental.
  
  In general, consider that queries attend to keys to yield the *attention probabilities* (via dot-product and softmax) which are used to weight the values. So they basically say which timestep (and so which work embedding in the sequence) is relevant in values (the learned representation by projection.) In other words, the general meaning is which words (in $V$) are relevant to attend at time $t$: so something that goes beyond a simple similarity (which may occur in the self-attention case.)
  
  > **Comments:**
  > **username_0**: Thanks, but I think you are wrong in calling the case Q = K = V self-attention. Self-attention means to compare the tokens of one sequence with its own, not of another sequence. Furthermore, my question has nothing to do with multi-head attention, I was asking for the three matrices inside one attention head.  
  > **username_1**: @Hans-PeterStricker I thought you were confusing the three matrices and multi-head attention, sorry. Anyway, the tensorflow's docs say "If $Q$, $K$, $V$ are the same, then this is self-attention." That's what I meant by $Q=K=V$: I should have written that all are the same sequence, indeed.  
  > **username_1**: @Hans-PeterStricker The authors say "instead of perfoming a single attention function [...] we found beneficial to linearly project the queries, keys and values $h$ times with different learned projections [...]" (see beginning of section 3.2.2). So, I think they validated such design with prior experiments. Probably the linear projections "rotate" the representation of $Q$, $K$, and $V$, such that learning is easier.  
  > **username_0**: Thanks a lot, Luca. To be honest, I find the quote from tensorflow's docs a bit misleading and I dare to doubt that this is a definition of "self-attention". I've always read that self-attention is defined as paying attention of the tokens of a sequence to the tokens of the same sequence.  
  > **username_0**: From [tf.keras.layers.Attention](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Attention): "Inputs are query tensor of shape [batch_size, Tq, dim], value tensor of shape [batch_size, Tv, dim] and key tensor of shape [batch_size, Tv, dim]." It states that Tk = Tv, which doesn't make sense. It must be Tk=Tq, and that's how it's stated in the Attention is all you need paper: "learned linear projections to dk, dk and dv dimensions". So the tensorflow documentation is faulty sometimes, and not too reliable a source?  
  > **username_1**: @Hans-PeterStricker Also PyTorch's [docs](https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html) states the same. Moreover in TF's doc it says that the key is optional, and if not provided will use value for both K and V. [This](https://sebastianraschka.com/blog/2023/self-attention-from-scratch.html) explanation, defines the self-attention inputs by projecting the embeddings $x$ three times to $Q,K$ and $V$: this is also my intuition, which should also match the fact that you want to pay attention to tokens of the same input sequence $x$ (although projected.)  
  
  ---
  
  