Answers.Body: <p>The answer is actually really simple: they are all randomly initialised. So they are to all intents and purposes &quot;normal&quot; weights of a neural network.</p>
  <p>This is also the reason why in the <a href="https://arxiv.org/pdf/1706.03762.pdf" rel="nofollow noreferrer">original paper</a> the authors tested several setting with single and multiple attention heads. If these matrices were somehow &quot;special&quot; or predetermined they would all serve the same purpose. Instead, because of their random initialisation, each attention head learn to contribute to solve a different task, like they show in Figure 3 and 4.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-01T02:16:59.413Z
Answers.Text: So how are those weight matrices updated? Is there a separate neural network for that?
Answers.UserDisplayName: Mark
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-01T10:55:43.277Z
Answers.Text: Why do you need another neural network? You just back propagate everything till these very first matrices. You can see from the tensorflow documentation that the matrices are just dense layers (or conv1D layers) https://github.com/tensorflow/tensor2tensor/blob/d9f807cf2738323d19aba0a20a8cf0c7f7da8b27/tensor2tensor/layers/common_attention.py#L2193
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-04T23:24:48.763Z
Answers.Text: Others said wights matrices(Wq Wk Wv) are trained by the network while still someone like you said it's randomly initialized. It make sense if u said wights matrices was randomly initialized on the very beginning of training stage, But I dont think there is any initialization of them on inference stage, am I correct?
Answers.UserDisplayName: Wuaner
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-31T14:29:28.267Z
Answers.LastActivityDate: 2020-08-31T14:29:28.267Z
Answers.UserDisplayName: Edoardo Guerriero
Answers.Body: <p>In my mind there are two weight matrices, the one you get prior to applying softmax:</p>
  <p><span class="math-container">$$ \alpha_{i,j} = \frac{\langle q_i, k_j \rangle}{\sqrt{d}}$$</span></p>
  <p>the other you get after applying the softmax:</p>
  <p><span class="math-container">$$ \text{Attn}_{i,j}(X) = \frac{\exp \left( \frac{\langle q_i, k_j \rangle}{\sqrt{d}}\right)}{\sum_k \exp\left( \frac{\langle q_i, k_k \rangle}{\sqrt{d}} \right)}.$$</span></p>
  <p>Either way, you can view them as directed &quot;<a href="https://ai.stackexchange.com/questions/32036/is-the-multi-head-attention-in-the-transformer-a-weighted-adjacency-matrix/39571#39571">attention graphs</a>&quot; and they can be interpreted in terms of <a href="https://docs.dgl.ai/en/0.8.x/tutorials/models/4_old_wines/7_transformer.html" rel="nofollow noreferrer">graph neural networks</a> using only complete graphs and graph attention:</p>
  <p><span class="math-container">$$ A_{i,j} = \frac{\exp(w(i,j))}{\sum_{j \in \mathcal{N}(i)} \exp(w(i,k))},$$</span></p>
  <p>where <span class="math-container">$w(i,j)$</span> is a weight function, which can be the scaled dot-product of attention for example. The sum is over the neighborhood of a node (think token) <span class="math-container">$\mathcal{N}(i)$</span>. Hope that helps a little with intuition and that it's close to what you meant.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-17T00:32:53.097Z
Answers.LastActivityDate: 2023-03-27T18:46:04.760Z
Answers.LastEditDate: 2023-03-27T18:46:04.760Z
Answers.UserDisplayName: Amelie Schreiber
Body: <p>I've been looking into self-attention lately, and in the articles that I've been seeing, they all talk about &quot;weights&quot; in attention. My understanding is that the weights in self-attention are not the same as the weights in a neural network.</p>
  <p>From this article, <a href="http://peterbloem.nl/blog/transformers" rel="noreferrer">http://peterbloem.nl/blog/transformers</a>, in the additional tricks section, it mentions,</p>
  <p>The query is the dot product of the query weight matrix and the word vector,
  <code>ie, q = W(q)x</code> and the key is the dot product of the key weight matrix and the word vector, <code>k = W(k)x</code> and similarly for the value it is <code>v = W(v)x</code>. So my question is, where do the weight matrices come from?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-30T00:56:54.080Z
Comments.Text: Hi. I just wanted to let you know that you can use latex on this site.
Comments.UserDisplayName: nbro
Tags: <neural-networks><transformer><attention>
Title: What is the weight matrix in self-attention?
UserDisplayName: Mark
ThreadText: # What is the weight matrix in self-attention?
  
  (Asked by: username_0 on 2020-08-29)
  
  I've been looking into self-attention lately, and in the articles that I've been seeing, they all talk about "weights" in attention. My understanding is that the weights in self-attention are not the same as the weights in a neural network.
  
  From this article, <http://peterbloem.nl/blog/transformers>, in the additional tricks section, it mentions,
  
  The query is the dot product of the query weight matrix and the word vector,
  `ie, q = W(q)x` and the key is the dot product of the key weight matrix and the word vector, `k = W(k)x` and similarly for the value it is `v = W(v)x`. So my question is, where do the weight matrices come from?
  
  **Tags:** neural-networks, transformer, attention
  > **Comments:**
  > **username_1**: Hi. I just wanted to let you know that you can use latex on this site.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-08-31. Score: 3
  
  The answer is actually really simple: they are all randomly initialised. So they are to all intents and purposes "normal" weights of a neural network.
  
  This is also the reason why in the [original paper](https://arxiv.org/pdf/1706.03762.pdf) the authors tested several setting with single and multiple attention heads. If these matrices were somehow "special" or predetermined they would all serve the same purpose. Instead, because of their random initialisation, each attention head learn to contribute to solve a different task, like they show in Figure 3 and 4.
  
  > **Comments:**
  > **username_0**: So how are those weight matrices updated? Is there a separate neural network for that?  
  > **username_2**: Why do you need another neural network? You just back propagate everything till these very first matrices. You can see from the tensorflow documentation that the matrices are just dense layers (or conv1D layers) https://github.com/tensorflow/tensor2tensor/blob/d9f807cf2738323d19aba0a20a8cf0c7f7da8b27/tensor2tensor/layers/common_attention.py#L2193  
  > **username_3**: Others said wights matrices(Wq Wk Wv) are trained by the network while still someone like you said it's randomly initialized. It make sense if u said wights matrices was randomly initialized on the very beginning of training stage, But I dont think there is any initialization of them on inference stage, am I correct?  
  
  ---
  
  ### Answer by username_4 on 2023-03-17. Score: 0
  
  In my mind there are two weight matrices, the one you get prior to applying softmax:
  
  $$ \alpha\_{i,j} = \frac{\langle q\_i, k\_j \rangle}{\sqrt{d}}$$
  
  the other you get after applying the softmax:
  
  $$ \text{Attn}\_{i,j}(X) = \frac{\exp \left( \frac{\langle q\_i, k\_j \rangle}{\sqrt{d}}\right)}{\sum\_k \exp\left( \frac{\langle q\_i, k\_k \rangle}{\sqrt{d}} \right)}.$$
  
  Either way, you can view them as directed "[attention graphs](https://ai.stackexchange.com/questions/32036/is-the-multi-head-attention-in-the-transformer-a-weighted-adjacency-matrix/39571#39571)" and they can be interpreted in terms of [graph neural networks](https://docs.dgl.ai/en/0.8.x/tutorials/models/4_old_wines/7_transformer.html) using only complete graphs and graph attention:
  
  $$ A\_{i,j} = \frac{\exp(w(i,j))}{\sum\_{j \in \mathcal{N}(i)} \exp(w(i,k))},$$
  
  where $w(i,j)$ is a weight function, which can be the scaled dot-product of attention for example. The sum is over the neighborhood of a node (think token) $\mathcal{N}(i)$. Hope that helps a little with intuition and that it's close to what you meant.
  
  
  ---
  
  