Answers.Body: <p>To my understanding, there isn't any theoretical reason why the query, key and values weights are absent.</p>
  <blockquote>
  <p>I feel that the difference may lie in the way the additive attention is calculated vs the dot-product one.</p>
  </blockquote>
  <p>In the equations for the Graph Attention Network (GAT), there is for sure a difference between GAT and Transformers. However, I think you might be misunderstanding the relationship between the two. The attention mechanism in GATs is not equivalent to the one in Transformers. This means that one is not simply a rearrangement of the other; they are different inductive biases.</p>
  <blockquote>
  <p>If my understanding is correct, this means that matrix W represents the query, the key and the value transformation matrices all in one. But how can it do that?</p>
  </blockquote>
  <p>The equation states:</p>
  <p><span class="math-container">$$
  e_{ij} = a(Wh_i,Wh_j )
  $$</span></p>
  <p>You are correct that a single weight matrix <span class="math-container">$W$</span> is used for the projection from the feature space to the feature map. However, this doesn't mean that <span class="math-container">$W$</span> represents the query, key, and value matrices, although it serves those roles if compared to the Transformer architecture.</p>
  <p>I feel that you understand the forward pass of the GAT but lack intuition on why it is done like that. According to Aleks Gordic's commentary in the following <a href="https://youtu.be/uFLeKkXWq2c?si=5iSh9Nlzf7WOmhgz&amp;t=952" rel="nofollow noreferrer">video</a>, he mentioned that he spoke with the authors of the paper. They said they had indeed tried using separate query, key, and value weight matrices but encountered overfitting in their experiments. They concluded that using a single weight matrix yielded more generalizable results. This makes sense when considering that the goal of GATs is not to construct a language model, which is a more complex problem. GATs most often operate on static graph data with features to learn. They are designed not to copy Transformers, but to adapt Transformer principles to graph data and objectives.</p>
  <blockquote>
  <p>Is it theoretical or technical?</p>
  </blockquote>
  <p>Based on what I understand, most inductive biases, if not all, are not theoretically grounded. If you think about it, there is no paper that says why one method theoretically performs better than another in neural networks. This is because it ultimately depends on your data and your final goal for the model. We only have experimental data to support our claims and a sense of intuition. The difficult part is to develop that intuition, which is based on previous experiments.</p>
  <p>What I'm trying to say is that there is no reason why you couldn't try adding three different weight matrices instead of one. For the original idea of the Transformer, it makes sense to divide the attention into queries and keys because we are trying to simulate language, and language has a semantic ordering component. This means that it is not the same to say &quot;Hello World&quot; as it is to say &quot;World Hello.&quot; Based on this, it is logical to have extra weights that can identify this difference.</p>
  <p>For the datasets that were tried with GAT, they were citation networks and protein predictions. Predicting classes in citations may not require using more weights because the problem statement does not seem to require it. However, this is not set in stone. In the end, the only way to know if one architecture is better than another is through experimentation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-03T06:30:06.467Z
Answers.Text: Thank you for the explanation, this clears things up, especially with the fact that they actually tried to use different matrices first.
Answers.UserDisplayName: oliver.c
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-03T09:28:52.810Z
Answers.Text: Nice insight that a transformer-style attention in GAT causes overfitting. The following points confuse me a bit though: 1) In what way are transformer-attention and GAT-attention different inductive biases? 2) Inductive biases often are actually theoretically backed (s. [Geometric Deep Learning](https://geometricdeeplearning.com/)). 3) You last point about the word ordering should be less related to the choice of attention but rather to the use of a positional encoding (PE), without PE, the attention is permutation insensitive. What do you think?
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-04T19:19:51.243Z
Answers.Text: @Chillston
  1) I define inductive biases as "a set of assumptions that the learner uses to predict outputs." GAT assumes that the data is a graph, and Transformers assume that the data is a complete graph. Using the same definition, we can say that our model assumes the data is simpler because we are using a single weight matrix. This is another reason why the models has  differents inductive biases.
Answers.UserDisplayName: Cesar Ruiz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-04T19:20:04.163Z
Answers.Text: 2)Thank you for introducing me to this term. I am not familiar with geometric deep learning, but I will try to inform myself. What I was trying to say in my response is that papers that introduce new architectures do not provide a theoretical reason why using them performs better. For example, GANS, Diffusions, and Transformers are state-of-the-art architectures, but their papers do not explain why they are better.
Answers.UserDisplayName: Cesar Ruiz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-04T19:22:50.597Z
Answers.Text: 3)Even without positional encoding, the attention mechanism is indeed permutation sensitive. This is why we have query and key vectors, and why we use different weights for each one. We don't want the same attention for "word1 - word2" and "word2 - word1."
Answers.UserDisplayName: Cesar Ruiz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-06T16:49:42.327Z
Answers.Text: Good point for 1), I definitely agree on that. For 2) I think there are a lot of logical arguments to say that a Transformer is more powerful in text processing than an RNN (e.g. Transformers are better scalable, use a more general relational inductive bias, ...). But I think there can be different opinions on this matter.
  Regarding 3): Transformer wo/ pos-encoding is permutation equivariant. It cannot use order information purley from the self-Attention and that's the reason why they put the pos-encoding ([s. abstract here](https://arxiv.org/abs/2104.08698)).
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-06T16:51:11.463Z
Answers.Text: Also yea I can really recommend the Geometric Deep Learning stuff - it opens up a very nice perspective to think more generally about Deep Learning methods :)
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-07T16:35:36.360Z
Answers.Text: @Chillston,
  
  You're right that transformers  self-attention is permutation-equivariant, but I was not referring to that. I may have confused you, but I'm not talking about whether permuting the sequence would give you different attention scores. Rather, I'm talking about the fact that transformers attention is asymmetrical. I was searching for examples  and I found this [thread](https://stackoverflow.com/questions/75772288/how-to-read-a-bert-attention-weight-matrix) that also talks about it.  And regarding Geometric DL, I totally agree with you, It's a very interesting topic :)
Answers.UserDisplayName: Cesar Ruiz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-03T00:37:18.960Z
Answers.LastActivityDate: 2023-10-03T00:37:18.960Z
Answers.UserDisplayName: Cesar Ruiz
Body: <p>In section 2.1 of the <a href="https://arxiv.org/pdf/1710.10903.pdf" rel="nofollow noreferrer">Graph attention network paper</a></p>
  <p>The graph attention layer is described as</p>
  <blockquote>
  <p>as an initial step, a shared
  linear transformation, parametrized by a weight matrix, W ∈ RF ′×F , is applied to every node. We then perform self-attention on the nodes—a shared attentional mechanism a : RF ′
  × RF ′→ R computes attention coefficients eij = a(Whi, Whj ) that indicate the importance of node j’s features to node i.</p>
  </blockquote>
  <p>(forgive my amateur formatting)</p>
  <p>The function <code>a</code> represents a fully connected neural network that takes the concatenated vector of <code>Whi</code> and <code>Whj</code>, then outputs a single value which is pushed through a <code>softmax</code> to get the attention score <code>aij</code>. Then, the embedding vectors <code>Whj</code> (for all neighboring nodes of <code>i</code>) are weighted and summed by the attention score as normal.</p>
  <p>If my understanding is correct, this means that matrix W represents the query, the key and the value transformation matrices all in one. But how can it do that? I feel that the difference may lie in the way the additive attention is calculated vs the dot-product one, but I cannot comprehend how this works. Why can we use a shared matrix here and not there? Is it theoretical or technical?</p>
  
Tags: <transformer><attention><graph-neural-networks>
Title: Why is there a shared matrix W in graph attention networks instead of the query-key-value trio like in regular transformers?
UserDisplayName: oliver.c
ThreadText: # Why is there a shared matrix W in graph attention networks instead of the query-key-value trio like in regular transformers?
  
  (Asked by: username_0 on 2023-10-02)
  
  In section 2.1 of the [Graph attention network paper](https://arxiv.org/pdf/1710.10903.pdf)
  
  The graph attention layer is described as
  
  > as an initial step, a shared
  > linear transformation, parametrized by a weight matrix, W ∈ RF ′×F , is applied to every node. We then perform self-attention on the nodes—a shared attentional mechanism a : RF ′
  > × RF ′→ R computes attention coefficients eij = a(Whi, Whj ) that indicate the importance of node j’s features to node i.
  
  (forgive my amateur formatting)
  
  The function `a` represents a fully connected neural network that takes the concatenated vector of `Whi` and `Whj`, then outputs a single value which is pushed through a `softmax` to get the attention score `aij`. Then, the embedding vectors `Whj` (for all neighboring nodes of `i`) are weighted and summed by the attention score as normal.
  
  If my understanding is correct, this means that matrix W represents the query, the key and the value transformation matrices all in one. But how can it do that? I feel that the difference may lie in the way the additive attention is calculated vs the dot-product one, but I cannot comprehend how this works. Why can we use a shared matrix here and not there? Is it theoretical or technical?
  
  **Tags:** transformer, attention, graph-neural-networks
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-10-03. Score: 2 | Accepted answer
  
  To my understanding, there isn't any theoretical reason why the query, key and values weights are absent.
  
  > I feel that the difference may lie in the way the additive attention is calculated vs the dot-product one.
  
  In the equations for the Graph Attention Network (GAT), there is for sure a difference between GAT and Transformers. However, I think you might be misunderstanding the relationship between the two. The attention mechanism in GATs is not equivalent to the one in Transformers. This means that one is not simply a rearrangement of the other; they are different inductive biases.
  
  > If my understanding is correct, this means that matrix W represents the query, the key and the value transformation matrices all in one. But how can it do that?
  
  The equation states:
  
  $$
  e\_{ij} = a(Wh\_i,Wh\_j )
  $$
  
  You are correct that a single weight matrix $W$ is used for the projection from the feature space to the feature map. However, this doesn't mean that $W$ represents the query, key, and value matrices, although it serves those roles if compared to the Transformer architecture.
  
  I feel that you understand the forward pass of the GAT but lack intuition on why it is done like that. According to Aleks Gordic's commentary in the following [video](https://youtu.be/uFLeKkXWq2c?si=5iSh9Nlzf7WOmhgz&t=952), he mentioned that he spoke with the authors of the paper. They said they had indeed tried using separate query, key, and value weight matrices but encountered overfitting in their experiments. They concluded that using a single weight matrix yielded more generalizable results. This makes sense when considering that the goal of GATs is not to construct a language model, which is a more complex problem. GATs most often operate on static graph data with features to learn. They are designed not to copy Transformers, but to adapt Transformer principles to graph data and objectives.
  
  > Is it theoretical or technical?
  
  Based on what I understand, most inductive biases, if not all, are not theoretically grounded. If you think about it, there is no paper that says why one method theoretically performs better than another in neural networks. This is because it ultimately depends on your data and your final goal for the model. We only have experimental data to support our claims and a sense of intuition. The difficult part is to develop that intuition, which is based on previous experiments.
  
  What I'm trying to say is that there is no reason why you couldn't try adding three different weight matrices instead of one. For the original idea of the Transformer, it makes sense to divide the attention into queries and keys because we are trying to simulate language, and language has a semantic ordering component. This means that it is not the same to say "Hello World" as it is to say "World Hello." Based on this, it is logical to have extra weights that can identify this difference.
  
  For the datasets that were tried with GAT, they were citation networks and protein predictions. Predicting classes in citations may not require using more weights because the problem statement does not seem to require it. However, this is not set in stone. In the end, the only way to know if one architecture is better than another is through experimentation.
  
  > **Comments:**
  > **username_0**: Thank you for the explanation, this clears things up, especially with the fact that they actually tried to use different matrices first.  
  > **username_2**: Nice insight that a transformer-style attention in GAT causes overfitting. The following points confuse me a bit though: 1) In what way are transformer-attention and GAT-attention different inductive biases? 2) Inductive biases often are actually theoretically backed (s. [Geometric Deep Learning](https://geometricdeeplearning.com/)). 3) You last point about the word ordering should be less related to the choice of attention but rather to the use of a positional encoding (PE), without PE, the attention is permutation insensitive. What do you think?  
  > **username_1**: @username_2 1) I define inductive biases as "a set of assumptions that the learner uses to predict outputs." GAT assumes that the data is a graph, and Transformers assume that the data is a complete graph. Using the same definition, we can say that our model assumes the data is simpler because we are using a single weight matrix. This is another reason why the models has  differents inductive biases.  
  > **username_1**: 2)Thank you for introducing me to this term. I am not familiar with geometric deep learning, but I will try to inform myself. What I was trying to say in my response is that papers that introduce new architectures do not provide a theoretical reason why using them performs better. For example, GANS, Diffusions, and Transformers are state-of-the-art architectures, but their papers do not explain why they are better.  
  > **username_1**: 3)Even without positional encoding, the attention mechanism is indeed permutation sensitive. This is why we have query and key vectors, and why we use different weights for each one. We don't want the same attention for "word1 - word2" and "word2 - word1."  
  > **username_2**: Good point for 1), I definitely agree on that. For 2) I think there are a lot of logical arguments to say that a Transformer is more powerful in text processing than an RNN (e.g. Transformers are better scalable, use a more general relational inductive bias, ...). But I think there can be different opinions on this matter. Regarding 3): Transformer wo/ pos-encoding is permutation equivariant. It cannot use order information purley from the self-Attention and that's the reason why they put the pos-encoding ([s. abstract here](https://arxiv.org/abs/2104.08698)).  
  > **username_2**: Also yea I can really recommend the Geometric Deep Learning stuff - it opens up a very nice perspective to think more generally about Deep Learning methods :)  
  > **username_1**: @username_2,  You're right that transformers  self-attention is permutation-equivariant, but I was not referring to that. I may have confused you, but I'm not talking about whether permuting the sequence would give you different attention scores. Rather, I'm talking about the fact that transformers attention is asymmetrical. I was searching for examples  and I found this [thread](https://stackoverflow.com/questions/75772288/how-to-read-a-bert-attention-weight-matrix) that also talks about it.  And regarding Geometric DL, I totally agree with you, It's a very interesting topic :)  
  
  ---
  
  