Body: <p>I was looking at two papers </p>
  
  <ul>
  <li><a href="https://arxiv.org/pdf/1710.10903.pdf" rel="nofollow noreferrer">Graph Attention Networks</a> (GAT) by <em>Petar Veličković</em> and</li>
  <li><a href="https://arxiv.org/pdf/1803.07294.pdf" rel="nofollow noreferrer">GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs</a> by <em>Jiani Zhang</em>.</li>
  </ul>
  
  <p>I'm trying to implement the second paper and I'm having some troubles understanding the differences between GAT and GaAN. By looking at equation 1 in GaAN paper I can see only two differences from GAT. </p>
  
  <ul>
  <li>The first difference is that they are doing a dot product with the initial feature map and </li>
  <li>Have another fully connected layer to project the result. </li>
  </ul>
  
  <p>Is there something else that I'm missing?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-18T13:10:22.527Z
Comments.Text: wow the stuff is realy hardcore. About (1), looks like they are implementing it differently but still maintaing the structure, so its basicly the same. And (2) fc-layer is not important.  
  But did you understand what do thay do further with input? to put a graph of any length they must do some aggregation over variable length and i
Comments.UserDisplayName: user8426627
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-21T13:48:36.273Z
Comments.Text: Regarding point 1 in the GaAN paper they are stating: "The difference
  between our aggregator and that in GAT is that we have adopted the key-value attention mechanism and the dot product attention while GAT does not compute additional value vectors and uses a fully-connected layer". From my understanding those two implementations are quite different. Lets try to break it down. Lets say we have 2 nodes i,j where each have an embedding size of 3. When we are computing the GAT aggregator we combine the two nodes embeddings and feed them to a single ff neural net so we are learning 6 parameters.
Comments.UserDisplayName: razvanc92
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-21T13:55:56.433Z
Comments.Text: When computing the GaAN aggregator we define two weight matrices of size [3,3]. We than multiply node i with the first one and node j with the second one and have two matrices of size [1,3]. Afterwards we can use the dot product between them so that we only have 1 (the attention score). Now looking at the number of parameters GAT uses 6 vs GaAN 18 I think we can say that the two implementations are not similar. Please correct me if I am wrong.
Comments.UserDisplayName: razvanc92
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-06T08:07:27.207Z
Comments.Text: Did you completed the implementation of GaAN? I have found the implementation of GaAN in here https://github.com/jennyzhang0215/GaAN, but environment setup for that implementation is quite hardly, so I haven't been able to run it yet!
Comments.UserDisplayName: phamduy
Tags: <neural-networks><deep-learning><comparison><attention><geometric-deep-learning>
Title: What is the difference between GAT and GaAN?
UserDisplayName: razvanc92
ThreadText: # What is the difference between GAT and GaAN?
  
  (Asked by: username_0 on 2019-06-18)
  
  I was looking at two papers
  
  * [Graph Attention Networks](https://arxiv.org/pdf/1710.10903.pdf) (GAT) by *Petar Veličković* and
  * [GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs](https://arxiv.org/pdf/1803.07294.pdf) by *Jiani Zhang*.
  
  I'm trying to implement the second paper and I'm having some troubles understanding the differences between GAT and GaAN. By looking at equation 1 in GaAN paper I can see only two differences from GAT.
  
  * The first difference is that they are doing a dot product with the initial feature map and
  * Have another fully connected layer to project the result.
  
  Is there something else that I'm missing?
  
  **Tags:** neural-networks, deep-learning, comparison, attention, geometric-deep-learning
  > **Comments:**
  > **username_1**: wow the stuff is realy hardcore. About (1), looks like they are implementing it differently but still maintaing the structure, so its basicly the same. And (2) fc-layer is not important.   But did you understand what do thay do further with input? to put a graph of any length they must do some aggregation over variable length and i  
  > **username_0**: Regarding point 1 in the GaAN paper they are stating: "The difference between our aggregator and that in GAT is that we have adopted the key-value attention mechanism and the dot product attention while GAT does not compute additional value vectors and uses a fully-connected layer". From my understanding those two implementations are quite different. Lets try to break it down. Lets say we have 2 nodes i,j where each have an embedding size of 3. When we are computing the GAT aggregator we combine the two nodes embeddings and feed them to a single ff neural net so we are learning 6 parameters.  
  > **username_0**: When computing the GaAN aggregator we define two weight matrices of size [3,3]. We than multiply node i with the first one and node j with the second one and have two matrices of size [1,3]. Afterwards we can use the dot product between them so that we only have 1 (the attention score). Now looking at the number of parameters GAT uses 6 vs GaAN 18 I think we can say that the two implementations are not similar. Please correct me if I am wrong.  
  > **username_2**: Did you completed the implementation of GaAN? I have found the implementation of GaAN in here https://github.com/jennyzhang0215/GaAN, but environment setup for that implementation is quite hardly, so I haven't been able to run it yet!  
  
  ---
  
  ## 0 answers
  
  