GRAPH ATTENTION NETWORKS
We present graph attention networks (GATs), novel neural network architectures that operate on graph-structured data, leveraging masked self-attentional layers to address the shortcomings of prior methods based on graph convolutions or their approximations. By stacking layers in which nodes are able to attend over their neighborhoods' features, we enable (implicitly) specifying different weights to different nodes in a neighborhood, without requiring any kind of computationally intensive matrix operation (such as inversion) or depending on knowing the graph structure upfront. In this way, we address several key challenges of spectral-based graph neural networks simultaneously, and make our model readily applicable to inductive as well as transductive problems. Our GAT models have achieved or matched state-of-the-art results across four established transductive and inductive graph benchmarks: the Cora, Citeseer and Pubmed citation network datasets, as well as a protein-protein interaction dataset (wherein test graphs remain unseen during training).
The paper introduces a neural network architecture to operate on graph-structured
data named Graph Attention Networks.
Key components are an attention layer and the possibility to learn how to
weight different nodes in the neighborhood without requiring spectral decompositions
which are costly to be computed.
I found the paper clearly written and very well presented. I want to thank
the author for actively participating in the discussions and in clarifying already
many of the details that I was missing.
As also reported in the comments by T. Kipf I found the lack of comparison to previous
works on attention and on constructions of NN for graph data are missing.
In particular MoNet seems a more general framework, using features to compute node
similarity is another way to specify the "coordinate system" for convolution.
I would argue that in many cases the graph is given and that one would have
to exploit its structure rather than the simple first order neighbors structure.
I feel, in fact, that the paper deals mainly with "localized metric-learning" rather than
using the information in the graph itself. There is no
explicit usage of the graph beyond the selection of the local neighborhood.
In many ways when I first read it I though it would be a modified version of
memory networks (which have not been cited). Sec. 2.1 is basically describing
a way to learn a matrix W so that the attention layer produces the weights to be
used for convolution, or the relative coordinate system, which is to me a
memory network like construction, where the memory is given by the neighborhood.
I find the idea to use the multi-head attention very interesting, but one should
consider the increase in number of parameters in the experimental section.
I agree that the proposed method is computationally efficient but the authors
should keep in mind that parallelizing across all edges involves lot of redundant
copies (e.g. in a distributed system) as the neighborhoods highly overlap, at
least for interesting graphs.
The advantage with respect to methods that try to use LSTM in this domain
in a naive manner is clear, however the similarity function (attention) in this
work could be interpreted as the variable dictating the visit ordering.
The authors seem to emphasize the use of GPU as the best way to scale their work
but I tend to think that when nodes have varying degrees they would be highly
unused. Main reason why they are widely used now is due to the structure in the
representation of convolutional operations.
Also in case of sparse data GPUs are not the best alternative.
Experiments are very well described and performed, however as explained earlier
some comparisons are needed.
An interesting experiment could be to use the attention weights as adjacency
matrix for GCN.
Overall I liked the paper and the presentation, I think it is a simple yet
effective way of dealing with graph structure data. However, I think that in
many interesting cases the graph structure is relevant and cannot be used
just to get the neighboring nodes (e.g. in social network analysis).