Does attention matter and, if so, when and how? Our study on both inductive and transductive learning suggests that datasets have a strong influence on the effects of attention in graph neural networks. Independent of learning setting, task and attention variant, attention mostly degenerate to simple averaging for all three citation networks, whereas they behave strikingly different in the protein-protein interaction networks and molecular graphs: nodes attend to different neighbors per head and get more focused in deeper layers. Consequently, attention distributions become telltale features of the datasets themselves. We further explore the possibility of transferring attention for graph sparsification and show that, when applicable, attention-based sparsification retains enough information to obtain good performance while reducing computational and storage costs. Finally, we point out several possible directions for further study and transfer of attention.
This paper presents an empirical study of the attention mechanism in the graph attention networks (GAT).  The study reveals that the attention patterns largely depend on the dataset, on some datasets they are sharp, but on others the attention patterns are almost uniform and not so different from the uniform aggregation weights in GNNs that does not have attention.  The authors further tried to utilize these findings and attempted to do attention-based graph sparsification, and showed that they can get a similar level of performance with only a fraction of the edges in the original graph if they do the sparsification based on the attention weights.
Given the popularity of the GAT model in the graph neural networks literature and the effectiveness of the attention mechanism in many deep learning architectures, the empirical study presented in this paper focused on attention is valuable.  The experiments are clearly motivated and executed, which I appreciate.
As this is an empirical paper, one (maybe) problem with it is that the findings presented aren’t that surprising in hindsight - of course the attention patterns should be data dependent, and doing attention-based graph sparsification seems like an obvious thing that should work.  The results on dataset-dependent attention patterns may have told us more about the datasets rather than the GAT model.
There are a few presentation issues that need clarification:
- sec 5.1: it is not clear from the text what \alpha^{static} is.  As mentioned earlier there are multiple possible static attention weights (GCN vs GraphSAGE).
- sec 5.1: I found it strange to normalize the discrepancy score by 2|V|.  The number of terms in the sum should be 2|E| where |E| is the number of edges as each edge is counted twice.  Normalizing by 2|V| does not guarantee the score is in [0, 1] as claimed in the paper.
- sec 5.1: “Besides, these attention do not get concentrated on self loops based on relatively stable values.” -- I don’t see why having stable attention values can show there’s no concentration on self-loops.
- Figure 5 left: looks like the curves can’t reach the right end, this means 1 <= k <= 8 is probably not a good range.
- Table 5 is a bit confusing.  From the text my understanding is that a GAT is trained first, and then do sparsification, and then train another GraphSAGE on the sparsified graph to do prediction.  It’s not immediately clear why the GAT performance in Table 5 is so bad, while the GraphSAGE performance is just way better.  After reading this a few times I realized a much smaller GAT is used (with much smaller hidden size) while the GraphSAGE model is always using a large hidden size.  I think this part needs some improvement.
Overall I liked the empirical study and think the community can benefit from this paper.