Graph Neural Networks (GNNs) are deep models that operate on data with arbitrary topology represented as graphs. We introduce an efficient memory layer for GNNs that can jointly learn node representations and coarsen the graph. We also introduce two new networks based on this layer: Memory-Based Graph Neural Network (MemGNN) and Graph Memory Network (GMN) that can learn hierarchical graph representations. The experimental results shows that the proposed models achieve state-of-the-art results in eight out of nine graph classification and regression benchmarks. We also show that the learned representations could correspond to chemical features in the molecule data.
The paper proposes to use memory network (self-attention) to summarize information from graphs. The proposed approach "soft clusters" the node embeddings into coarser representation and eventually fed to a final MLP for classification. The empirical results on some standard datasets show promising gains of the algorithm.
The proposed approach stacks a few layers of self-attention on top of either some features of the nodes (including projected edge connections, parametrized by W_0) or the node embeddings of some form of graph neural network. And this stacking seems to be simple combination of existing approaches, without fully integrating them. In fact, the training process is also separated: "task-specific loss are back-propagated batch-wise while the gradients of the unsupervised loss are applied
epoch-wise". It makes me wonder whether we can just separate it into two stages, i.e., first learn a node embedding using graph neural network, then learn this self attention transformation. Due to the above issues, I feel the novelty of the approach is limited and incremental.
Another issue with the paper is that the notations seem to be messed up and some concepts are not explained clearly. For example, the C_{i,j} soft assignment matrix is normalized row-wise, then Eqn (3) seems very suspicious, because it averages the queries using weights along the other direction, thus not normalized. Also, the dimension of the MLP weights do not align well with inputs, for instance in Eqn (4), it should be written as V^(l) W.
There are more questions that are not clearly specified in the current manuscript. For example, where does the keys K come from? From the text description, it seems to be cluster results, and do you do the clustering on every gradient update? Or are they learned from scratch? The distribution P defined in Eqn (11) also seems to be difficult to optimize since it depends on C_{i,j} and is connected to different entries. Is simple SGD sufficient to optimize over P? Moreover, in the experiment section, it is unknown how many layers of self-attention is applied and what are the important parameters. For better comparison, the experiment section should include some estimate of parameter size as well.
The experiment results seem interesting since the approach indeed achieves good performance across many datasets. Also the visualized keys are interesting as well because it captures some meaningful patterns from the data.
There is some related work that you should cite:
Hanjun Dai, Bo Dai and Le Song. Discriminateive Embeddings of Latent Variable Models for Structured Data. International Conference on Machine Learning (ICML) 2016.
============================================================
Based on the authors' reply and other reviews, I have changed my rating to "Weak accept".
The authors' reply has clarified the important detail of how the key matrix is learned and now it is clear that the algorithm is not just two-stage separate learning. In light of this clarification, I think the proposed algorithm is novel enough and the jointly training mechanism is also beneficial for the state-of-the-arts results reported in the experiments.