CAPSULE GRAPH NEURAL NETWORK
The high-quality node embeddings learned from the Graph Neural Networks (GNNs) have been applied to a wide range of node-based applications and some of them have achieved state-of-the-art (SOTA) performance. However, when applying node embeddings learned from GNNs to generate graph embeddings, the scalar node representations may not suffice to preserve the node/graph properties efficiently, resulting in sub-optimal graph embeddings. Inspired by the Capsule Neural Network (CapsNet) (Sabour et al., 2017), we propose the Capsule Graph Neural Network (CapsGNN), which adopts the concept of capsules to address the weakness in existing GNN-based graph embeddings algorithms. By extracting node features in the form of capsules, routing mechanism can be utilized to capture important information at the graph level. As a result, our model generates multiple embeddings for each graph to capture graph properties from different aspects. The attention module incorporated in CapsGNN is used to tackle graphs with various sizes which also enables the model to focus on critical parts of the graphs. Our extensive evaluations with 10 graph-structured datasets demonstrate that CapsGNN has a powerful mechanism that operates to capture macroscopic properties of the whole graph by data-driven. It outperforms other SOTA techniques on several graph classification tasks, by virtue of the new instrument.
The authors provide an architecture that applies recent advances in the field of capsule networks in the graph neural network domain. First, hierarchical node level capsules are extracted using GCN layers. Second, after weighting each capsule by the output of a proposed attention module, graph level capsules are computed by performing a global dynamic routing. These graph level capsules are used for training a capsule classifier using a margin loss and a reconstruction loss.
The general architecture seems to be a reasonable application of the capsule principle in the graph domain, following the proof of concept MNIST architecture proposed by Sabour et al.
My main concern is that I have problems grasping the motivation behind using capsules in the given scenario. Besides an unprecise motivation in the introduction, there is no clear reason why the routing mechanism helps with solving the given tasks. Capsule networks capture pose covariances by applying a linear, trainable transformation to pose vectors and computing the agreement of the resulting votes. It is not clear to me how discrete information like graph connectivity can be encoded in a pose vector so that linear transformations are able to match different "connectivity poses".
Is there a more formal argument that explains why capsules should be able to capture more information about the input graph than other GCNNs?
Also, some design choices seem to be quite arbitrary. One example is using the last feature maps of the GCN as positions for coordinate addition. Is there a theoretical/intuitive motivation for this?
Results for the given experiments show improvement on some graphs. However, the authors proposed several concepts: a global pooling method using dynamic routing, an attention mechanism, a novel reconstruction loss, interpreting deep node embeddings as spatial positions. It is not clear to what extent the individual aspects of the method contribute to the gains. The qualitative capsule embedding analysis is interesting. However, this part needs a comparison to standard global graph embeddings to see if there is a significant difference.
In my opinion, the paper needs:
1) a clear experimental evaluation showing that capsules and the dynamic routing lead to improved results (i.e. by providing an ablation study to show which gains result from the attention-based global pooling mechanism, the reconstruction loss, the dynamic routing and from the coordinate addition), or
2) a more precise motivation for the use of dynamic routing to capture correlation between pose vectors in graphs in general (i.e. formal arguments why the method is stronger in capturing statistics or for what types of graphs it provides more discriminative power).
Overall, the paper does not convince me that capsules and dynamic routing provide advantages if used like the authors propose. Therefore, I tend to voting for rejecting the paper as long as points 1) and 2) are not addressed properly.
Minor remarks:
- There are quite a lot of grammatical errors (especially missing articles).
--------------------------
Update:
The authors addressed some of the weak points mentioned above adequately. The experimental evaluation was significantly improved and the results are a nice contribution. However, the theoretical contribution and the poor motivation of capsules in the graph context remain weak points. I have updated my rating accordingly.