In this work, we develop quantitative results to the learnability of a two-layers Graph Convolutional Network (GCN). Instead of analyzing GCN under some classes of functions, our approach provides a quantitative gap between a two-layers GCN and a two-layers MLP model. Our analysis is based on the graph signal processing (GSP) approach, which can provide much more useful insights than the message-passing computational model. Interestingly, based on our analysis, we have been able to empirically demonstrate a few cases when GCN and other state-of-the-art models cannot learn even when true vertex features are extremely low-dimensional. To demonstrate our theoretical findings and propose a solution to the aforementioned adversarial cases, we build a proof of concept graph neural network model with stacked filters named Graph Filters Neural Network (gfNN).
Summary
Several theoretical analyses have been conducted on MPNN-type NNs. On the other hand, although many graph neural network models take the approach of graph filtering, there is little theoretical analysis on graph NNs that take the filtering approach. This paper analyzed the learnability of filtering type graph NNs.
First, the authors hypothesized that the data used practical tasks consisted of true low-frequency information and high-frequency noise (Assumption 2), and verified the correctness of the assumption.
Next, the authors quantitatively evaluated the difference between the case where we feed noisy data to a graph NN and the case where we feed noiseless data to an MLP. The authors showed that a two-layer GCN approximately behaves as a noise-filtering + a two-layered MLP (Theorem 8). Based on this observation, the authors proposed gfNN, which directly models the latter architecture. We can interpret gfNN as an SGC whose final linear transformation is replaced with an MLP.
Finally, the authors empirically showed that gfNN achieved the same level of accuracy as existing GNNs in the citation network tasks. Furthermore, the authors created a dataset that has meaningful information in the high-frequency area (BA-High). It was shown that the existing GNN, including gfNN, did not perform well, while a GNN designed to pass high frequency (gfNN-high) can predict accurately.
Decision
I judge that this paper contributes to deepning the understanding of graph NNs and is worth to be accepted based on the following three points.
First, it enabled us to understand what causes the oversmoothing phenomena. Several studies have shown that Laplacian-type graph convolution works as a low pass filter. However, most of them considered a linear setting and did not explain how the graph convolution behaves when a graph NN has non-linear activation functions. Compared to them, their result admits non-linearity.
Second, by showing that we can approximate a GCN by a noise filter followed by an MLP, this paper has made the relationship between a GCN and an SGC clearer.
Finally, the authors experimentally showed that existing graph NNs do not have predictive power when useful information is in high-frequency domains. It gives insights on what graph NNs can and cannot solve.
For these reasons, I think this paper has contributed to a deeper understanding of graph NNs and sufficiently significant.
Suggestions
- If I understand correctly, the title of Section 4 reflects the content of Lemma 5 in Section 5 rather than the content of Theorem 3 and (5). I recommend the authors to reconsider the titles of Sections 4 and 5.
- Could you add more explanations to the proof of Theorem 8 in Appendix A. Especially, I could not understand how the term $\tilde{O}(\epsilon^{1/4})$ is derived.
Questions
- I could not find implementations of graph NNs in the notebook to the code (I only found the ls result of dataset directories). Do you plan to release the experiment code?
- At the beginning of Section 6, the authors wrote that they set $k$ to $2$ (hence two-layered GNNs are in consideration). But Theorems 7 and 8 considered the optimal $k^\ast$ in Corollary 6. Which is correct?
- In Section 3, the authors claimed that the performance of MLPs is relatively more robust to the Gaussian noise in the low-frequency regime compared to the high-frequency regime. Certainly, the decrease in performance at $\sigma = 0.01$ is massive in the high-frequency setting for the CiteSeer dataset. However, it is hard to see this trend in Cora and Pubmed. Therefore, I think it is a little too aggressive to conclude that claim.