Graph Neural Networks (graph NNs) are a promising deep learning approach for analyzing graph-structured data. However, it is known that they do not improve (or sometimes worsen) their predictive performance as we pile up many layers and add non-lineality. To tackle this problem, we investigate the expressive power of graph NNs via their asymptotic behaviors as the layer size tends to infinity. Our strategy is to generalize the forward propagation of a Graph Convolutional Network (GCN), which is a popular graph NN variant, as a specific dynamical system. In the case of a GCN, we show that when its weights satisfy the conditions determined by the spectra of the (augmented) normalized Laplacian, its output exponentially approaches the set of signals that carry information of the connected components and node degrees only for distinguishing nodes. Our theory enables us to relate the expressive power of GCNs with the topological information of the underlying graphs inherent in the graph spectra. To demonstrate this, we characterize the asymptotic behavior of GCNs on the Erdős -Rényi graph. We show that when the Erdős -Rényi graph is sufficiently dense and large, a broad range of GCNs on it suffers from the "information loss" in the limit of infinite layers with high probability. Based on the theory, we provide a principled guideline for weight normalization of graph NNs. We experimentally confirm that the proposed weight scaling enhances the predictive performance of GCNs in real data.
In the paper, the authors carry out theoretically analysis on the expressive power for GNN. The analysis focused on the limiting case when the depth of layers goes to infinite. The authors prove that if the weights of the GNN satisfy certain condition based on the graph Laplacian, then the transformed features contain only degree and connected component information. Then the authors study the G_{np} random graph as a special case. Finally, empirical experiments are carried out to corroborates the theoretical results.
Strength:
1. The authors study a very important problem. It has long been observed that additional depth does not help GNN. The authors provide a convincing theoretical explanation for the behavior.
2. The theoretical results are very close to practical models with little assumptions. Most importantly, the non-linearity is kept compared to other analysis that takes essential components out from the model. Moreover, the technique used in the paper could potentially benefit other theoretical analysis for NN.
3. The authors provide supporting empirical experiments for the theoretical analysis. The observation on s in section 6.3 and the perpendicular component experiment on section 6.4 provide interesting insight to the performance of GNN.
Weakness:
1. The analysis is mostly for dense graphs. However, most of the real-world networks are large-scale sparse networks.
2. It would be great if the authors could provide suggestions on how to eliminate the information loss in GNN. For example, it would be super interesting if the authors can generalize their analysis when residual links exists.