Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering
In this work, we are interested in generalizing convolutional neural networks (CNNs) from low-dimensional regular grids, where image, video and speech are represented, to high-dimensional irregular domains, such as social networks, brain connectomes or words' embedding, represented by graphs. We present a formulation of CNNs in the context of spectral graph theory, which provides the necessary mathematical background and efficient numerical schemes to design fast localized convolutional filters on graphs. Importantly, the proposed technique offers the same linear computational complexity and constant learning complexity as classical CNNs, while being universal to any graph structure. Experiments on MNIST and 20NEWS demonstrate the ability of this novel deep learning system to learn local, stationary, and compositional features on graphs.
This paper presents an extension of Convolutional Neural Networks to irregular grids in arbitrary dimension, using the properties of the Graph Laplacian Spectrum. The proposed architecture overcomes a computational burden of previous spectral methods, leading to efficient training and evaluation, as demonstrated on image classification and text categorization.
This paper builds upon previous work to propose a computationally efficient model that extends convolutions to irregular graphs. The major innovation is to express graph filtering with a compactly supported kernels as linear combinations of order K polynomials of the graph Laplacian, thus overcoming the burden of computing the Laplacian eigendecomposition, which is needed in both forward and backward passes in the previous spectral construction.  Moreover, the Chebyshev recurrence implies that order K filters can be computed with O(K E) operations, where E is the number of edges and controls the cost of computing L x (L = Laplacian operator on the graph). This appears to be a very clean, efficient parametrization that translates into efficient learning and evaluation and overcomes an important computational limitation.  I have a couple of comments that I would like the authors to develop during the rebuttal phase:   - A previous model that operates in general graphs is the so-called Graph Neural Network (GNN) [Scarselli et al.,'09]. The basic model can be seen as follows. Given a generic vector-valued function x on a graph G, x: G--> R^d, we compute its linear diffusion Wx(i) = \sum_{j neighbor of i} w_{i,j} x(j) and its normalized scaling Dx(i) = (sum_{j neighbor of i} w_{i,j}) x(i). Then a "GNN" layer transforms x into Phi(x): G-->R^{d'} via a generic point-wise operator combining x and Wx: Phi(x)(i) = Psi( Dx(i), Wx(i)), with Psi being a generic pointwise operator: Psi: R^d x R^d --> R^{d'}. For example, if Psi(x,y) = x - y we obtain the Graph Laplacian Phi(x) = L(x), and thus the order-K graph filtering can be realized with a K-layer GNN using linear pointwise operators Psi. The model which computes filters as linear combinations of powers of graph laplacians can thus be seen as a special case of the previous model, which interleaves linear layers Psi (responsible of computing the powers of the laplacian) with nonlinear layers. It would be interesting for the authors to comment on the pros/cons of their model with respect to the general GNN.   - The numerical experiments in MNIST show good performance of the graph CNN, but also a gap with respect to the classic CNN architectures. One can thus ask whether this gap is simply due to model tuning or due to some intrinsic limitations of graph CNNs when used on regular, low-dimensional grids. The filters learnt by the proposed graph CNN are polynomials of the Graph Laplacian, and therefore are rotationally invariant, whereas in the grid the Euclidean embedding allows for directional filters, which bring a crucial advantage to capture geometric structures in images. Is this analysis correct? In other words, there seems to be a fundamental price to pay for giving up the knowledge of the grid, its translation invariance and the notion of orientation. Obviously this only concerns the examples where the grid is available.   Overall, and despite my previous two comments, this paper brings an important improvement to the table, that can extend the use of data driven, end-to-end learning with excellent learning complexity to other datasets beyond images, speech and text. I hope the rebuttal addresses my previous comments though!