Bayesian Semi-supervised Learning with Graph Gaussian Processes
We propose a data-efficient Gaussian process-based Bayesian approach to the semisupervised learning problem on graphs. The proposed model shows extremely competitive performance when compared to the state-of-the-art graph neural networks on semi-supervised learning benchmark experiments, and outperforms the neural networks in active learning experiments where labels are scarce. Furthermore, the model does not require a validation data set for early stopping to control over-fitting. Our model can be viewed as an instance of empirical distribution regression weighted locally by network connectivity. We further motivate the intuitive construction of the model with a Bayesian linear model interpretation where the node features are filtered by an operator related to the graph Laplacian. The method can be easily implemented by adapting off-the-shelf scalable variational inference algorithms for Gaussian processes.
## [Updated after author feedback] Thank you for the feedback. The explanation in L32-L34 is very helpful. I hope you will add it to the main paper. I am happy that figure 2 now has error bars and the note on the small errors on the SOPT-* models is informative. It is good to see that the SOPT-* results are indeed significantly different. I hope, however, that you will also update table 1 in the main paper to show the standard deviations on the 10 random trials.  ## Summary The paper presents a Gaussian process-based model for semi-supervised learning on graphs (GGP). The model consists of a novel covariance function, which considers nodes in the 1-hop neighbourhood. The authors further present two distinct views on this new covariance function, namely that of a kernel mean embedding and that of a Bayesian linear model relating to the graph Laplacian. The latter is remarked as being particularly interesting since it can be viewed in terms of a spectral filter, which can be designed to have attractive properties. Finally, the authors cast the model in a variational inference framework with inducing inputs, thus allowing the model to scale to large graphs.  ## Quality The paper is of good technical quality. The authors provide discussions and explanations along the way, providing good insights for the reader.  The experiments section show a very fair comparison with graph convolutional neural networks by replicating an experiment where these are known to work well. The GGP model is thus compared to nine other methods in a semi-supervised setting. While I like this set-up, I would prefer to have uncertainties on the classification accuracies. There are several competitive methods, all showing promising accuracies, and it is not clear how significant these results are. An additional, interesting experiment shows the method's performance in an active learning setting. This is an interesting experiment, but I would again like to see errorbars on the plots in figure 2 corresponding to the standard deviation over the 10 trials performed.  ## Clarity The paper reads well and is easy to follow. The structure makes sense, and figure 1 is very helpful in understanding the method. One section that confused me, however, is lines 156-163. While I understand what you are saying, I am not sure what the implications for your model should be.  ## Originality As the authors mention in the (quite extensive) related work section, GPs on graphs have been studied before. The novelty lies in the covariance function, which is simple, yet shown to be effective. By presenting two different views on the developed covariance function, the authors further define interesting directions for future research, which is valuable information for the community.  ## Significance I think this paper constitutes a nice addition to the literature on graph Gaussian processes. The proposed covariance function is easy to interpret, yet powerful, and the authors point out interesting research directions. While not groundbreaking novel, it is solid paper that fits NIPS.