This paper presents a new Graph Neural Network (GNN) type using feature-wise linear modulation (FiLM). Many standard GNN variants propagate information along the edges of a graph by computing "messages" based only on the representation of the source of each edge. In GNN-FiLM, the representation of the target node of an edge is additionally used to compute a transformation that can be applied to all incoming messages, allowing feature-wise modulation of the passed information.
Results of experiments comparing different GNN architectures on three tasks from the literature are presented, based on re-implementations of baseline methods. Hyperparameters for all methods were found using extensive search, yielding somewhat surprising results: differences between baseline models are smaller than reported in the literature. Nonetheless, GNN-FiLM outperforms baseline methods on a regression task on molecular graphs and performs competitively on other tasks.
The paper proposes a new Graph Neural Network (GNN) architecture that uses Feature-wise Linear Modulation (FiLM) to condition the source-to-target node message-passing based on the target node representation. In this way, GNN-FiLM aims to allow a GNN's message propagation to "focus on feature that are especially relevant for the update of the target node." The authors clearly describe prior GNN architectures, showing that the do not incorporate such forms of message propagation. The authors then describe several intuitive ways of adding such a form of message propagation, before describing why those approaches do not work in practice. Finally, the authors introduce GNN-FiLM, which is computationally reasonable and works well in practice, as evaluated according to several GNN benchmarks. The GNN-FiLM model is also quite simple and elegant, which makes me think it is likely to work on more tasks than the authors experiment on.
The paper is clear and easy to follow. The authors' description of other GNNs architectures is clear, and their own approach seems well motivated and clearly described in relation to previous GNNs.
The authors have released the code for method, where they have also reimplemented several popular GNN methods. The open-source codebase also seems to be valuable contribution for future research and reproducibility in work on GNNs.
The empirical evaluation seems thorough. GNN-FiLM works well on 3 graph tasks (PPI, QM9, VarMisuse) with different properties. To compare models, the authors conduct a search of hyperparameter ranges for each model. The authors even improve several of the baseline methods, generalizing some approaches to include different edge types and to add self-loops, as well as using better networks (adding dropout and increasing hidden dimensions). The paper even finds that one existing/obvious GNN architecture (GNN-MLP) is underrated. The paper reads like an honest analysis of existing methods, even though it also introduces its own, new method that works better.
Questions:
* Do you have any intuition about why the Eqn. 5 model is less stable than GNN-FiLM?
* On GNN-FiLM's training stability and regularization: I am interested in the negative result described in the following sentence: "Preliminary experiments on the citation network data showed results that were at best comparable to the baseline methods, but changes of a random seed led to substantial fluctuations (mirroring the problems with evaluation on these tasks reported by Shchur et al. (2018))" Do the authors have any intuition about why the results are highly dependent on the random seed? The results on other tasks seems to not have much variance. It could be interesting to try various techniques to stabilize learning on that task. A simple approach like gradient clipping might work, or there are perhaps other techniques. For example, in "TADAM: Task dependent adaptive metric for improved few-shot learning", the authors find it important to regularize FiLM parameters \gamma and \beta towards 1 and 0, respectively, which may make learning more stable here. In general, previous work seems to find it important to regularize the parameters that predict FiLM parameters, which may also fix GNN-FiLM's overfitting on VarMisuse. Exploring such approaches could make it easier for future work to use FiLM with GNNs.
* On potential related work: GNN-FiLM is a "self-conditioned" model which learns to apply feature-wise transformations based on the activations at the current layer. If I am not mistaken, the self-conditioning aspect of GNN-FiLM makes it related to the self-conditioned models described in "Feature-wise Transformations" ([Feature-wise transformations](https://distill.pub/2018/feature-wise-transformations/)) - for example, see the "Image Recognition" section (Squeeze-and-Excitation Networks and Highway Networks) and the "Natural Language Processing" section (LSTMs, gated linear units, and gated-attention reader). Do the authors see a connection with such models? If so, it would be interesting to hear these works discussed (in the rebuttal and paper) and the exact connection described.