Recently, progress has been made towards improving relational reasoning in machine learning field. Among existing models, graph neural networks (GNNs) is one of the most effective approaches for multi-hop relational reasoning. In fact, multi-hop relational reasoning is indispensable in many natural language processing tasks such as relation extraction. In this paper, we propose to generate the parameters of graph neural networks (GP-GNNs) according to natural language sentences, which enables GNNs to process relational reasoning on unstructured text inputs. We verify GP-GNNs in relation extraction from text. Experimental results on a human-annotated dataset and two distantly supervised datasets show that our model achieves significant improvements compared to baselines. We also perform a qualitative analysis to demonstrate that our model could discover more accurate relations by multi-hop relational reasoning.
This work proposes a method for parametrising relation matrices in graph neural networks (GNNs) using text. The model is applied in a relation extraction task, and specific dataset subsections are identified to test and analyse “hopping” behaviour: a model’s ability to combine multiple relations for inferring a new one.
Strengths
- strong results
- testing on both bag-level / single-level relation extraction
- Insights via multiple ablations — variation of the number of layers, exploring densely connected data subsections with cycles to identify examples for multi-hop inference
- evaluation on new, human-annotated test set
Issues
- evaluation only on one task, and one dataset (although, with more detail).
- unclear how/why the specific result metrics and their particular range of precision are chosen. Why is F1/Acc only reported for the human-labelled part, but not for the distantly labelled part? Why is precision cut off at ~25% recall? A comprehensive aggregate measure across all levels of recall would be more informative, e.g. PR-AUC., and consistently applied in all experiments.
- task varies from previous versions of the task, posing a potential problem with comparability. What is the motivation behind augmenting the dataset with “NA” labels? Why is the task from previous work altered to predicting relationships between _every_ entity pair? Also unclear: is predicting “NA” actually part of the training loss, and of the evaluation? Is training of the previous “baseline” models adapted accordingly?
- some claims appear too bold and vague — see below.
- comparatively small modelling innovation
- There is similar prior work to this, most prominently Schlichtkrull et al. 2017 [https://arxiv.org/pdf/1703.06103.pdf] who evaluate on Fb15k-237, but also De Cao et al. 2018 [https://arxiv.org/pdf/1808.09920.pdf, published within 1 month before ICLR submission deadline] who evaluate on Wikihop. These previous methods in fact do the following: “Most existing GNNs only process multi-hop relational reasoning on pre-defined graphs and cannot be directly applied in natural language relational reasoning.” It would be good to work out differences to these previous models in more detail, especially to Schlichtkrull et al. 2017.
- unclear how big the specific contribution of the language-generated matrices is. I would normally not obsess about such a baseline, but it seems that the “generate using NL” aspect is a core to the paper (even in the title), and this isn’t worked out clearly.
More comments / questions:
- “multi-hop relational reasoning is indispensable in many natural language processing tasks such as relation extraction”. This isn’t clear to me, “indispensable” is a very strong wording.
- “state-of-the-art baselines”. SOTA defines the best previous work. How can there be several baselines (plural) which are all best? What does SOTA mean when a slight redefinition of the task is proposed, as in this work?
- “Relation extraction from text is a classic natural language relational reasoning task” — reference would be useful. 
- not a big issue, though this does sound somewhat contradictory: 1) “the number of layers K is chosen to be of the order of the graph diameter”. 2) “We treat K as a hyperparameter”
- not clear: is LSTM indeed exactly the same as GP-GNN with K=1? I assume there is a difference, as the LSTM encodes the entire sentence at once, conditioning entity representations on their local context, whereas in GP-GNN this would not be the case.
- The distinction to Context-Aware RE (CARE) is not clear. The authors argue that CARE models co-occurrence of multiple relations, but is this not what a multi-hop relation extraction _should_ learn? It is also not clear how GP-GNN differs in this respect.
- It would be interesting to compare with a model which does not use language to define relation matrices (A), but learns them directly as parameters (independently from the text). 
- It would be interesting to see an analysis of the matrices A_{i,j}. What does the text generate, and how can one find this out?