Reasoning over text and Knowledge Bases (KBs) is a major challenge for Artificial Intelligence, with applications in machine reading, dialogue, and question answering. Transducing text to logical forms which can be operated on is a brittle and error-prone process. Operating directly on text by jointly learning representations and transformations thereof by means of neural architectures that lack the ability to learn and exploit general rules can be very data-inefficient and not generalise correctly. These issues are addressed by Neural Theorem Provers (NTPs) , neuro-symbolic systems based on a continuous relaxation of Prolog's backward chaining algorithm, where symbolic unification between atoms is replaced by a differentiable operator computing the similarity between their embedding representations. In this paper, we first propose Neighbourhoodapproximated Neural Theorem Provers (NaNTPs) consisting of two extensions to NTPs, namely a) a method for drastically reducing the previously prohibitive time and space complexity during inference and learning, and b) an attention mechanism for improving the rule learning process, deeming them usable on real-world datasets. Then, we propose a novel approach for jointly reasoning over KB facts and textual mentions, by jointly embedding them in a shared embedding space. The proposed method is able to extract rules and provide explanations-involving both textual patterns and KB relations-from large KBs and text corpora. We show that NaNTPs perform on par with NTPs at a fraction of a cost, and can achieve competitive link prediction results on challenging large-scale datasets, including WN18, WN18RR, and FB15k-237 (with and without textual mentions) while being able to provide explanations for each prediction and extract interpretable rules.
[Summary]
This paper scales NTPs by using approximate nearest neighbour search over facts and rules during unification. Additionally, the paper incorporates mentions as additional facts where the predicate is the text that the entities of the mention are contained in. The paper also suggests parameterizing predicates using attention over known predicates. The increments presented are reasonable and justified, but the experimental results, specifically on the larger datasets, warrant further investigation.
[Pros]
- Reasonable and interesting increments on top of NTP.
- Scaling the approach to larger datasets is well motivated.
- Utilizing text is an interesting direction for NTP in terms of integrating it with past work on KG completion.
[Cons]
- Empirical performance on larger datasets needs further investigation.
- No ablation study is performed so the effect of incorporating mentions and attention are unclear.
- Baseline performance on FB15k-237 seems weak compared to the original papers as well as more recent papers re-examining baselines for KG completion (http://aclweb.org/anthology/W17-2609). Is this due to the d=100 restriction, or were pretrained embeddings not used? Without further explanation, the claim that scores are competitive with SOTA seems unjustified, at least for FB15k-237 since the model performs significantly worse than the baselines which seem to be worse than previously reported.
[Comments]
- For reproducibility: it is unclear whether evaluation in FB15k-237 is carried out on the KB+Text, KB, or Text portions of the dataset.
[Overall]
It’s great that NTP was scaled up to handle larger datasets, however further analysis is needed. The argument that performance is given up for interpretability needs more discussion, and the effect of each addition to the system should be discussed as well.