Symbolic logic allows practitioners to build systems that perform rule-based reasoning which is interpretable and which can easily be augmented with prior knowledge. However, such systems are traditionally difficult to apply to problems involving natural language due to the large linguistic variability of language. Currently, most work in natural language processing focuses on neural networks which learn distributed representations of words and their composition, thereby performing well in the presence of large linguistic variability. We propose to reap the benefits of both approaches by applying a combination of neural networks and logic programming to natural language question answering. We propose to employ an external, nondifferentiable Prolog prover which utilizes a similarity function over pretrained sentence encoders. We fine-tune these representations via Evolution Strategies with the goal of multi-hop reasoning on natural language. This allows us to create a system that can apply rule-based reasoning to natural language and induce domainspecific natural language rules from training data. We evaluate the proposed system on two different question answering tasks, showing that it complements two very strong baselines -BIDAF (Seo et al., 2016a) and FASTQA (Weissenborn et al., 2017) -and outperforms both when used in an ensemble.
UPDATE: Given the authors' rebuttal and the clear improvements to their paper, I've increased my rating of the work.
=======================
This paper presents NLProlog, a method that combines logical rules with distributed representations for reasoning on natural language statements. Natural language statements (first converted to logical triples) and templated logical rules are embedded in a vector space using a pretrained sentence encoder. These embedded "symbols" can be compared in vector space, and their similarity used in a theorem prover (Prolog) modified to support weak unification. The theorem prover determines the answer to a natural language query by constructing a proof according to its logical rules.
Training through the non-differentiable theorem prover occurs via an "evolutionary strategy," which enables the model to fine-tune its sentence encoders and learn domain-specific logic rules directly from text. The authors also propose a Gradual Rule Learning (GRL) algorithm that seems necessary for the optimization process to converge on good solutions.
Despite the model's complexity, the paper was fairly clear to me.
Although the proposed model is a conglomeration of pre-existing parts, the combination is original to my knowledge. The use of Open Information Extraction to transform natural language statements to logical statements, which amenable to theorem provers, is novel and also circumvents the complicated preprocessing required by previous related works.
The authors evaluate the proposed approach on subsets of the Wikihop dataset and BABI-1k. NLProlog performs competitively with neural models, similarly augmented with Sent2Vec but lacking explicit logical rules, only on the 'country' subset of Wikihop. It does not compete with or clearly outperform these models in general. As the authors state, it further "struggles to find meaningful rules for the predicates 'developer' and 'publisher'."
NLProlog demonstrates strong performance on a subset of problems labelled unanimously by annotators to require multi-hop reasoning. Unfortunately, this is only done for the "country" subset of Wikihop, on which the model was already shown to have the strongest performance. I'd find this more convincing if similar improvements were shown on the other subsets (publisher, developer).
Taking into account also that the BABI subset was used only for ablation, the limited results call into question the significance of the work. It would definitely benefit from more extensive experimental validation. On the other hand, it's very positive to see that NLProlog seems to succeed where the neural models fail, and vice versa, so that the two approaches can be combined in an ensemble to achieve state-of-the-art results. This suggests that the paper's line of research has something to add to the community and should be pursued further. I'd find this result more interesting if an error analysis elucidated some characteristics of the examples that each approach does well/poorly on.
I'd like to see more analysis in general, that answers questions including:
- How reliable is the Open IE system and how does its performance impact the end task?
- How well-specified must the a priori rule structures be to achieve good performance? Further, how does the number and structure of the rules (a hyperparameter in this work) affect performance?
- What is the run-time/complexity of the exhaustive proof search during training?
- Relatedly, you state that you limit the rule complexity to two body atoms in the rule templates for BABI. Can you estimate what rule complexity is required in the Wikihop tasks?
I would like to recommend this work more confidently because it tackles such an important problem and does so in an interesting, well-conceived way. My reluctance arises from the limited experimental validation and analysis. Given more analysis details and experimental evidence from the authors, I'm happy to raise my recommendation.
Pros:
- the method complements standard deep QA models to achieve state-of-the-art results in an ensemble.
- unifying neural representations with logical/symbolic formalisms is an important research direction.
- a code release is planned.
Cons:
- a very complex model, whose details are occasionally unclear; the algorithms in Appendix A are helpful but they are not in the main text.
- the model expresses only a limited subset of first order logic; dynamically changing world states are not supported (yet).
- limited experimental validation.
- it's good to be able to incorporate prior knowledge, but it seems like it's quite necessary to pre-specify rules (in template form).
Minor quibble: Evolutionary learning strategies, such as genetic algorithms, go back a long way. It's strange using only a reference from 2017 to introduce them.