Modeling informal inference in natural language is very challenging. With the recent availability of large annotated data, it has become feasible to train complex models such as neural networks to perform natural language inference (NLI), which have achieved state-of-the-art performance. Although there exist relatively large annotated data, can machines learn all knowledge needed to perform NLI from the data? If not, how can NLI models benefit from external knowledge and how to build NLI models to leverage it? In this paper, we aim to answer these questions by enriching the state-of-the-art neural natural language inference models with external knowledge. We demonstrate that the proposed models with external knowledge further improve the state of the art on the Stanford Natural Language Inference (SNLI) dataset.
Update:
The response addressed all my major concerns, and I think the paper is sound. (I'm updating my confidence to a 5.) So, the paper makes an empirically *very* small step in an interesting line of language understanding research. This paper should be published in some form, but my low-ish score is due simply to my worry that ICLR is not the venue. I think this would be a clear 'accept' as a *ACL short paper, and would probably be viable as a *ACL long paper, but it will definitely have less impact on the overall field of representation learning than will the typical ICLR paper, so I can recommend it only with reservations.
--
This paper presents a method to use external lexical knowledge (word–word relations from WordNet) as an auxiliary input when solving the problem of textual entailment (aka NLI). The idea of accessing outside commonsense knowledge within an end-to-end trained model is one that I expect to be increasingly important in work on language understanding. This paper does not make that much progress on the problem in general—the methods here are quite specific to words and to NLI—and the proposed methods yields only yield large empirical gains in a reduced-data setting, but the paper serves as a well-executed proof of concept. In short, the paper is likely to be of low technical impact, but it's interesting and thorough enough that I lean slightly toward acceptance.
My only concern is on fair comparison: Numbers from this model are compared with numbers from the published ESIM model in several places (Table 3, Figure 2, etc.) as a way to provide evidence for the paper's core claim (that the added knowledge in the proposed model helps). This only constitutes clear evidence if the proposed model is identical to ESIM in all of its unimportant details—word representations, hyperparameter tuning methods, etc. Can the authors comment on this?
For what it's worth, the existence of another paper submission on roughly the same topic with roughly the same results (https://openreview.net/pdf?id=B1twdMCab) makes more confident that the main results in this paper are sound, since they've already been replicated, at least to a coarse approximation.
Minor points:
For TransE, what does this mean:"However, these kind of approaches usually need to train a knowledge-graph embedding beforehand."
You should say more about why you chose the constant 8 in Table 1 (both why you chose to hard code a value, and why that value).
There's a mysterious box above the text 'Figure 1'. Possibly a figure rendering error?
The LSTM equations are quite widely known. I'd encourage you to cite a relevant source and remove them.
Say more about why you choose equation (9). This notably treats all five relation types equally, which seems like a somewhat extreme simplifying assumption.
Equation (15) is confusing. Is a^m a matrix, since it doesn't have an index on it?
What is "early stopping with patience of 7"? Is that meant to mean 7 epochs?
The opening paragraph of 5.1 seems entirely irrelevant, as do the associated results in the results table. I suspect that this might be an opportunity for a gratuitous self-citation.
There are plenty of typos: "... to make it replicatibility purposes."; "Specifically, we use WordNet to measure the semantic relatedness of the *word* in a pair"; etc.