In natural language inference, the semantics of some words do not affect the inference. Such information is considered superficial and brings overfitting. How can we represent and discard such superficial information? In this paper, we use first order logic (FOL) -a classic technique from meaning representation language -to explain what information is superficial for a given sentence pair. Such explanation also suggests two inductive biases according to its properties. We proposed a neural network-based approach that utilizes the two inductive biases. We obtain substantial improvements over extensive experiments.
This paper presents an approach to treat natural language inference using first-order logic, and to infuse neural NLI models with logical information to be more robust at inference. However, the paper does not contain a single reference to the computational semantics literature, where logical approaches towards semantics were the dominant trend for many years (see e.g. [1, 2]). Indeed, 'neuralising' first order logic has been an active area of recent research ([3] or indeed much of the recent work coming from Sebastian Riedel's group). This is a glaring oversight.
The paper starts by introducing background on first-order logic, and then gives a definition of a 'superficial' predicate, namely one whose extension is not necessary to prove an implication for any collection of background facts. However, by extension, this makes s_1 -> s_2 a tautology, which is the 'true' notion that the authors are looking for. Indeed, if |- (s_1 -> s_2), then for any collection of formulae \Delta then \Delta |- (s_1 -> s_2) (by monotonicity of entailment) and clearly if for any \Delta we have \Delta |- (s_1 -> s_2), we can take \Delta to be the empty set. Finally, the authors show that tautologies are still tautologies under change of predicates (i.e. if we only require logical rules to prove one statement from another, then the extensions of predicates in those statements do not matter).
The authors then use this to motivate two extensions to inference models. One is to 'drop out' word information, and the other is to treat different occurrences of the same word as reflecting the same underlying predicate. The first somewhat transparently forces the model to care less about the exact meaning (i.e. extension in the logical world) of words (indeed, word vectors have been shown to capture extensional information [4, 5]), and so may force the inference model to learn more 'logical' inference rules. Further, the word dropout calculation includes whether the word is in both sentences, which is a strong signal that its extension may not be necessary. However, the second only forces the intuition that different mentions of the same word are likely to be coreferent, which is a weak assumption that models may already pick up. Indeed, it is noticeable that this component seems to be less necessary in the authors' ablation study.
In summary, while I am sympathetic to the aim of grounding neural models in explicit notions of semantics, this paper shows such a lack of awareness of previous literature that I cannot recommend acceptance. 
[1] The Meaning Factory: Formal Semantics for Recognizing Textual Entailment and Determining Semantic Similarity, Bjerva et al. 2014
[2] Natural Logic for Textual Inference, MacCartney and Manning 2009
[3] End-to-end Differentiable Proving, Rocktaschel and Riedel 2017
[4] Building a shared world: mapping distributional to model-theoretic semantic spaces, Vecchi and Herbelot 2015
[5] Deriving Boolean structures from distributional vectors, Kreuzewski et al 2015