We introduce Uncertain Natural Language Inference (UNLI), a refinement of Natural Language Inference (NLI) that shifts away from categorical labels, targeting instead the direct prediction of subjective probability assessments. We demonstrate the feasibility of collecting annotations for UNLI by relabeling a portion of the SNLI dataset under a probabilistic scale, where items even with the same categorical label differ in how likely people judge them to be true given a premise. We describe a direct scalar regression modeling approach, and find that existing categorically labeled NLI data can be used in pre-training. Our best models approach human performance, demonstrating models may be capable of more subtle inferences than the categorical bin assignment employed in current NLI tasks. Variants of entailment tasks have been used for decades in benchmarking systems for natural language understanding. Recognizing Textual Entailment (RTE) or Natural Language Inference (NLI) is traditionally a categorical classification problem: predict which of a set of discrete labels apply to an inference pair, consisting of a premise (  ) and hypothesis (  ). The FraCaS consortium offered the task as an evaluation mechanism, along with a small challenge set (Cooper et al., 1996), which was followed by the RTE challenges (Dagan et al., 2005). Despite differences between these and recent NLI datasets (Marelli et al., 2014; Lai et al., 2017; Williams et al., 2018; Khot et al., 2018, i.a. ), NLI hsa remained a categorical prediction problem. However, entailment inference is uncertain and has a probabilistic nature (Glickman et al., 2005). Maintaining NLI as a categorical classification  Equal contribution. problem is not ideal since coarse categorical labels mask the uncertain and probabilistic nature of entailment inference. NLI pairs may share a coarse label, but the probabilities that the hypotheses are entailed by their corresponding premises may vary greatly (see Table 1). Hence, not all contradictions are equally contradictory and not all entailments are equally entailed . We propose Uncertain Natural Language Inference (UNLI), a refinement of NLI that captures more subtle distinctions in meaning by shifting away from categorical labels to the direct prediction of human subjective probability assessments. We illustrate that human-elicited probability assessments contain subtle distinctions on the likelihood of a hypothesis conditioned on a premise, and UNLI captures these distinctions far beyond categorical labels in popular NLI datasets. We demonstrate how to elicit UNLI annotations. Using recent large-scale language model pre-training, we provide experimental results illustrating that systems can often predict UNLI judgments, but with clear gaps in understanding. We conclude that scalar annotation protocols should be adopted in future NLI-style dataset creation, which should enable new work in modeling a richer space of interesting inferences. We elicit subjective probabilities from crowdsource workers (MTurk) for premise-hypothesis pairs from existing NLI data. Annotators are asked to estimate how likely the situation described in the hypothesis sentence would be true given the premise. Following the Efficient Annotation of Scalar Labels framework (EASL; Sakaguchi and Durme, 2018), we present annotators 5 sentence-pairs, each with a slider bar enabling direct assessment for each pair and ask annotators to calibrate their score for a sentence-pair based on the scores they provided to the other four pairs. 1 In contrast to the uniform scale employed in the original EASL protocol, we modify the interface to allow finer-grained values near 0.0 and 1.0, following psychological findings that humans are especially sensitive to values near the ends of the probability spectrum (Tversky and Kahneman, 1981). 2 This interface decision is a key distinction of this work contrasting prior efforts that averaged Likert-scale (ordinal) annotations. This allows us to capture the difference between NLI pairs that are both appropriately contradicted or entailed under NLI, but that have a perceived difference of less than 1% probability. In order to capture the sensitivity near these ends, we adopt a more fine-grained slider bar with 10,000 steps with a logistic transformation. Specifically, for raw score   [ 0 , 10000 ] , we apply a scaled logistic function  (  ) =  (  (   5000 )) to re-scale the final result range to [ 0 , 1 ] . We ran pilots to tune  , and determine that people tend to choose much lower probability for some events even though they are just slightly less likely (e.g., just below 50%). 3 1 Example pairs were provided in the instructions along with suggested probability values. See Appendix A for details of the annotation interface and qualifications. 2 This is called the certainty effect : more sensitivity to the difference between, e.g., 0% and 1% than 50% and 51%. 3 This phenomenon accords with the weighting function in Prospect Theory (Kahneman and Tversky, 1979; Tversky and Kahneman, 1992), where people tend to downweight probabilities with around 0.4 or above. Therefore, we use different  's depending on the range of [ 0 , 0 . 5 ] or ( 0 . 5 , 1 ] . Each sentence pair is annotated with 2or 3-way redundancy. The individual responses are averaged to create a gold standard label for a premise-hypothesis pair. Data We annotate, i.e. elicit a probability   [ 0 , 1 ] , for a subset of SNLI (Bowman et al., 2015) examples and refer to this data as u -SNLI. 4 SNLI's training set contains 7,931 distinct premises paired with at least 5 distinct neutral ( NEU ) hypotheses. For each premise, we sample 5 neutral hypotheses, resulting in 39,655 of these NEU pairs annotated. An additional 15,862 contradicted ( CON ) and entailed ( ENT ) pairs are annotated for our training set, resulting in 55,517 training examples. For our dev and test sets, we respectively annotated 3,040 examples sampled from SNLI's dev and test splits. In total, we annotated 61,597 examples, about 12% of all examples in SNLI. Figure 1 plots the resultant median and quartile for each categorical SNLI label in the u -SNLI dev set, showing the wide range of probability judgments elicited for each label (see Table 2 for examples). 5 3 Prediction Formally, given a premise   P and a hypothesis   H , a UNLI model  : P  H  [ 0 , 1 ] should output an uncertainty score    [ 0 , 1 ] of the 4 We use SNLI due to its popularity and its feature that each premise is paired with multiple hypotheses. premise-hypothesis pair that correlates well with a human-provided subjective probability assessment. We train a regression UNLI model to predict the probability that a premise entails a hypothesis. We modify the sentence pair classifier 6 in BERT to exploit recent advancements in large-scale language model pre-training. Following Devlin et al. (2019), we concatenate the premise and the hypothesis, with a special sentinel token ( CLS ) inserted at the beginning and a separator ( SEP ) inserted after each sentence, tokenized using WordPiece. After encoding the concatenated token sequence with BERT, we take the encoding of the first sentinel token. We pass the resulting feature vector f ( ,  ) through a sigmoid-activated linear layer to obtain a probability, instead of a softmax used in categorical NLI. We directly model UNLI as a regression problem, trained using a binary cross-entropy loss 7 between the human annotation  and the model output   . Owing to the concerns raised with annotation artifacts in SNLI (Gururangan et al., 2018; Tsuchiya, 2018; Poliak et al., 2018), we include a hypothesis-only baseline . 8 Metrics We compute Pearson correlation (  ), the Spearman rank correlation (  ), and the mean square error (MSE) between y and   as the metrics to measure the to performance of UNLI models. Pearson  measures the linear correlation between the gold probability assessments and model's output; Spearman  measures the ability of the model ranking the premise-hypothesis pairs with 6 The neural architecture for MultiNLI (Williams et al., 2018) in Devlin et al. (2019). respect to their subjective probability; MSE measures whether the model can recover the subjective probability value from premise-hypothesis pairs. A high  and  , but a low MSE is desired. Table 4 reports results on u -SNLI dev and test sets. Just training on 55 , 517 u -SNLI examples yields a 62.71% Pearson  on test. The hypothesis-only baseline achieved a correlation around 40%. This result corroborates the findings that a hidden bias exists in the SNLI dataset's hypotheses, and shows this bias may also exist in u -SNLI. 9 Hyp-only Full-model Dev Test Dev Test  0.3759 0.4120 0.6383 0.6271  0.3853 0.4165 0.6408 0.6346 MSE 0.1086 0.1055 0.0751 0.0777 Table 4: Metrics for training on u -SNLI. Human Performance We elicit additional annotations on u -SNLI dev set to establish a randomly sampled human performance. We use the same annotators as before but ensure each annotator has not previously seen the pair they are annotating. We average the scores from three-way redundant elicitation, 10 yielding  = 0 . 6978 ,  = 0 . 7273 , and MSE = 0 . 0759 : our regression model trained on u SNLI is therefore approaching human performance. While encouraging, the model fails drastically for some examples. 9 This is unsurprising because u -SNLI examples are sampled from SNLI. 10 This setting approximates the performance of a randomly sampled human on u -SNLI, and is therefore a reasonable lower bound on the performance one could achieve with a dedicated, trained single human annotator. Qualitative Error Analysis Table 3 illustrates examples with large gaps between the gold probability assessment and the BERT-based model output. The model seems to have learned lexicon-level inference (e.g., race cars (cid:123) going fast , but ignored crucial information ( sits in the pits ), and fails to learn certain commonsense patterns (e.g. riding amusement park ride (cid:123) screaming ; man and woman drinking at a bar (cid:123) on a date ). These examples illustrate the model's insufficient commonsense reasoning and plausibility estimation. Pre-training with SNLI Can we leverage the remaining roughly 500,000 SNLI training pairs that only have categorical labels? One method would be to train a categorical NLI model on SNLI and when fine-tuning on u -SNLI, replace the last layer of the network from a categorical prediction with a sigmoid function. 11 However, a typical categorical loss function would not take into account the ordering between the different categorical labels. 12 Instead, we derive a surrogate function  : T  [ 0 , 1 ] that maps SNLI categorical labels   { ENT , NEU , CON } to the average score of all u -SNLI training annotations labeled with  in SNLI. 13 SNLI SNLI + u -SNLI Dev Test Dev Test  0.5198 0.4958 0.6762 0.6589  0.5238 0.5231 0.6806 0.6708 MSE 0.1086 0.0928 0.0694 0.0733 Table 5: Metrics for training only on mapped SNLI or fine-tuning on u -SNLI. We use this mapping to pre-train a regression model on the SNLI training examples not included in u -SNLI. We also fine-tune the model on u SNLI's training set. Table 5 reports the results evaluated on u -SNLI's dev and test sets. The model trained on the roughly 500  mapped SNLI examples, performs much worse than when trained on just about 55  u -SNLI examples. When we pretrain the model on the mapped SNLI and fine-tune on u -SNLI, results noticeably improve. This improvement is akin to the Phang et al. (2018)'s find-ing that many NLI datasets cover informative signal 11 This is similar to how Pavlick and Callison-Burch (2016) pre-train on SNLI, then fine-tune the model using their AddOne pairs. Model behavior Figure 2 depicts the model behavior when training just on SNLI or fine-tuning with u -SNLI. When using the original SNLI data, under the surrogate regression setting, the model's prediction concentrates on the 3 surrogate scalar values of the 3 SNLI classes. After fine-tuning on u -SNLI, the model learns smoother predictions for premise-hypothesis pairs, supported by the supe-rior Pearson correlation score. The darker boxes in bottom-right corner of the heatmaps (Figure 2) indicate high accuracy on samples with  1 . 0 gold u -SNLI labels and  1 . 0 model predictions, signifying that our UNLI models are very good at recognizing entailments. The probabilistic nature and the uncertainty of NLI has been considered from a variety of perspectives. Glickman et al. (2005) modified the task to explicitly include the probabilistic aspect of NLI, stating that   probabilistically entails  ... if  increases the likelihood of  being true, while Lai and Hockenmaier (2017) noted how predicting the conditional probability of one phrase given another would be helpful in predicting textual entailment. Other prior work has elicited ordinal annotations (e.g. Likert scale) reflecting likelihood judgments (Pavlick and Callison-Burch, 2016; Zhang et al., 2017), but then collapsed the annotations into coarse categorical labels for modeling. Vulic et al. (2017) proposed graded lexical entailment , which is similar to our idea but applied to lexical-level inference, asking to what degree  is a type of  . Additionally, Lalor et al. (2016, 2018) tried capturing the uncertainty of each inference pair by item response theory (IRT), showing fine-grained differences in discriminative power in each label. Pavlick and Kwiatkowski (2019) recently argued that models should  explicitly capture the full distribution of plausible human judgments  as plausible human judgments cause inherent disagreements. Our concern is different as we are interested in the uncertain and probabilistic nature of NLI. We are the first to propose a method for direct elicitation of subjective probability judgments on NLI pairs and direct prediction of these scalars, as opposed to reducing to categorical classification. Recent work have also modeled the uncertainty of other semantic phenomena as direct scalar regression (and collected scalar versions of data for them) instead of categorical classification, e.g. factuality (Lee et al., 2015; Stanovsky et al., 2017; Rudinger et al., 2018), and semantic proto-roles (Teichert et al., 2017). Plausiblity tasks such as COPA (Roemmele et al., 2011) and ROCStories (Mostafazadeh et al., 2016) ask models to choose the most probable examples given a context, capturing relative uncertainty between examples, but do not force a model to predict the probability of  given  . Li et al. (2019) viewed the plausibility task of COPA as a learning to rank problem, where the model is trained to assign the highest scalar score to the most plausible alternative given context. Our work can be viewed as a variant to this, with the score being an explicit human probability judgment instead. Linguists such as van Eijck and Lappin (2014), Goodman and Lassiter (2015), Cooper et al. (2015) and Bernardy et al. (2018) have described models for natural language semantics that introduce probabilities into the compositional, model-theoretic tradition begun by those such as Davidson (1967) and Montague (1973). Where they propose probabilistic models for interpreting language, we are concerned with illustrating the feasibility of eliciting probabilistic judgments on examples through crowdsourcing, and contrasting with prior efforts restricted to limited categorical label sets. We proposed Uncertain Natural Language Inference (UNLI), a new task of directly predicting human likelihood judgments on NLI premise-hypothesis pairs. In short, we have shown that not all NLI contradictions are created equal, nor neutrals, nor entailments. We demonstrated that (1) eliciting supporting data is feasible, and (2) annotations in the data can be used for improving a scalar regression model beyond the information contained in existing categorical labels, using recent contex-tualized word embeddings, e.g. BERT. Humans are able to make finer distinctions between meanings than is being captured by current annotation approaches; we advocate the community strives for systems that can do the same, and therefore shift away from categorical NLI labels and move to something more fine-grained such as our UNLI protocol. We thank anonymous reviewers from current and past versions of the article for their insightful comments and suggestions. This research bene-fited from support by DARPA AIDA and DARPA LORELEI. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes. The views and conclusions contained in this publication are those of the authors and should not be interpreted as representing official policies or endorsements of DARPA or the U.S. Government.
abstain objective result method abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain objective result objective abstain abstain method abstain abstain abstain method abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain method method method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain other other other abstain other other abstain objective other other other abstain other objective objective result objective abstain other other other other