We present two neural models for event factuality prediction, which yield significant performance gains over previous models on three event factuality datasets: FactBank, UW, and MEANTIME. We also present a substantial expansion of the It Happened portion of the Universal Decompositional Semantics dataset, yielding the largest event factuality dataset to date. We report model results on this extended factuality dataset as well. A central function of natural language is to convey information about the properties of events. Perhaps the most fundamental of these properties is factuality : whether an event happened or not. A natural language understanding system's ability to accurately predict event factuality is important for supporting downstream inferences that are based on those events. For instance, if we aim to construct a knowledge base of events and their participants, it is crucial that we know which events to include and which ones not to. The event factuality prediction task (EFP) involves labeling event-denoting phrases (or their heads) with the (non)factuality of the events denoted by those phrases (Saur and Pustejovsky, 2009, 2012; de Marneffe et al., 2012). Figure 1 exemplifies such an annotation for the phrase headed by leave in (1), which denotes a factual event (  =factual, (cid:9) =nonfactual). In this paper, we present two neural models of event factuality (and several variants thereof). We show that these models significantly outperform previous systems on four existing event factuality datasets  FactBank (Saur and Pustejovsky, 2009), the UW dataset (Lee et al., 2015), MEANTIME (Minard et al., 2016), and Universal De-failed Jo leave to trace no . compositional Semantics It Happened v1 (UDS-IH1; White et al., 2016)  and we demonstrate the efficacy of multi-task training and ensembling in this setting. In addition, we collect and release an extension of the UDS-IH1 dataset, which we refer to as UDS-IH2, to cover the entirety of the English Universal Dependencies v1.2 (EUD1.2) treebank (Nivre et al., 2015), thereby yielding the largest event factuality dataset to date. 1 We begin with theoretical motivation for the models we propose as well as discussion of prior EFP datasets and systems (  2). We then describe our own extension of the UDS-IH1 dataset (  3), followed by our neural models (  4). Using the data we collect, along with the existing datasets, we evaluate our models (  6) in five experimental settings (  5) and analyze the results (  7). Words from effectively every syntactic category can convey information about the factuality of an event. For instance, negation (2a), modal auxiliaries (2b), determiners (2c), adverbs (2d), verbs (2e), adjectives (2f), and nouns (2g) can all con-1 Data available at decomp.net. vey that a particular event  in the case of (2), leaving event  did not happen. (2) a. Jo didn't leave . b. Jo might leave . c. Jo left no trace. d. Jo never left . e. Jo failed to leave . f. Jo's leaving was fake. g. Jo's leaving was a hallucination. Further, such words can interact to yield nontrivial effects on factuality inferences: (3a) conveys that the leaving didn't happen, while the su-perficially similar (3b) does not. A main goal of many theoretical treatments of factuality is to explain why these sorts of interactions occur and how to predict them. It is not possible to cover all the relevant literature in depth, and so we focus instead on the broader kind of interactions our models need to be able to capture in order to correctly predict the factuality of an event denoted by a particular predicatenamely, interactions between that predicate's outside and inside context, exemplified in Figure 1. Outside context Factuality information coming from the outside context is well-studied in the domain of clause-embedding predicates, which break into at least four categories: factives, like know and love (Kiparsky and Kiparsky, 1970; Karttunen, 1971b; Hintikka, 1975); implicatives, like manage and fail (Karttunen, 1971a, 2012, 2013; Karttunen et al., 2014), veridicals, like prove and verify (Egre, 2008; Spector and Egre, 2015), and non-veridicals, like hope and want . Consider the factive-implicative verb forget (Karttunen, 1971a; White, 2014). (4) a. Jo forgot that Bo left . b. Jo forgot to leave . (cid:9) (5) a. Jo didn't forget that Bo left . b. Jo didn't forget to leave . When a predicate directly embedded by forget is tensed, as in (4a) and (5a), we infer that that predicate denotes a factual event, regardless of whether forget is negated. In contrast, when a predicate directly embedded by forget is untensed, as in (4b) and (5b), our inference is dependent on whether forget is negated. Thus, any model that correctly predicts factuality will need to not only be able to represent the effect of individual words in the outside context on factuality inferences, it will furthermore need to represent their interaction. Inside context Knowledge of the inside context is important for integrating factuality information coming from a predicate's argumentse.g. from determiners, like some and no . In simple monoclausal sentences like those in (6), the number of arguments that contain a negative quantifier, like no , determine the factuality of the event denoted by the verb. An even number (or zero) will yield a factuality inference and an odd number will yield a nonfactuality inference. Thus, as for outside context, any model that correctly predicts factuality will need to integrate interactions between words in the inside context. The (non)necessity of syntactic information One question that arises in the context of inside and outside information is whether syntactic information is strictly necessary for capturing the relevant interactions between the two. To what extent is linear precedence sufficient for accurately computing factuality? We address these questions using two bidirectional LSTMsone that has a linear chain topology and another that has a dependency tree topology. Both networks capture context on either side of an event-denoting word, but each does it in a different way, depending on its topology. We show below that, while both networks outperform previous models that rely on deterministic rules and/or hand-engineered features, the linear chain-structured network reliably outperforms the tree-structured network. Saur and Pustejovsky (2009) present the FactBank corpus of event factuality annotations, built on top of the TimeBank corpus (Pustejovsky et al., 2006). These annotations (performed by trained annotators) are discrete, consisting of an epistemic modal { certain , probable , possible } and a polarity { + , } . In FactBank, factuality judgments are with respect to a source ; following recent work, here we consider only judgments with respect to a single source: the author. The smaller MEANTIME corpus (Minard et al., 2016) includes sim-732 Dataset Train Dev Test Total FactBank 6636 2462 663 9761 MEANTIME 967 210 218 1395 UW 9422 3358 864 13644 UDS-IH2 22108 2642 2539 27289 Table 1: Number of annotated predicates. ilar discrete factuality annotations. de Marneffe et al. (2012) re-annotate a portion of FactBank using crowd-sourced ordinal judgments to capture pragmatic effects on readers' factuality judgments. Lee et al. (2015) construct an event factuality dataset  henceforth, UW  on the TempEval-3 data (UzZaman et al., 2013) using crowdsourced annotations on a [  3 , 3] scale ( certainly did not happen to certainly did ), with over 13,000 predicates. Adopting the [  3 , 3] scale of Lee et al. (2015), Stanovsky et al. (2017) assemble a Unified Factuality dataset, mapping the discrete annotations of both FactBank and MEANTIME onto the UW scale. Each scalar annotation corresponds to a token representing the event, and each sentence may have more than one annotated token. The UDS-IH1 dataset (White et al., 2016) consists of factuality annotations over 6,920 event tokens, obtained with another crowdsourcing protocol. We adopt this protocol, described in  3, to collect roughly triple this number of annotations. We train and evaluate our factuality prediction models on this new dataset, UDS-IH2, as well as the unified versions of UW, FactBank, and MEANTIME. Table 1 shows the number of annotated predicates in each split of each factuality dataset used in this paper. Annotations relevant to event factuality and polarity appear in a number of other resources, including the Penn Discourse Treebank (Prasad et al., 2008), MPQA Opinion Corpus (Wiebe and Riloff, 2005), the LU corpus of author belief commitments (Diab et al., 2009), and the ACE and ERE formalisms. Soni et al. (2014) annotate Twitter data for factuality. Nairn et al. (2006) propose a deterministic algorithm based on hand-engineered lexical features for determining event factuality. They associate certain clause-embedding verbs with implication signatures (Table 2), which are used in a recursive polarity propagation algorithm. TruthTeller is also a recursive rule-based system for factuality (predicate truth) prediction using implication signatures, as well as other lexicaland dependency tree-based features (Lotan et al., 2013). Several systems use supervised models trained over rule-based features. Diab et al. (2009) and Prabhakaran et al. (2010) use SVMs and CRFs over lexical and dependency features for predicting author belief commitments, which they treat as a sequence tagging problem. Lee et al. (2015) train an SVM on lexical and dependency path features for their factuality dataset. Saur and Pustejovsky (2012) and Stanovsky et al. (2017) train support vector models over the outputs of rule-based systems, the latter with TruthTeller. Even the largest currently existing event factuality datasets are extremely small from the perspective of related tasks, like natural language inference (NLI). Where FactBank, UW, MEANTIME, and the original UDS-IH1 dataset have on the order of 30,000 labeled examples combined, standard NLI datasets, like the Stanford Natural Language Inference (SNLI; Bowman et al. 2015) dataset, have on the order of 500,000. To begin to remedy this situation, we collect an extension of the UDS-IH1 dataset. The resulting UDS-IH2 dataset covers all predicates in EUD1.2. Beyond substantially expanding the amount of publicly available event factuality annotations, another major benefit is that EUD1.2 consists entirely of gold parses and has a variety of other annotations built on top of it, making future multitask modeling possible. We use the protocol described by White et al. (2016) to construct UDS-IH2. This protocol involves four kinds of questions for a particular predicate candidate: 1. UNDERSTANDABLE : whether the sentence is understandable 2. PREDICATE : whether or not a particular word refers to an eventuality (event or state) 3. HAPPENED : whether or not, according to the author, the event has already happened or is currently happening 4. CONFIDENCE : how confident the annotator is about their answer to HAPPENED from 0-4 If an annotator answers no to either UNDERSTANDABLE or PREDICATE , HAPPENED and CONFIDENCE do not appear. The main differences between this protocol and the others discussed above are: (i) instead of asking about annotator confidence, the other proto-733 lll l lll l llll l lll llll llll l l l l l l l l 3 2 1 0 1 2 3 l l l l FactBank UW MEANTIMEUDSIH2 Figure 2: Relative frequency of factuality ratings in training and development sets. cols ask the annotator to judge either source confidence or likelihood; and (ii) factuality and confidence are separated into two questions. We choose to retain White et al.'s protocol to maintain consistency with the portions of EUD1.2 that were already annotated in UDS-IH1. Annotators We recruited 32 unique annotators through Amazon's Mechanical Turk to annotate 20,580 total predicates in groups of 10. Each predicate was annotated by two distinct annotators. Including UDS-IH1, this brings the total number of annotated predicates to 27,289. Raw inter-annotator agreement for the HAPPENED question was 0.84 (Cohen's  =0.66) among the predicates annotated only for UDS-IH2. This compares to the raw agreement score of 0.82 reported by White et al. (2016) for UDS-IH1. To improve the overall quality of the annotations, we filter annotations from annotators that display particularly low agreement with other annotators on HAPPENED and CONFIDENCE . (See the Supplementary Materials for details.) Pre-processing To compare model results on UDS-IH2 to those found in the unified datasets of Stanovsky et al. (2017), we map the HAPPENED and CONFIDENCE ratings to a single FACTUALITY value in [-3,3] by first taking the mean confidence rating for each predicate and mapping FACTUALITY to 34 CONFIDENCE if HAPPENED and 34 CONFIDENCE otherwise. Response distribution Figure 2 plots the distribution of factuality ratings in the train and dev splits for UDS-IH2, alongside those of FactBank, UW, and MEANTIME. One striking feature of these distributions is that UDS-IH2 displays a much more entropic distribution than the other datasets. This may be due to the fact that, unlike the newswire-heavy corpora that the other datasets annotate, EUD1.2 contains text from genres  weblogs, newsgroups, email, reviews, and question-answers  that tend to involve less reporting of raw facts. One consequence of this more entropic distribution is that, unlike the datasets discussed above, it is much harder for systems that always guess 3  i.e. factual with high confi-dence/likelihood  to perform well. We consider two neural models of factuality: a stacked bidirectional linear chain LSTM (  4.1) and a stacked bidirectional child-sum dependency tree LSTM (  4.2). To predict the factuality v t for the event referred to by a word w t , we use the hidden state at t from the final layer of the stack as the input to a two-layer regression model (  4.3). We use a standard stacked bidirectional linear chain LSTM (stacked L-biLSTM), which extends the unidirectional linear chain LSTM (Hochreiter and Schmidhuber, 1997) by adding the notion of a layer l  { 1 , . . . , L } and a direction d  { , } (Graves et al., 2013; Sutskever et al., 2014; Zaremba and Sutskever, 2014). (cid:16) (cid:17) where  is the Hadamard product; prev  ( t ) = t  1 and prev  ( t ) = t + 1 , and x ( l,d ) t = x t if l = 1 ; and x ( l,d ) t = [ h ( l  1 ,  ) t ; h ( l  1 ,  ) t ] otherwise. We set g to the pointwise nonlinearity tanh. We use a stacked bidirectional extension to the child-sum dependency tree LSTM (T-LSTM; Tai et al., 2015), which is itself an extension of a standard unidirectional linear chain LSTM (L-LSTM). One way to view the difference between the L-LSTM and the T-LSTM is that the T-LSTM re-defines prev  ( t ) to return the set of indices that 734 correspond to the children of w t in some dependency tree. Because the cardinality of these sets varies with t , it is necessary to specify how multiple children are combined. The basic idea, which we make explicit in the equations for our extension, is to define f tk for each child index k  prev  ( t ) in a way analogous to the equations in  4.1  i.e. as though each child were the only child  and then sum across k within the equations for i t , o t ,  c t , c t , and h t . Our stacked bidirectional extension (stacked T-biLSTM) is a minimal extension to the T-LSTM in the sense that we merely define the downward computation in terms of a prev  ( t ) that returns the set of indices that correspond to the parents of w t in some dependency tree (cf. Miwa and Bansal 2016, who propose a similar, but less minimal, model for relation extraction). The same method for combining children in the upward computation can then be used for combining parents in the downward computation. This yields a minimal change to the stacked L-biLSTM equations. (cid:16) (cid:17) We use a ReLU pointwise nonlinearity for g . These minimal changes allow us to represent the inside and the outside contexts of word t (at layer l ) as single vectors:  h ( l,  ) t and  h ( l,  ) t . An important thing to note here is that  in contrast to other dependency tree-structured T-LSTMs (Socher et al., 2014; Iyyer et al., 2014)  this T-biLSTM definition does not use the dependency labels in any way. Such labels could be straightforwardly incorporated to determine which parameters are used in a particular cell, but for current purposes, we retain the simpler structure (i) to more directly compare the Land T-biLSTMs and (ii) because a model that uses dependency labels substantially increases the number of trainable parameters, relative to the size of our datasets. To predict the factuality v t for the event referred to by a word w t , we use the hidden states from the final layer of the stacked Lor T-biLSTM as the input to a two-layer regression model. h ( L ) t = [ h ( L,  ) t ; h ( L,  ) t ]  v t = V 2 g (cid:16) V 1 h ( L ) t + b 1 (cid:17) + b 2 where  v t is passed to a loss function L (  v t , v t ) : in this case, smooth L1  i.e. Huber loss with  = 1 . This loss function is effectively a smooth variant of the hinge loss used by Lee et al. (2015) and Stanovsky et al. (2017). We also consider a simple ensemble method, wherein the hidden states from the final layers of both the stacked L-biLSTM and the stacked T-biLSTM are concatenated and passed through the same two-layer regression model. We refer to this as the H(ybrid)-biLSTM. 2 5 Experiments Implementation We implement both the L-biLSTM and T-biLSTM models using pytorch 0.2.0 . The L-biLSTM model uses the stock implementation of the stacked bidirectional linear chain LSTM found in pytorch , and the T-biLSTM model uses a custom implementation, which we make available at decomp.net. Word embeddings We use the 300-dimensional GloVe 42B uncased word embeddings (Penning-ton et al., 2014) with an UNK embedding whose dimensions are sampled iid from a Uniform[-1,1]. We do not tune these embeddings during training. Hidden state sizes We set the dimension of the hidden states h ( l,d ) t and cell states c ( l,d ) t to 300 for all layers of the stacked Land stacked T-biLSTMs  the same size as the input word embeddings. This means that the input to the regression model is 600-dimensional, for the stacked Land T-biLSTMs, and 1200-dimensional, for the stacked H-biLSTM. For the hidden layer of the regression component, we set the dimension to half the size of the input hidden state: 300, for 2 See Miwa and Bansal 2016; Bowman et al. 2016 for alternative ways of hybridizing linear and tree LSTMs for semantic tasks. We use the current method since it allows us to make minimal changes to the architectures of each model, which in turn allows us to assess the two models' ability to capture different aspects of factuality. Bidirectional layers We consider stacked L-, T, and H-biLSTMs with either one or two layers. In preliminary experiments, we found that networks with three layers badly overfit the training data. Dependency parses For the Tand H-biLSTMs, we use the gold dependency parses provided in EUD1.2 when training and testing on UDS-IH2. On FactBank, MEANTIME, and UW, we follow Stanovsky et al. (2017) in using the automatic dependency parses generated by the parser in spaCy (Honnibal and Johnson, 2015). 3 Lexical features Recent work on neural models in the closely related domain of generic-ity/habituality prediction suggests that inclusion of hand-annotated lexical features can improve clas-sification performance (Becker et al., 2017). To assess whether similar performance gains can be obtained here, we experiment with lexical features for simple factive and implicative verbs (Kiparsky and Kiparsky, 1970; Karttunen, 1971a). When in use, these features are concatenated to the net-work's input word embeddings so that, in princi-ple, they may interact with one another and inform other hidden states in the biLSTM, akin to how verbal implicatives and factives are observed to influence the factuality of their complements. The hidden state size is increased to match the input embedding size. We consider two types: Signature features We compute binary features based on a curated list of 92 simple implicative and 95 factive verbs including their their type-level implication signatures, as compiled by Nairn et al. (2006). 4 These signatures characterize the 3 In rebuilding the Unified Factuality dataset (Stanovsky et al., 2017), we found that sentence splitting was potentially sensitive to the version of spaCy used. implicative or factive behavior of a verb with respect to its complement clause, how this behavior changes (or does not change) under negation, and how it composes with other such verbs under nested recursion. We create one indicator feature for each signature type. Mined features Using a simplified set of pattern matching rules over Common Crawl data (Buck et al., 2014), we follow the insights of Pavlick and Callison-Burch (2016)  henceforth, PC  and use corpus mining to automatically score verbs for implicativeness. The insight of PC lies in Karttunen's (1971a) observation that the main sentence containing an implicative predicate and the complement sentence necessarily agree in tense. Accordingly, PC devise a tense agreement score  effectively, the ratio of times an embedding predicate's tense matches the tense of the predicate it embeds  to predict implicativeness in English verbs. Their scoring method involves the use of fine-grained POS tags, the Stanford Temporal Tagger (Chang and Manning, 2012), and a number of heuristic rules, which resulted in a confirma-tion that tense agreement statistics are predictive of implicativeness, illustrated in part by observing a near perfect separation of a list of implicative and non-implicative verbs from Karttunen (1971a). We replicate this finding by employing a simplified pattern matching method over 3B sentences of raw Common Crawl text. We efficiently search for instances of any pattern of the form: I $VERB to * $TIME , where $VERB and $TIME are pre-instantiated variables so their corresponding tenses are known, and  * ' matches any one to three whitespace-separated tokens at runtime (not pre-instantiated). 5 Our results in Table 3 are a close lnr/Lexical_Resources 5 To instantiate $VERB , we use a list of 1K clause-embedding verbs compiled by (White and Rawlins, 2016) as well as the python package pattern-en to conjugate each verb in past, present progressive, and future tenses; all conjugations are first-person singular. $TIME is instantiated 736 FactBank UW Meantime UDS-IH2 MAE r MAE r MAE r MAE r All-3.0 0.8 NAN 0.78 NAN 0.31 NAN 2.255 NAN Lee et al. 2015 -0.511 0.708 --Stanovsky et al. 2017 0.59 0.71 0.42  0.66 0.34 0.47 -L-biLSTM(2)-S 0.427 0.826 0.508 0.719 0.427 0.335 0.960  0.768 T-biLSTM(2)-S 0.577 0.752 0.600 0.645 0.428 0.094 1.101 0.704 L-biLSTM(2)-G 0.412 0.812 0.523 0.703 0.409 0.462 -T-biLSTM(2)-G 0.455 0.809 0.567 0.688 0.396 0.368 -L-biLSTM(2)-S+lexfeats 0.429 0.796 0.495 0.730 0.427 0.322 1.000 0.755 T-biLSTM(2)-S+lexfeats 0.542 0.744 0.567 0.676 0.375 0.242 1.087 0.719 L-biLSTM(2)-MultiSimp 0.353 0.843 0.503 0.725 0.345 0.540 -T-biLSTM(2)-MultiSimp 0.482 0.803 0.599 0.645 0.545 0.237 -L-biLSTM(2)-MultiBal 0.391 0.821 0.496 0.724 0.278 0.613  -T-biLSTM(2)-MultiBal 0.517 0.788 0.573 0.659 0.400 0.405 -L-biLSTM(1)-MultiFoc 0.343 0.823 0.516 0.698 0.229  0.599 -L-biLSTM(2)-MultiFoc 0.314 0.846 0.502 0.710 0.305 0.377 -T-biLSTM(2)-MultiFoc 1.100 0.234 0.615 0.616 0.395 0.300 -L-biLSTM(2)-MultiSimp w/UDS-IH2 0.377 0.828 0.508 0.722 0.367 0.469 0.965 0.771  T-biLSTM(2)-MultiSimp w/UDS-IH2 0.595 0.716 0.598 0.609 0.467 0.345 1.072 0.723 H-biLSTM(2)-S 0.488 0.775 0.526 0.714 0.442 0.255 0.967 0.768 H-biLSTM(1)-MultiSimp 0.313  0.857  0.528 0.704 0.314 0.545 -H-biLSTM(2)-MultiSimp 0.431 0.808 0.514 0.723 0.401 0.461 -H-biLSTM(2)-MultiBal 0.386 0.825 0.502 0.713 0.352 0.564 -H-biLSTM(2)-MultiSimp w/UDS-IH2 0.393 0.820 0.481 0.749  0.374 0.495 0.969 0.760 Table 4: All 2-layer systems, and 1-layer systems if best in column. replication of PC's findings. Prior work such as by PC is motivated in part by the potential for corpus-linguistic findings to be used as fodder in downstream predictive tasks: we include these agreement scores as potential input features to our networks to test whether contemporary models do in fact benefit from this information. Training For all experiments, we use stochastic gradient descent to train the LSTM parameters and regression parameters end-to-end with the Adam optimizer (Kingma and Ba, 2015), using the default learning rate in pytorch ( 1e-3 ). We consider five training regimes: 6 1. SINGLE-TASK SPECIFIC (-S) Train a separate instance of the network for each dataset, training only on that dataset. 2. SINGLE-TASK GENERAL (-G) Train one instance of the network on the simple concatenation of all unified factuality datasets, { FactBank, UW, MEANTIME } . 3. MULTI-TASK SIMPLE (-MULTISIMP ) Same with each of five past tense phrases (yesterday, last week, etc.) and five corresponding future tense phrases (tomor-row, next week, etc). See Supplement for further details. 6 Multi-task can have subtly different meanings in the NLP community; following terminology from Mou et al. (2016), our use is best described as semantically equivalent transfer with simultaneous (MULT) network training. as SINGLE-TASK GENERAL , except the network maintains a distinct set of regression parameters for each dataset; all other parameters (LSTM) remain tied. w/UDS-IH2 is specified if UDS-IH2 is included in training. 4. MULTI-TASK BALANCED (-MULTIBAL ) Same as MULTI-TASK SIMPLE but upsampling examples from the smaller datasets to ensure that examples from those datasets are seen at the same rate. 5. MULTI-TASK FOCUSED (-MULTIFOC ) Same as MULTI-TASK SIMPLE but upsampling examples from a particular target dataset to ensure that examples from that dataset are seen 50% of the time and examples from the other datasets are seen 50% (evenly distributed across the other datasets). Calibration Post-training, network predictions are monotonically re-adjusted to a specific dataset using isotonic regression (fit on train split only). Evaluation Following Lee et al. (2015) and Stanovsky et al. (2017), we report two evaluation measures: mean absolute error (MAE) and Pearson correlation (r). We would like to note, however, that we believe correlation to be a better indicator of performance for two reasons: (i) for datasets with a high degree of label imbalance 737 Mean Linear Tree Modal Negated Label MAE MAE # NONE no 1.00 0.93 1.03 2244 NONE yes -0.19 1.40 1.69 98 may no -0.38 1.00 0.99 14 would no -0.61 0.85 0.99 39 ca(n't) yes -0.72 1.28 1.55 11 can yes -0.75 0.99 0.86 6 (wi)'ll no -0.94 1.47 1.14 8 could no -1.03 0.97 1.32 20 can no -1.25 1.02 1.21 73 might no -1.25 0.66 1.06 6 would yes -1.27 0.40 0.86 5 should no -1.31 1.20 1.01 22 will no -1.88 0.75 0.86 75 Table 5: Mean gold labels, counts, and MAE for LbiLSTM(2)-S and T-biLSTM(2)-S model predictions on UDS-IH2-dev, grouped by modals and negation. (Figure 2), a baseline that always guesses the mean or mode label can be difficult to beat in terms of MAE but not correlation, and (ii) MAE is harder to meaningfully compare across datasets with different label mean and variance. Development Under all regimes, we train the model for 20 epochs  by which time all models appear to converge. We save the parameter values after the completion of each epoch and then score each set of saved parameter values on the development set for each dataset. The set of parameter values that performed best on dev in terms of Pearson correlation for a particular dataset were then used to score the test set for that dataset. Table 4 reports the results for all of the 2-layer L-, T-, and H-biLSTMs. 7 The best-performing system for each dataset and metric are highlighted in purple, and when the best-performing system for a particular dataset was a 1-layer model, that system is included in Table 4. New state of the art For each dataset and metric, with the exception of MAE on UW, we achieve state of the art results with multiple systems. The highest-performing system for each is reported in Table 4. Our results on UDS-IH2 are the first reported numbers for this new factuality resource. 7 Full results are reported in the Supplementary Materials. Note that the 2-layer networks do not strictly dominate the 1-layer networks in terms of MAE and correlation. topology (T-biLSTM). However, the hybrid topology (H-biLSTM), consisting of both a Land T-biLSTM is the top-performing system on UW for correlation (Table 4). This suggests that the T-biLSTM may be contributing something complementary to the L-biLSTM. Evidence of this complementarity can be seen in Table 6, which contains a breakdown of system performance by governing dependency relation, for both linear and tree models, on UDS-IH2-dev. In most cases, the L-biLSTM's mean prediction is closer to the true mean. This appears to arise in part because the T-biLSTM is less confident in its predictions  i.e. its mean prediction tends to be closer to 0. This results in the L-biLSTM being too confident in certain cases e.g. in the case of the xcomp governing relation, where the T-biLSTM mean prediction is closer to the true mean. Lexical features have minimal impact Adding all lexical features (both SIGNATURE and MINED ) yields mixed results. We see slight improvements on UW, while performance on the other datasets mostly declines (compare with SINGLETASK SPECIFIC ). Factuality prediction is precisely the kind of NLP task one would expect these types of features to assist with, so it is notable that, in our experiments, they do not. Multi-task helps Though our methods achieve state of the art in the single-task setting, the best performing systems are mostly multi-task (Table 4 and Supplementary Materials). This is an ideal setting for multi-task training: each dataset is relatively small, and their labels capture closely-related (if not identical) linguistic phenomena. UDS-IH2, the largest by a factor of two, reaps the smallest gains from multi-task. As discussed in  2, many discrete linguistic phenomena interact with event factuality. Here we provide a brief analysis of some of those interactions, both as they manifest in the UDS-IH2 dataset, as well as in the behavior of our models. This analysis employs the gold dependency parses present in EUD1.2. Table 5 illustrates the influence of modals and negation on the factuality of the events they have direct scope over. The context with the highest factuality on average is no direct modal and no negation (first row); all other modal contexts have varying degrees of negative mean factuality scores, with will as the most negative. This is likely a result of UDS-IH2 annotation instructions to mark future events as not having happened. Table 7 shows results from a manual error analysis on 50 events from UDS-IH2-dev with highest absolute prediction error (using H-biLSTM(2)-MultiSim w/UDS-IH2). Grammatical errors (such as run-on sentences) in the underlying text of UDS-IH2 appear to pose a particular challenge for these models; informal language and grammatical errors in UDS-IH2 is a substantial distinction from the other factuality datasets used here. In  6 we observe that the linguistically-motivated lexical features that we test (+lexfeats) do not have a big impact on overall performance. Tables 8 and 9 help nuance this observation. Table 8 shows that we can achieve similar separation between implicatives and non-implicatives as the feature mining strategy presented in  5. That is, those features may be redundant with information already learnable from factuality datasets (UDS-IH2). Despite the un-derperformance of these features overall, Table 9 shows that they may still improve performance in the subset of instances where they appear. We have proposed two neural models of event factuality prediction  a bidirectional linear-chain LSTM (L-biLSTM) and a bidirectional child-sum dependency tree LSTM (T-biLSTM)  which yield substantial gains over previous models based on deterministic rules and hand-engineered features. We found that both models yield such gains, though the L-biLSTM outperforms the T-biLSTM; for some datasets, an ensemble of the two (H-biLSTM) improves over either alone. We have also extended the UDS-IH1 dataset, yielding the largest publicly-available factuality dataset to date: UDS-IH2. In experiments, we see substantial gains from multi-task training over the three factuality datasets unified by Stanovsky et al. (2017), as well as UDS-IH2. Future work will further probe the behavior of these models, or extend them to learn other aspects of event semantics. This research was supported by the JHU HLT-COE, DARPA LORELEI, DARPA AIDA, and NSF-GRFP (1232825). The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes. The views and conclusions contained in this publication are those of the authors and should not be interpreted as representing official policies or endorsements of DARPA or the U.S. Government.
result method result abstain abstain abstain objective abstain abstain method result abstain method objective method result other other other other other other other other other other other other other other other other other other other other other method objective other other other other other other other method other abstain other other method other other other other other other other method objective abstain other other other other other other other other other abstain abstain method abstain abstain method other abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain other abstain method abstain abstain method abstain abstain abstain method abstain abstain abstain abstain method method other abstain method abstain method abstain method abstain abstain abstain abstain method method abstain method method abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain method abstain abstain other abstain abstain method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain method other abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective result objective result abstain other other other