We present a novel document-level model for finding argument spans that fill an event's roles, connecting related ideas in sentence-level semantic role labeling and coreference resolution. Because existing datasets for cross-sentence linking are small, development of our neural model is supported through the creation of a new resource, R oles A cross M ultiple S entences (RAMS), which contains 9,124 annotated events across 139 types. We demonstrate strong performance of our model on RAMS and other event-related datasets. 1 1 Introduction Textual event descriptions may span multiple sentences, yet large-scale datasets predominately annotate for events and their arguments at the sentence level. This has driven researchers to focus on sentence-level tasks such as semantic role labeling (SRL), even though perfect performance at such tasks would still enable a less than complete understanding of an event at the document level. In this work, we approach event understanding as a form of linking , more akin to coreference resolution than sentence-level SRL. An event trigger evokes a set of roles regarded as latent arguments, with these implicit arguments then potentially linked to explicit mentions in the text. Consider the example in Figure 1: the AirstrikeMissileStrike event (triggered by bombarding) gives rise to a frame or set of type-level roles ( attacker , target , instrument , place ) with the referents (Russians, rebel out-post, aircraft, Syria). 2 Intuitively we recognize the possible existence of fillers for these roles, for example, the place of the particular AirEqual Contribution 1 Data and code at http://nlp.jhu.edu/rams/ . strikeMissileStrike event. These implicit arguments are linked to explicit arguments in the document (i.e., text spans). We refer to the task of finding explicit argument(s) to fill each role for an event as argument linking . Prior annotation of cross-sentence argument links has produced small datasets, with a focus either on a small number of predicate types (Gerber and Chai, 2010, 2012; Feizabadi and Pado, 2014) or on a small number of documents (Ruppenhofer et al., 2010). To enable the development of a neural model for argument linking, we produce R oles A cross M ultiple S entences (RAMS), a dataset of 9,124 annotated events from news based on an ontology of 139 event types and 65 roles. In a 5-sentence window around each event trigger, we annotate the closest argument span for each role. Our model builds on recent ideas in span selection models (Lee et al., 2018; He et al., 2018; Ouchi et al., 2018), used in this work for the multi-sentence argument linking task for RAMS and for several other event-based datasets (Gerber and Chai, 2012; Pradhan et al., 2013; Pavlick et al., 2016, AIDA Phase 1). On RAMS our best model achieves 68.3 F 1 , and it achieves 73.3 F 1 when event types are also known, outperforming strong baselines. We also demonstrate effective use of RAMS as pre-training for a related dataset. Our main contributions are a novel model for argument linking and a new large-scale dataset for the task. Our dataset is annotated for arguments across multiple sentences and has broader coverage of event types and more examples than similar work. Our experiments highlight our model's adaptability to multiple datasets. Together, these contributions further the automatic understanding of events at the document level. We are not the first to consider non-local event arguments; here we review prior work and refer to O'Gorman (2019) for further reading. Whereas local (sentence-level) event arguments are well-studied as semantic role labelingutilizing large datasets such as OntoNotes 5.0 (Weischedel et al., 2013; Pradhan et al., 2013)existing datasets annotated for non-local arguments are too small for training neural models. Much of the effort on non-local arguments, sometimes called implicit SRL, has focused on two datasets: SemEval-2010 Task 10 (Ruppen-hofer et al., 2010) and Beyond NomBank (hence-forth BNB ) (Gerber and Chai, 2010, 2012). These datasets are substantially smaller than RAMS: the SemEval Task 10 training set contains 1,370 frame instantiations over 438 sentences, while BNB contains 1,247 examples covering just 10 nominal predicate types. Multi-sentence AMR (MS-AMR) (O'Gorman et al., 2018; Knight et al., 2020) contains 293 documents annotated with a document-level adaptation of the Abstract Meaning Representation (AMR) formalism. O'Gorman (2019) notes that the relatively small size of the MS-AMR and SemEval datasets hinders supervised training. In contrast to these datasets, RAMS contains 9,124 annotated examples covering a wide range of nominal and verbal triggers. Under the DARPA AIDA program, the Linguistic Data Consortium (LDC) has annotated document-level event arguments under a three-level hierarchical event ontology (see Figure 2) in-fluenced by prior LDC-supported ontologies such as ERE and ACE. These have been packaged as the AIDA Phase 1 Practice 3 and Eval 4 releases (henceforth AIDA-1 ), currently made available to performers in the AIDA program and participants 3 LDC2019E04 (data); LDC2019E07 (annotations) 4 LDC2019E42 (data); LDC2019E77 (annotations) Correspondence CommandOrder Negotiate FirearmAttack Yield Conflict { Communicator, Recipient, Place } { Attacker, Target, Instrument, Place }    Attack Contact Meet  Figure 2: Subset of the AIDA-1 ontology illustrating the three-level Type/Subtype/Sub-subtype event hierarchy. in related NIST evaluations. 5 AIDA-1 documents focus on recent geopolitical events relating to interactions between Russia and Ukraine. Unless otherwise noted, statistics about AIDA-1 pertain only to the Practice portion of the dataset. For each document in LDC's collection, only AIDA-salient events are annotated. This protocol does not guarantee coverage over the event ontology: 1,559 event triggers are annotated in the text portion of the collection, accounting for only 88 of the 139 distinct event sub-subtypes in the ontology. Our dataset, RAMS, employs the same annotation ontology but is substantially larger and covers all 139 types in the ontology. Figure 3 (  3) compares the two datasets. Across multiple datasets, a substantial number of event arguments are observed to be nonlocal. For example, Gerber and Chai (2012) found that their annotation of non-local arguments added 71% (relative) role coverage to NomBank annotations. Additionally, 38.1% of the annotated events in AIDA-1 have an argument outside the sentence containing the trigger. This phenomenon is not surprising in light of the analysis of zero anaphora and definite null complements by Fillmore (1986) and the distinction between core and non-core frame elements or roles in FrameNet (Baker et al., 1998) and PropBank (Palmer et al., 2005). As previous datasets have been small, various approaches have been taken to handle scarcity. To obtain more training data, Silberer and Frank (2012) created artificial instances from data annotated jointly for coreference and semantic roles. Roth and Frank (2013) automatically induced implicit arguments from pairs of comparable texts, but recovered a proportionally small set of additional arguments. Feizabadi and Pado (2015) 5 While rarely freely released, historically such collections are eventually made available under a license to anyone, under some timeline established within a program. combined existing corpora to increase and diversify sources of model supervision. Cheng and Erk (2018, 2019) approached the data scarcity problem by recasting implicit SRL as a cloze task and as a reading comprehension task, for which data can be generated automatically. The TAC KBP event argument extraction task also seeks arguments from document contexts. However, in our work we are concerned with rei-fied events (explicit mentions) and links between event mentions and argument mentions rather than entity-level arguments (coreference clusters). Motivated by the scarcity of data for training neural models to predict non-local arguments, we constructed R oles A cross M ultiple S entences (RAMS), a crowd-sourced dataset with annotations for 9,124 events following the AIDA ontology. We employed the AIDA ontology in RAMS so-as to be most similar to an existing corpus already being investigated by various members of the community. Each example consists of a typed trigger span and 0 or more argument spans in an English document. A trigger span is a word or phrase that evokes a certain event type in context, while argument spans denote role-typed participants in the event (e.g., the Recipient ). Trigger and argument spans are token-level [ start, end ] offsets into a tokenized document. Typically, event and relation datasets annotate only the argument spans that are in the same sentence as the trigger, but we present annotators with a multi-sentence context window surrounding the trigger. Annotators may select argument spans in any sentence in the context window. Data Source We used Reddit, a popular internet forum, to filter a collection of news articles to be topically similar to AIDA-1. After applying a set of criteria based on keywords, time period, and popularity (listed in Appendix A.1) we identified approximately 12,000 news articles with an average length of approximately 40 sentences. Annotation We manually constructed a mapping from each event ((sub-)sub)type to a list of lexical units (LUs) likely to evoke that type. 6 This mapping was designed to give high precision and 6 For example, Conflict/Attack/SetFire is evoked by inferno , blaze , and arson (and word forms). low recall, in that for a given ( Type , LUs ) pair, the items in LUs are all likely to evoke the Type , although LUs can omit items that also evoke the Type . On average, | LUs | = 3 . 9 . We performed a soft match 7 between every LU and every word in our text collection to select candidate sentences for each event type. This matching procedure produced approximately 94,000 candidates, which we balanced by sampling the same number of sentences for each LU. Candidate sentences were then vetted by crowdsourcing to ensure that they evoked their associated event type and had positive factuality. We collected judgments on approximately 17,500 candidate sentences, of which 52% were determined to satisfy these constraints, yielding 9,124 sentences containing a LU trigger. Using these sentences we then collected multi-sentence annotations, presenting annotators with a 5-sentence window containing two sentences of context before the sentence with the trigger and two sentences after. 8 Annotators then selected in the context window a span to fill each of the event's roles. A window size of five sentences was chosen based on internal pilots and supported by our finding that 90% of event arguments in AIDA-1 are recoverable in this window size. Similarly, Gerber and Chai (2010) found that in their data almost 90% of implicit arguments can be resolved in the two sentences preceding the trigger. 9 Arguments fall close to the trigger in RAMS as well: 82% of arguments occur in the same sentence as the trigger. On average, we collected 66 full annotations (trigger and arguments) per event type. Table 1 shows dataset size and coverage. All aspects of the protocol, including the annotation interface and instructions, are included in Appendix A. 7 We stem all words and ignore case. 8 If fewer than two sentences appeared before/after the trigger, annotators were shown as many sentences as were available. Inter-Annotator Agreement We randomly selected 93 tasks for redundant annotation in order to measure inter-annotator agreement, collecting five responses per task from distinct users. 68.5% of the time, all annotators mark the role as either absent or present. Less frequently (21.7%), four of the five annotators agree, and rarely (9.8%) is there strong disagreement. We compute pairwise agreement for span boundaries. For each annotated (event, role) combination, we compare pairs of spans for which both annotators believe the role is present. 55.3% of the pairs agree exactly. Allowing for a fuzzier match, such as to account for whether one includes a determiner, spans whose boundaries differ by one token have a much higher agreement of 69.9%. Fewer spans agree on the start boundary (59.8%) than on the end (73.5%), while 78.0% match at least one of the two boundaries. We demonstrate data quality in  5.2 by showing its positive impact on a downstream task. Comparisons to Related Datasets Comparisons of event type coverage among RAMS, AIDA-1, and BNB (Gerber and Chai, 2010, 2012) are given in Figure 3. RAMS provides larger and broader coverage of event types than do AIDA-1 and BNB. By design, BNB focuses on only a few predicate types, but we include its statistics for reference. More figures regarding type and role coverage are included in Appendix A.4. Related Protocols Feizabadi and Pado (2014) also considered the case of crowdsourcing annotations for cross-sentence arguments. Like us, they provided annotators with a context window rather than the whole document, annotating two frames each with four roles over 384 predicates. Annotators in that work were shown the sentence containing the predicate and the three previous sentences, unlike ours which shows two preceding and two following sentences. Rather than instructing annotators to highlight spans in the text (marking), Feizabadi and Pado (2014) directed annotators to fill in blanks in tem-platic sentences (gap filling). We in contrast require annotators to highlight mention spans directly in the text. Our protocol of event type verification followed by argument finding is similar to the protocol supported by interfaces such as SALTO (Burchardt et al., 2006) and that of Fillmore et al. (2002). We formulate argument linking as follows, similar to the formulation in Das et al. (2010). Assume a document D contains a set of described events E , each designated by a triggera text span in D . The type of an event e determines the set of roles the event's arguments may take, denoted R e . For each e  E , the task is to link the event's roles with argumentstext spans in D if they are attested. Specifically, one must find for each e all ( r, a ) pairs such that r  R e and a  D . This formulation does not restrict each role to be filled by only one argument, nor does it restrict each explicit argument to take at most one role. Our model architecture is related to recent models for SRL (He et al., 2018; Ouchi et al., 2018). Contextualized text embeddings are used to form candidate argument span representations, A . These are then pruned and scored alongside the trigger span and learned role embeddings to determine the best argument span (possibly none) for each event and role, i.e., argmax a A P ( a | e, r ) for each event e  E and role r  R e . Representations To represent text spans, we adopt the convention from Lee et al. (2017) that has been used for a broad suite of core NLP tasks (Swayamdipta et al., 2018; He et al., 2018; Tenney et al., 2019b). A bidirectional LSTM encodes each sentence's contextualized embeddings (Pe-ters et al., 2018; Devlin et al., 2018). The hidden states at the start and end of the span are concatenated along with a feature vector for the size of the span and a soft head word vector produced by a learned attention mask over the word vectors (GloVe embeddings (Pennington et al., 2014) and character-level convolutions) within the span. We use this method to form representations of trigger spans, e , and of candidate argument spans, a . We learn a separate embedding, r , for each role in the ontology, r  R . Since our objective is to link candidate arguments to event-role pairs, we construct an event-role representation 10 by applying a feed-forward neural network (F  a ) to the event trigger span and role embedding:  a e,r = F  a ([ e ; r ]) (1) This method is similar to one for forming edge representations for cross-sentence relation extraction (Song et al., 2018), but contrasts with prior work which limits the interaction between r and e (He et al., 2018; Tenney et al., 2019b). Pruning Given a document with n tokens, there are O ( n 2 ) candidate argument text spans, which leads to intractability for large documents. Following Lee et al. (2017) and He et al. (2018), we consider within-sentence spans up to a certain width (giving O ( n ) spans) and score each span, a , using a learned unary function of its representation: s A ( a ) = w (cid:62) AFA ( a ) . We keep the top  A n spans (  A is a hyperparameter) and refer to this set of high-scoring candidate argument spans as A . In an unpruned model, we need to create at least (cid:80) e |R e | event-role representations and evaluate ( n (cid:80) e |R e | ) combinations of events, roles, and arguments, which can become prohibitively large when there are numerous events and roles. Assuming the number of events is linear in document length, the number of combinations would be quadratic in document length (rather than quadratic in sentence length as in He et al. (2018)). Lee et al. (2018) addressed this issue in coreference resolution, a different document-level task, by implementing a coarse pruner to limit the number of candidate spans that are subsequently scored. For our model, any role can potentially be filled (if the event type is not known). Thus, we do not wish to prematurely prune ( e, r ) pairs, so we must further prune A . Rather than scoring a  A with every event-role pair ( e, r ) , we assign a score between a and every event e . This relaxation reflects a loose notion of how likely an 10 As a role for an event evokes an implicit discourse referent , this can be regarded as an implicit discourse referent representation. argument span is to participate in an event, which can be determined irrespective of a role: s c ( e, a ) = e (cid:62) W c a + s A ( a ) + s E ( e ) +  c ( e, a ) where W c is learned and  c ( e, a ) are task-specific features. We use A e  A to refer to the topk scoring candidate argument spans in relation to e . Scoring We introduce a link scoring function, l ( a,  a e,r ) , between candidate spans a  A e and event-role pairs  a e,r = ( e, r )  E  R . 11 The scoring function decomposes as: l ( a,  a e,r ) = s E,R ( e, r ) + s A,R ( a, r ) + s l ( a,  a e,r ) + s c ( e, a ) , a (cid:54) = (cid:15) (2) s E ( e ) = w (cid:62) EFE ( e ) s E,R ( e, r ) = w (cid:62) E,R F E,R ([ e ; r ]) s A,R ( a, r ) = w (cid:62) A,R F A,R ([ a ; r ]) s l ( a,  a e,r ) = w (cid:62) l F l ([ a ;  a e,r ; a   a e,r ;  l ( a,  a e,r )]) (3) where  l ( a,  a e,r ) is a feature vector containing information such as the (bucketed) token distance between e and a . 12 F x are feed-forward neural networks, and w x are learned weights. The decomposition is inspired by Lee et al. (2017) and He et al. (2018), while the direct scoring of candidate arguments against event-role pairs, s l ( a,  a e,r ) , bears similarities to the approach taken by Schenk and Chiarcos (2016), which finds the candidate argument whose representation is most similar to the prototypical filler of a frame element (role). Learning We denote no explicit argument by (cid:15) and assign it link score l ( (cid:15),  a e,r ) (cid:44) 0 , which acts as a threshold for the link function. For every event-role-argument triple ( e, r, a ) , we maximize P ( a | e, r ) = exp { l ( a,  a e,r ) } (cid:80) a (cid:48) A e { (cid:15) } exp { l ( a (cid:48) ,  a e,r ) } . Decoding We experiment with three decoding strategies: argmax , greedy , and type-constrained . If we assume each role is satisfied by exactly one argument (potentially (cid:15) ), we can perform argmax decoding independently for each role:  a = argmax a A e { (cid:15) } P ( a | e, r ) 11 If the type of e is known, then we could restrict r  R e . To instead predict multiple non-overlapping arguments per role, we could use P ( (cid:15) | e, r ) as a threshold in greedy decoding (Ouchi et al., 2018). We may know the gold event types and the mapping between events e and their permitted roles, R e . While this information can be used during training, we take a simpler approach of using it for type-constrained decoding (TCD). If an event type allows m r arguments for role r , we keep only the top-scoring m r arguments based on link scores. Our model is inspired by several recent span selection models (He et al., 2018; Lee et al., 2018; Ouchi et al., 2018), as well as the long line of neural event extraction models (Chen et al., 2015; Nguyen et al., 2016, inter alia ). O'Gorman (2019) speculates a joint coreference and SRL model in which implicit discourse referents are generated for each event predicate and subsequently clustered with the discovered referent spans using a model for coreference, which is similar to the approach of Silberer and Frank (2012). O'Gorman (2019) further claims that span selection models would be difficult to scale to the document level, which is the regime we are most interested in. We focus on the implicit discourse referents (i.e., the event-role representations) for an event and link them to argument mentions, rather than cluster them using a coreference resolution system or aggregate event structures across multiple events and documents (Wolfe et al., 2015). Our approach is also similar to the one used by Das et al. (2010) for FrameNet parsing. CoNLL 2012 SRL As our model bears similarities to the SRL models proposed by He et al. (2018) and Ouchi et al. (2018), we evaluate our model on the sentence-level CoNLL 2012 dataset as a sanity check. Based on a small hyperparameter sweep, our model achieves 81.4 F 1 when given gold predicate spans and 81.2 F 1 when not given gold predicates. 13 Our model's recall is harmed because our span pruning occurs at the document level rather than at the sentence level, which leads to overpruning in some sentences. Although our model is designed to accommodate cross-sentence links, it maintains competitive performance on sentence-level SRL. 13 We use ELMo (Peters et al., 2018) in these experiments. He et al. (2018) achieve 85.5 F 1 with gold predicates and 82.9 F 1 without gold predicates, and Ouchi et al. (2018) achieve 86.2 F 1 with gold predicates. In the following experiments, for each event the model is given the (gold) trigger span and the (gold) spans of the arguments. The model finds for each role the best argument(s) to fill it. Predictions are returned as trigger-role-argument triples. We use feature-based BERT-base (Devlin et al., 2018)mixing layers 9 through 12by splitting the documents into segments of size 512 subto-kens and encoding each segment separately. 14 We perform preliminary sweeps across hyperparameter values, which are then fixed while we perform a more exhaustive sweep across scoring features. We also compare argmax decoding with greedy decoding during training. The best model is selected based on F 1 on the development set, and ablations are reported in Table 3. Our final model uses greedy decoding, s A,R , and s l and omits s E,R and s c (see Equation 2). More details can be found in Appendix B. The results using our model with greedy decoding and TCD are reported in Table 2. We also report performance of the following baselines: 1) choosing for each link the most common role ( place ), 2) using the same fixed trigger representation across examples, and 3) using the full context window as the trigger. Additionally, we experiment with two other data conditions: 1) linking the correct argument(s) from among a set of distractor candidate arguments provided by a constituency parser (Kitaev and Klein, 2018), 15 and 2) finding the correct argument(s) from among all possible spans up to a fixed length. 14 0.2% of the training documents span multiple segments. 15 We take as the distractor arguments all (potentially overlapping) NP s predicted by the parser. On average, this yields 44 distractors per training document. For the distractor experiment, we use the same hyperparameters as for the main experiment. When not given gold argument spans, we consider all spans up to 5 tokens long and change only the hyperparameters that would prune less aggressively. We hypothesize that the low performance in this setting is due to the sparsity of annotated spans compared to the set of all enumerated spans. In contrast, datasets such as CoNLL 2012 are more densely annotated, so the training signal is not as affected when the model must determine argument spans in addition to linking them. Finally, we examine the effect of TCD to see whether the model effectively uses gold event types if they are given. TCD filters out illegal predictions, boosting precision. Recall is still affected by this decoding strategy because the model may be more confident in the wrong argument for a given role, thus filtering out the less confident, correct one. Nevertheless, using gold types at test time generally leads to gains in performance. Ablations Ablation studies on development data for components of the link score as well as the contextualized encoder and decoding strategy are shown in Table 3. Type-constrained decoding based on knowledge of gold event types improves F 1 in all cases because it removes predictions that are invalid with respect to the ontology. The most important link score component is the score between a combined event-role and a candidate argument. This result follows intuitions that s l is the primary component of the link score since it directly captures the compatibility of the explicit argument and the implicit argument represented by the event-role pair. We also experiment with both ELMo (Peters et al., 2018) and BERT layers 69, which were found to have the highest mixture weights for SRL by Tenney et al. (2019a). We found that BERT generally improves over ELMo and layers 912 often perform better than layers 69. ArgumentTrigger Distance One of the differentiating components of RAMS compared to SRL datasets is its non-local annotation of arguments. At the same time, RAMS uses naturally occurring text so arguments are still heavily distributed within the same sentence as the trigger (Figure 5). This setting allows us to ask whether our model accurately finds arguments outside of the sentence containing the trigger despite the non-uniform distribution. In Table 4, we report F 1 based on distance on the development set and find that performance on distant arguments is comparable to performance on local arguments, demonstrating the model's ability to handle non-local arguments. Role Embeddings and Confusion We present in Figure 4 the cosine similarities between the learned 50-dimensional role embeddings in our model and also the errors made by the model under argmax decoding on the dev set. 16 Some roles are highly correlated. For example, origin and destination have the most similar embeddings, possibly because they co-occur frequently and have the same entity type. Conversely, negatively correlated roles have different entity types or occur in different events, such as communicator compared to destination and artifact . We also observe that incorrect predictions are made more often between highly correlated roles and err 16 Analysis of the confusion matrix with type-constrained decoding is less meaningful because the constraints, which rely on gold event types, filter out major classes of errors. on the side of the more frequent role, as most errors occur below the diagonal. Examples We present predictions from the development set which demonstrate some phenomena of interest. These are made without TCD, illustrating the model's predictions without knowledge of gold event types. In Table 5, the first example demonstrates the model's ability to link a non-local argument which occurs in the sentence before the trigger. Greedy decoding helps the model find multiple arguments satisfying the same participant role, which also appear on either side of the trigger. In the second example, the model correctly predicts the driverpassenger , one of the rarer roles in RAMS (17 instances in the training set), consistent with the gold AccidentCrash event type. In Table 6, the model fills roles corresponding to both the Death and the gold JudicialConsequences event types, thereby mixing roles from different event types. The predictions are plausible when interpreted in context and would be more accurate under TCD. We also investigate how well RAMS serves as pre-training data for AIDA-1. A model using the hyperparameters of our best-performing RAMS model and trained on just English AIDA-1 Practice data achieves 19.1 F 1 on the English AIDA-1 Eval data under greedy decoding and 18.2 F 1 with TCD. When our best-performing RAMS model is fine-tuned to the AIDA task by further training on the AIDA-1 data, performance is improved to 24.4 F 1 under greedy decoding and 24.8 F 1 with TCD. The crowdsourced annotations in RAMS are therefore of sufficient quality to serve as augmentation to LDC's AIDA-1. Experimental details are available in Appendix D. 6 Other Datasets 6.1 Beyond NomBank The Beyond NomBank (BNB) dataset collected by Gerber and Chai (2010) and refined by Gerber and Chai (2012) contains nominal predicates (event triggers) and multi-sentence arguments, both of which are properties shared with RAMS. To accommodate our formulation of the argu-Field Baseline* Our Model Victim Name 9.3 (54.1) 62.2 (69.6) Shooter Name 4.7 (24.1) 53.1 (57.8) Location 12.2 (18.9) 34.9 (63.3) Time 68.1 (69.3) 62.9 (69.4) Weapon 1.1 (17.9) 32.5 (49.6) Table 7: Strict (and approximate) match F 1 on GVDB. ment linking task, we modify the BNB data in two ways: 1) we merge split arguments, which in all but one case are already contiguous spans; and 2) we reduce each cluster of acceptable argument fillers to a set containing only the argument closest to the trigger. We also make modifications to the data splits for purposes of evaluation. Gerber and Chai (2012) suggest evaluation be done using cross-validation on shuffled data, but this may cause document information to leak between the train and evaluation folds. To prevent such leakage and to have a development set for hyperparameter tuning, we separate the data into train, dev, and test splits with no document overlap. Additional data processing details and hyperparameters are given in Appendix E. When given gold triggers and argument spans, our model achieves 75.4 F 1 on dev data and 76.6 F 1 on test data. The Gun Violence Database (GVDB) (Pavlick et al., 2016) is a collection of news articles from the early 2000s to 2016 with annotations specifically related to a gun violence event. We split the corpus chronologically into a training set of 5,056 articles, a development set of 400, and a test set of 500. We use this dataset to perform a MUC-style information extraction task (Sund-heim, 1992). While GVDB's schema permits any number of shooters or victims, we simply predict the first mention of each type. Pavlick et al. (2016) perform evaluation in two settings: a strict match is awarded if the predicted string matches the gold string exactly, while an approximate match is awarded if either string contains the other. Assuming each document contains a single gun violence event triggered by the full document, our goal is to predict the value (argument) for each slot (role) for the event. As each slot is filled by exactly one value, we use argmax decoding. While the baseline experiments of Pavlick et al. (2016) made sentence-level predictions focusing on five attributes, we make document-level predictions and consider the larger set of attributes. Table 7 shows our model's performance on the shared subset of attributes, but the numerical values are not directly comparable because the prior work makes predictions on the full dataset and also combines some roles. Our results show that our model is suitable for information extraction tasks like slot filling. Appendix F contains information on hyperparameters and performance on the full set of roles. To our knowledge, our results are a substantial improvement over prior attempts to predict attributes of gun violence event reports, and we make our models available in the hopes of assisting social scientists in their corpus studies. We introduced a novel model for document-level argument linking. Because of the small amount of existing data for the task, to support training our neural framework we constructed the RAMS dataset consisting of 9,124 events covering 139 event types. Our model outperforms strong baselines on RAMS, and we also illustrated its applicability to a variety of related datasets. We hope that RAMS will stimulate further work on multi-sentence argument linking. We thank Craig Harman for his help in developing the annotation interface. We also thank Tongfei Chen, Yunmo Chen, members of JHU CLSP, and the anonymous reviewers for their helpful discussions and feedback. This work was supported in part by DARPA AIDA (FA8750-18-2-0015) and IARPA BETTER (#2019-19051600005). The views and conclusions contained in this work are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, or endorsements of DARPA, ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
objective objective objective abstain abstain abstain abstain abstain other abstain abstain result abstain method abstain method result objective objective abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method method abstain abstain abstain abstain method method abstain method method abstain method abstain abstain method abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method other abstain method abstain method abstain abstain abstain abstain abstain abstain abstain method abstain other abstain other abstain abstain method method method abstain method abstain abstain abstain abstain method method abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain result result abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result result abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain result abstain method method abstain abstain abstain method method abstain abstain abstain abstain objective method result abstain other other other other other