Extracting informative arguments of events from news articles is a challenging problem in information extraction, which requires a global contextual understanding of each document. While recent work on document-level extraction has gone beyond single-sentence and increased the cross-sentence inference capability of end-to-end models, they are still restricted by certain input sequence length constraints and usually ignore the global context between events. To tackle this issue, we introduce a new global neural generation-based framework for document-level event argument extraction by constructing a document memory store to record the contextual event information and leveraging it to implicitly and explicitly help with decoding of arguments for later events. Empirical results show that our framework outperforms prior methods substantially and it is more robust to adversarially annotated examples with our constrained decoding design. 1 1 Introduction An event is a specific occurrence involving participants (people, objects, etc.). Understanding events in the text is necessary for building machine reading systems, as well as for downstream tasks such as information retrieval, knowledge base population, and trend analysis of real-life world events (Sundheim, 1992). Event Extraction has long been studied as a local sentence-level task (Gr-ishman and Sundheim, 1996; Ji and Grishman, 2008b; Grishman, 2019; Lin et al., 2020). This has driven researchers to focus on developing approaches for sentence-level predicate-argument extraction. This is problematic when events and their arguments spread across multiple sentences  in real-world cases, events are often written through-1 Our code and resources are available at https:// github.com/xinyadu/memory_docie for research purpose. [S3] After having a shootout with several [ policemen including Collin ] last Thursday, both [ Tamerlan Tsarnaev ] and his younger brother [ Dzhokhar ] were captured a day later. [S6] Two week ago, in Boston, authorities on Wednesday reopened [ Boylston Street ], the city thoroughfare where the explosion occurred near the finish line of the race. [S7]  a memorial service for campus policeman Sean Collin, who authorities say the brothers shot to death three days after the bombings   [S1] After having a shootout with police last Thursday, both [Tamerlan Tsarnaev]_Detainee and his younger brother [Dzhokhar]_Detainee was captured a day later. In Figure 1, the excerpt of a news article describes two events in the 3rd sentence (an arrest event triggered by captured) and the 6th sentence (an attack event triggered by explosion). S6 on its own contains little information about the ar-guments/participants of the explosion event, but together with the context of S3 and S7, we can find the informative arguments for the ATTACKER role. In this work, we focus on the informative argument extraction problem, which is more practical and requires much a broader view of cross-sentence context (Li et al., 2021). For example, although the brothers also refers to Tamerlan T. and Dzhokhar (and closer to the trigger word), it 2 In WIKIEVENTS (Li et al., 2021), nearly 40% of events have an argument outside the sentence containing the trigger. In recent years, there have been efforts focusing on event extraction beyond sentence boundaries with end-to-end learning (Ebner et al., 2020; Du, 2021; Li et al., 2021). Most of the work still focuses on modeling each event independently (Li et al., 2021) and ignores the global context partially because of the pretrained models' length limit and their lack of attention for distant context (Khan-delwal et al., 2018). Du et al. (2021) propose to model dependency between events directly via the design of generation output format, yet it is not able to handle longer documents with more events  whereas in real-world news articles there are often more than fifteen inter-related events (Table 2). In addition, previous work often overlooks the consistency between extracted event structures across the long document. For example, if one person has been identified as a JAILER in an event, it's unlikely that the same person is an ATTACKER in another event in the document (Figure 1), according to world event knowledge (Sap et al., 2019; Yao et al., 2020). In this paper, to tackle these challenges and have more consistent/coherent extraction results, we propose a document-level memory-enhanced training and decoding framework (Figure 2) for the problem. It can leverage relevant and necessary context beyond the length constraint of end-to-end models, by using the idea of a dynamic memory store. It helps the model leverage previously generated/extracted event information during both training (implicitly) and during test/decoding (explicitly). More specifically, during training, it retrieves the most similar event sequence in the memory store as additional input context to mode. Plus, it performs constrained decoding based on the memory store and our harvested global knowledge-based argument pairs from the ontology. We conduct extensive experiments and analysis on the WIKIEVENTS corpus and show that our framework significantly outperforms previous methods either based on neural sequence labeling or text generation. We also demonstrate that the framework achieves larger gains over baseline non memory-based models as the number of events grows in the document, and it is more robust to manually designed adversarial examples. In this work, we focus on the challenging problem of extracting informative arguments of events 3 from the document. Each event consists of (1) a trigger expression which is a continuous span in the document, it is of a type E which is predefined in an ontology; (2) and a set of arguments { arg 1 , arg 2 , ... } , each of them has a role predefined in the ontology, for event type E . In the annotation guideline/ontology, the template that describes the connections between arguments of the event type is also provided. For example, when E is Arrest , its corresponding arguments to be extracted should have roles: JAILER ( <arg1> ), DETAINEE ( <arg2> ), CRIME ( <arg3> ), PLACE ( <arg4> ). Its description template is: <arg1> arrested or jailed <arg2> for <arg3> crime at <arg4> place Given a long news document Doc = { ..., <Trg1> , ..., x i , ..., <Trg2> , ..., x n } with given event triggers, our goal is to extract all the informative argument spans to fill in the role of E 1 , E 2 , etc. For the example piece in Figure 1, E 1 is Arrest (triggered by <Trg1> captured) and E 2 is Attack-Detonate (<Trg2> is explosion). The ontology is constructed by the DARPA KAIROS project 4 for event annotation. It defines 67 event types in a three-level hierarchy, which is richer than the ACE05 ontology with only 33 event types for sentence-level extraction. In this section, we describe our memory-enhanced neural generation-based framework (Figure 2) for extracting informative event arguments from the document. Our base model is based on a sequence-to-sequence pretrained language model for text generation. We first introduce how we leverage previously extracted events as additional context for training the text generation-based event extraction model to help the model automatically capture event dependency knowledge (Section 3.1). To explicitly help the model satisfy the global event knowledge-based constraints (e.g., it is improbable that one person would be JAILER in event A and then ATTACKER in event B), we propose a dynamic 3 Name entity mentions are recognized as more informative than nominal mentions. decoding process with world knowledge-based argument pair constraints (Section 3.2). Following Li et al. (2021), the main model of our framework is based on the pretrained encoder-decoder model BART (Lewis et al., 2020). The intuition behind using BART for the extraction task is that it is pre-trained as a denoising autoencoder  reconstruct the original input sequence. This fits our objective of extracting argument spans from the input document because the extracted arguments' tokens are from the input sequence. The generation model takes (1) context: the concatenation of the piece of text x (of document D ) containing the current event trigger 5 and the event type's corresponding template in the ontology; (2) memory store m : of previously extracted events of the same document D , as input, and learns a distribution p ( y | x, m ) over possible outputs y . The ground truth sequence y is a sequence of a template where the placeholder <arg> s are filled by 5 Up to the maximum length limit of the pre-trained model. the gold-standard argument spans of the current event. 6 p ( y | x, m ) = N (cid:89) i p ( y i | y 1: i  1 , x, m ) (1) To be more specific on building the dependency between events across the document, we use the most relevant event in the memory store m as additional context, instead of the entire memory store. To retrieve the most relevant event (i.e., a generated sequence) from the memory store m = { m 1 , m 2 , ... } , we use S-BERT (Reimers and Gurevych, 2019) for dense retrieval (i.e., retrieval with dense representations provided by NN). S-BERT is a modification of the BERT model (Devlin et al., 2019) that uses siamese and triplet network structures to obtain semantically meaningful embeddings for text sequences. We can compare the distance between two input sequences with cosine-similarity in an easier and faster way. Given a current input document piece x , we encode all of 6 The gold sequence for the 1st event in Figure 1 would be [policemen including Collin] arrested or jailed [Tamerlan T. and Dzhokhar] for <arg> crime at <arg> place 5266 the previously generated event sequences in the memory store and x . Then we calculate the similarity scores with vector space cosine-similarity and normalization: score ( m i | x ) = exp f ( x, m i ) (cid:80) m i  m exp f ( x, m i ) f ( x, m i ) = Embed ( x ) T Embed ( m i ) Afterwards, we select the m i with the highest similarity score: m R = arg max i score ( m i | x ) To summarize, the input sequence for the memory-enhanced model consists of the retrieved generated event sequence ( m R ), the template for the current event type ( T )  provided by the ontol-ogy/dataset, and the context words from the document ( x 1 , ..., x n ): <S> m R 1 , m R 2 , ..., </S> <S> T 1 , T 2 , ... </S> x 1 , x 2 , ..., x n [EOS] During training time, the memory store consists of gold-standard event sequences  while at test time, it contains real generated event sequences. The training objective is to minimize the negative log likelihood over all (( x, m R , T ) , y ) instances. Since we fix the parameters from S-BERT, the retrieval module's parameters are not updated during training. Thus the training time cost of our memory-based training is almost the same to the simple generation-based model. The constrained/dynamic decoding is an important stage in our framework. We first harvest a number of world knowledge-based event argument pairs that are probable/improbable of happening with the same entity being the argument. For example, (<Event Type: Arrest, Argument Role: JAILER > | <Event Type: Attack-Detonate, Argument Role: ATTACKER >) is an improbable pair. In the framework (Figure 2), they are called argument pairs. Then based on the argument pairs constraints, the dynamic decoding is conducted throughout the document  if one entity is decoded in an event in the earlier part of the document, it should not be decoded later in another event if the results are incompatible with the improbable argument pairs. Harvesting Global Knowledge-based Argument Pairs from the Ontology We first run an algorithm to automatically harvest all candidate argument pairs (Algorithm 1). Basically, we  First enumerate all possible event type pairs, and count how many times they co-occur in the training set (Line 26). Then we enumerate all possible argument types pairs that share the same entity type from the ontology (e.g., argument ORGANIZATION (ORG) and argument VICTIM (PER) don't have the same entity type), and count how many times both of the args are of the same entity in training docs (e.g., Dzhokhar are both DETAINEE and ATTACKER in two events in Figure 1) (Line 711). Finally we add into the set of probable argument pairs, whose normalized score is above a threshold (99% of the candidate arguments with nonzero score); and the rest into the set of improba-5267 [policemen] including [Collin] arrested or jailed >Jailer: policeman, Colin Detainee: Tamerlan T., Dzhokhar enc previously generated event sequences <S> [Dzhokhar] and  Knowledge constraint-based decoder Argument Pairs Dzhokhar and Colin  policemen   Tamerlan  <input> Arg Type: Attacker Improbable Arg. After automatic harvesting, since there is noise in the dataset as well as cases not covered, we conduct a human curation process to mark certain improbable argument pairs as probable, based on world knowledge. Finally, we obtain 1,568 improbable argument pairs and 687 probable pairs. Dynamic Decoding Process During the decoding process, we keep an explicit data structure in the memory store, to record what entities have been decoded and what argument roles they are assigned to (Figure 3). During decoding the arguments of later events in the document, assuming we are at a time step t for generating the sequence for event E i , to generate token y t , we first determine the argument role ( A k ) it corresponds to. Then we search through the memory store if there are extracted entities e that have argument role A h , where < A k , A h > is an improbable argument pair. Then when decoding to token at time step t , we decrease the probability (after softmax) of generating/extracting tokens in entity e according to the improbable argument pair rule. Compared to decreasing the probability of extracting certain conflicting entities, we are more reserved in utilizing the probable argument pairs, only if the same entity has been assigned the argument role for more than 5 times in the document, we are increasing the probability of extracting the same entity (generat-ing the token of the entity) for the corresponding argument role (the most co-occurred). After the generation process for the current event, we add the newly generated event sequence (ex-tracted arguments) back into the memory store. We conduct evaluations on the newly released WIKIEVENTS dataset (Li et al., 2021). As compared to the ACE05 7 sentence-level extraction benchmark, WIKIEVENTS focuses on annotations for informative arguments and for multiple events in the document-level event extraction setting, and is the only benchmark dataset for this purpose to now. It contains real-world news articles annotated with the DARPA KAIROS ontology. As shown in the dataset paper, the distance between informative arguments and event trigger is 10 times larger than the distance between local/uninformative arguments (including pronouns) and event triggers. This demonstrates more needs for modeling long document context and event dependency and thus it requires a good benchmark for evaluating our proposed models. The statistics of the dataset are shown in Table 2. We use the same data split and preprocessing step as in the previous work. As for evaluation, we use the same criteria as in previous work. We consider an argument span to be correctly identified if its offsets match any of the gold/reference informative arguments of the current event (i.e., argument identification); and it is correctly classified if its semantic role also matches (i.e., argument classification) (Li et al., 2013). To judge whether the extracted argument and the gold-standard argument span match, since the exact match is too strict that some correct candidates are considered as spurious (e.g., the 22 policemen and 22 policemen do not match under the exact match standard). Following Huang and Riloff (2012); Li et al. (2021), we use head word match 7 http://www.itl.nist.gov/iad/mig/tests/ace/2005/ 5268 Argument Identification Argument Classification Models Head Match Coref Match Head Match Coref Match P R F1 P R F1 P R F1 P R F1 BERT-CRF (Shi and Lin, 2019) -52.71 -58.12 -43.29 -47.70 BART-Gen (Li et al., 2021) 58.62 55.64 57.09 62.84 59.64 61.19 54.02 51.27 52.61 57.47 54.55 55.97 Memory-based Training 61.07 56.18 58.52 66.21 60.91 63.45 55.93 51.45 53.60 60.47 55.64 57.95 w/ knowledge constrained decoding 62.45 56.55 59.35 67.67 61.27 64.31  57.23 51.82 54.39 61.85 56.00 58.78  Table 3: Performance (%) on the informative argument extraction task. F1 ( Head F1 ). We also report performance under a more lenient metric  Coref F1 : the extracted argument span gets full credit if it is coreferential with the gold-standard arguments (Ji and Grishman, 2008a). The coreference links information between informative arguments across the document are given in the gold annotations. We compare our framework to a number of competitive baselines. (Shi and Lin, 2019) is a popular baseline for semantic role labeling (predicate-argument prediction). It performs sequence labeling based on automatically extracted features from BERT (Devlin et al., 2019) and uses Conditional Random Fields (Lafferty et al., 2001) for structured prediction ( BERT-CRF ). Li et al. (2021) propose to use conditional neural text generation model for the document-level argument extraction problem, it handles each event in isolation ( BART-Gen ). For our proposed memory-enhanced training with retrieved additional context, we denote it as Memory-based Training . We also present the argument pairs constrained decoding results separately to see both components' contributions. 8 In Table 3, we present the main results for the document-level informative argument extraction. The score for argument identification is strictly higher than argument classification since it only requires span offset match. We observe that:  The neural generation-based models (BART-Gen and our framework) are superior in this document-level informative argument extraction problem, as compared to the sequence labeling-based approaches. Plus, generation-based methods only require one pass as 8 All significance tests for F-1 are computed using the paired bootstrap procedure of 5k samples of generated sequences (Berg-Kirkpatrick et al., 2012). compared to span enumeration-based methods (Wadden et al., 2019; Du and Cardie, 2020). As compared to the raw BART-Gen, with our memory-based training  leveraging previously closest extracted event information substantially helps increase precision (P) and F-1 scores, with smaller but notable improvement in recall especially under Coref Match. With additional argument pair constrained decoding, there is an additional significant improvement in precision and F-1 scores. This can be mainly attributed to two factors: (I) during constrained decoding, we relied more on improbable arg. pairs as a checklist to make sure that the same entity not generated for conflicting argument roles in the same document, and only utilize very few top proba-ble arg. pairs for promoting the decoding for frequently appearing entities; (II) If an entity has been decoded in previous event A by mistake then under the argument pair rule, it will not be decoded in event B even if it correct  which might hurt the recall. Robustness to Adversarial Examples To test how the models react to specially designed adversarial examples, we select a quarter of documents from the original test set, and add one more adversarial event into each of them by adding a few 5269 new sentences. The additional event is designed to attract the model to make mistakes that are against our global knowledge-based argument pair rules. 9 An excerpt for one example: Tandy, then 19, talks to his close friend, Stephen Silva, about ... Tandy and Silva both died as lifeguards together at the Harvard pool. Later a kid was killed by a Stephen Silva-lookalike guy. In this example, we know Stephen Silva died in the second event Life.Die triggered by died . Although it is also mentioned in the last sentence, Stephen Silva should not be extracted as the KILLER . In Table 4, we summarize the F-1 scores of argument classification models. Firstly we see on the adversarial examples, the performance scores all drop as compared to the normal setting (Table 3), proving it's harder to maintain robustness in this setting. Our best model with argument pair constrained decoding outperforms substantially both BART-Gen and our memory-based training model. The gap is larger than the general evaluation setting, which shows the advantage of explicitly enforcing the reasoning/constraint rules. Influence of Similarity-based Retrieval In Table 5, we first investigate what happens when our similarity-based retrieval module is removed  we find that the F-1 scores substantially drop. There's also a drop of scores across metrics when we retrieve a random event from the memory store. It is interesting that the model gets slightly better performance with random memory than not using any retrieved/demonstration sequences. This corresponds to the findings in other domains of NLP on how demonstrations lead to performance gain when using pre-trained language models (especially in the few-shot learning setting). Document Length and # of Events In Figure 4, we examine how performances change as the document length and the number of events per document grow. First we observe that as the document length grows, challenges grow for both the baseline and 9 In our open-sourced repository, readers will be able to find our designed adversarial examples under the data folder. our framework (F-1 drops from over 70% to around 55%). While our framework maintains a larger advantage when document is longer than 250 words. As the number of events per document grows (from <=8 to around 25), our model's performance is not affected much (F-1 all over 60%). While the baseline system's F-1 score drops to around 50%. Qualitative Analysis We present a couple of representative examples (Table 6). In the first example, for the event triggered by wounds , it's hard to find the VICTIM argument Ahmad Khan Rahimi since it's explicitly mentioned far before the current sentence. But with retrieved additional context, both our framework variants are able to extract the full name correctly. In the second example, Cuba was mentioned in two sentences with two events (Impede event triggered by sidesteps and Arrest triggered by capture ). But it only participated in the first event. According to our argument pair 5270 BART-Gen Baseline Memory-enhanced Training w/ Constrained Decoding Input Doc. constraints  it's improbable that one entity is both an IMPEDER and a JAILER , our framework with constrained decoding conducts reasoning to avoid the wrong extraction. Error Analysis and Remaining Challenges Table 7 categorizes types of argument extraction errors made by our best model. The majority of errors is from missing arguments and only around 7% of cases are caused by incorrectly-assigned argument roles (e.g., a PLACE argument is mistakenly labeled as a TARGET argument). Interestingly, from Figure 5's distribution, we see that as compared to the distance of gold-standard informative arguments to the trigger (avg. 80.41 words), the missing arguments are far away (avg. 136.39 words)  showing the hardness of extracting distant arguments as compared to local arguments. Finally we examine deeper the example predictions and categorize reasons for errors into the following types: (1) Challenge to obtain an accurate boundary of the argument span. In the example excerpt On Sunday, a suicide bombing in the southeastern province of [Logar] left eight ..., our model extracts southeastern province as PLACE . Similarly in ... were transported to [Kabul] city.., our model extracts city as DESTINATION . In both cases the model gets no credit. To mitigate this problem, models should be able to identify certain noun phrase boundaries with external knowledge. Plus, the improvement of data annotation and evaluation is also needed  the model should get certain credit though the span does not overlap but related to the gold argument. (2) Long distance dependency and deeper context understanding. In news, most of the contents are written by the author while certain content is cited from participants. While models usually do not distinguish the difference and consider the big stance difference. In the excerpt Bill Richard, whose son, Martin, was the youngest person killed in the bombing, said Tsarnaev could have backed out ... Instead, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Richard (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) said, (cid:58)(cid:58) he (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) chose (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) hate. (cid:58)(cid:58)(cid:58) he (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) chose (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) destruction. (cid:58)(cid:58)(cid:58) He (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) chose (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) death. (cid:58)(cid:58) ..., the full name of the informative argument (D. Tsarnaev) was mentioned at the very beginning of the document. Although our model can leverage previously decoded events, it is not able to fully understand the speaker's point of view and misses the full KILLER argument span. edge with heuristic-based rules or crowdsourcing-based methods. Sap et al. (2019) propose to use crowdsourcing for obtaining if-then relations between events. Bosselut et al. (2019) use generative language models to generate new event knowledge based on crowdsourced triples. Yao et al. (2020) propose a weakly-supervised approach to extract sub-event relation tuples from the text. In our work, we focus on harvesting knowledge-based event argument pair constraints from the predefined ontology with training data co-occurrence statistics. Plus, the work above on knowledge acquisition has not investigated explicitly encoding the knowl-edge/constraints for improving the performance of models of document-level event extraction related tasks. Document-level Event Extraction Event extraction has been mainly studied under the document-level setting (the template filling tasks from the MUC conferences (Grishman and Sundheim, 1996)) and the sentence-level setting (using the ACE data (Doddington et al., 2004) and BioNLP shared tasks (Kim et al., 2009)). In this paper, we focus on the document-level event argument extraction task which is a less-explored and challenging topic (Du et al., 2021; Li et al., 2021). To support the progress for the problem, Ebner et al. (2020) built RAMS dataset, and it contains annotations for cross-sentence arguments but for each document it contains only one event. Later Li et al. (2021) built the benchmark WIKIEVENTS with complete event annotations for each document. Regarding the methodology, neural text generation-based models have been proved to be superior at this document-level task (Huang et al., 2021; Du et al., 2021; Li et al., 2021). But they are still limited by the maximum length context issue and mainly focus on modeling one event at a time. Yang and Mitchell (2016) proposed a joint extraction approach that models cross-event dependencies  but it's restricted to events co-occurring within a sentence and only does trigger typing. In our framework, utilizing the memory store can help better capture global context and avoid the document length constraint. Apart from event extraction, in the future, it's worth investigating how to leverage the global memory idea for other document-level IE problems like ( N ary) relation extraction (Quirk and Poon, 2017; Yao et al., 2019). In this work, we examined the effect of global document-level memory on informative event argument extraction. In the new framework, we propose to leverage the previously extracted events as additional context to help the model learn the dependency across events. At test time, we propose to use a dynamic decoding process to help the model satisfy global knowledge-based argument constraints. Experiments demonstrate that our approach achieves substantial improvements over prior methods and has a larger advantage when document length and events number increase. For future work, we plan to investigate how to extend our method to multi-document event extraction cases. We thank the anonymous reviewers helpful suggestions. This research is based upon work supported by U.S. DARPA KAIROS Program No. FA8750-19-2-1004, U.S. DARPA AIDA Program No. FA8750-18-2-0014 and LORELEI Program No. HR0011-15-C-0115. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
abstain abstain objective result abstain abstain abstain abstain other abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method abstain method method other abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain method method method method method abstain abstain abstain abstain abstain method method method abstain other abstain method abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain other abstain other other other other method other other abstain other other other other other method other abstain objective objective objective objective other other other other other other other