Document-level event extraction aims to recognize event information from a whole piece of article. Existing methods are not effective due to two challenges of this task: a) the target event arguments are scattered across sentences; b) the correlation among events in a document is non-trivial to model. In this paper, we propose Heterogeneous G raph-based I nteraction Model with a T racker (GIT ) to solve the aforementioned two challenges. For the first challenge, GIT constructs a heterogeneous graph interaction network to capture global interactions among different sentences and entity mentions. For the second, GIT introduces a Tracker module to track the extracted events and hence capture the interdependency among the events. Experiments on a large-scale dataset (Zheng et al., 2019) show GIT outperforms the existing best methods by 2.8 F1. Further analysis reveals GIT is effective in extracting multiple correlated events and event arguments that scatter across the document. Our code is available at https: //github.com/RunxinXu/GIT . Event Extraction (EE) is one of the key and challenging tasks in Information Extraction (IE), which aims to detect events and extract their arguments from the text. Most previous methods (Chen et al., 2015; Nguyen et al., 2016; Liu et al., 2018; Yang et al., 2019; Du and Cardie, 2020b) focus on sentence-level EE, extracting events from a single sentence. The sentence-level model, however, fails to extract events whose arguments spread in multiple sentences, which is much more common in real-world scenarios. Hence, extracting events at the document-level is critical. It has attracted much attention recently (Yang et al., 2018; Zheng et al., 2019; Du and Cardie, 2020a; Du et al., 2020). * Corresponding author. [1] On Nov 6, 2014 , the company received a letter of share reduction from Mingting Wu , the shareholder of the company. [2] Mingting Wu decreased his holding of 7.2 million shares of the company on the Shenzhen Stock Exchange on Nov 6, 2014 . [3] The 7.2 million shares of the company Mingting Wu reduced this time were transferred to Xiaoting Wu . [4] Xiaoting Wu is the daughter of Mingting Wu , and they were identified as persons acting in concert according to relevant regulations. EventTypeEquityHolderTradedShares StartDate 7.2 million Nov 6, 2014 Xiaoting Wu EO EU Mingting Wu 7.2 million Nov 6, 2014    Figure 1: An example document from a Chinese dataset proposed by Zheng et al. (2019) in the financial domain, and we translate it into English for illustration. Though promising, document-level EE still faces two critical challenges. Firstly , the arguments of an event record may scatter across sentences, which requires a comprehensive understanding of the cross-sentence context. Figure 1 illustrates an example that one Equity Underweight (EU) and one Equity Overweight (EO) event records are extracted from a financial document. It is less challenging to extract the EU event because all the related arguments appear in the same sentence (Sentence 2 ). However, for the arguments of EO record, Nov 6, 2014 appears in Sentence 1 and 2 while Xiaoting Wu in Sentence 3 and 4 . It would be quite challenging to identify such events without considering global interactions among sentences and entity mentions. Secondly , a document may express several correlated events simultaneously, and recognizing the interdependency among them is fundamental to successful extraction. As shown in Figure 1, the two events are interdependent because they correspond to exactly the same transaction and therefore share the same StartDate . Effective modeling on such interdependency among the correlated events remains a key challenge in this task. Yang et al. (2018) extracts events from a central sentence and query the neighboring sentences for missing arguments, which ignores the cross-sentence correspondence between augments. Though Zheng et al. (2019) takes a first step to fuse the sentences and entities information via Transformer, they neglect the interdependency among events. Focusing on single event extraction, Du and Cardie (2020a) and Du et al. (2020) concatenate multiple sentences and only consider a single event, which lacks the ability to model multiple events scattered in a long document. To tackle the aforementioned two challenges, in this paper, we propose a Heterogeneous G raph-based I nteraction Model with a T racker (GIT ) for document-level EE. To deal with scattered arguments across sentences, we focus on the Global Interactions among sentences and entity mentions. Specifically, we construct a heterogeneous graph interaction network with mention nodes and sentence nodes, and model the interactions among them by four types of edges (i.e., sentence-sentence edge, sentence-mention edge, intra-mention-mention edge, and inter-mention-mention edge) in the graph neural network. In this way, GIT jointly models the entities and sentences in the document from a global perspective. To facilitate the multi-event extraction, we target on the Global Interdependency among correlated events. Concretely we propose a Tracker module to continually tracks the extracted event records with a global memory. In this way, the model is encouraged to incorporate the interdependency with other correlated event records while predicting. We summarize our contributions as follows:  We construct a heterogeneous graph interaction network for document-level EE. With different heterogeneous edges, the model could capture the global context for the scattered event arguments across different sentences. We introduce a novel Tracker module to track the extracted event records. The Tracker eases the difficulty of extracting correlated events, as interdependency among events would be taken into consideration. Experiments show GIT outperforms the previous state-of-the-art model by 2 . 8 F1 on the large-scale public dataset (Zheng et al., 2019) with 32 , 040 documents, especially on cross-sentence events and multiple events scenarios (with 3 . 7 and 4 . 9 absolute increase on F1). We first clarify some important notions. a) entity mention : a text span within document that refers to an entity object; b) event argument : an entity playing a specific event role. Event roles are predefined for each event type; c) event record : an entry of a specific event type containing arguments for different roles in the event. For simplicity, we use record for short in the following sections. Following Zheng et al. (2019), given a document composed of sentences D = { s i } |D| i =1 and a sentence containing a sequence of words s i = { w j } | s i | j =1 , the task aims to handle three sub-tasks : 1) entity extraction : extracting entities E = { e i } |E| i =1 from the document to serve as argument candidates. An entity may have multiple mentions across the document. 2) event types detection : detecting specific event types that are expressed by the document. 3) event records extraction : find-ing appropriate arguments for the expressed events from entities, which is the most challenging and also the focus of our paper. The task does not require to identify event triggers (Zeng et al., 2018; Liu et al., 2019b), which reduces manual effort of annotation and the application scenarios becomes more extensive. As shows in Figure 2, GIT first extracts candidate entities through sentence-level neural extractor (Sec 3.1). Then we construct a heterogeneous graph to model the interactions among sentences and entity mentions (Sec 3.2), and detect event types expressed by the document (Sec 3.3). Finally we introduce a Tracker module to continuously track all the records with global memory, in which we utilize the global interdependency among records for multi-event extraction (Sec 3.4). corresponding token and position embeddings. We extract entities at the sentence level and formulate it as a sequence tagging task with BIO (Be-gin, Inside, Other) schema. We leverage a conditional random field (CRF) layer to identify entities. For training, we minimize the following loss: L ner =  (cid:88) s D log P ( y s | s ) (1) where y s is the golden label sequence of s . An event may span multiple sentences in the document, which means its corresponding entity mentions may also scatter across different sentences. Identifying and modeling these entity mentions in the cross-sentence context is fundamental in document EE. Thus we build a heterogeneous graph G which contains entity mention nodes and sentence nodes in the document D . In the graph G , interactions among multiple entity mentions and sentences can be explicitly modeled. For each entity mention node e , we initialize node embedding h (0) e = Mean ( { g j } j  e ) by averaging the representation of the contained words. For each sentence node s , we initialize node embedding h (0) s = Max ( { g j } j  s ) + SentPos ( s ) by max-pooling all the representation of words within the sentence plus sentence position embedding. To capture the interactions among sentences and mentions , we introduce four types of edges. Sentence-Sentence Edge (S-S) Sentence nodes are fully connected to each other with S-S edges. In this way, we can easily capture the global properties in the document with sentence-level interactions, e.g., the long range dependency between any two separate sentences in the document would be modeled efficiently with S-S edges. Sentence-Mention Edge (S-M) We model the local context of an entity mention in a specific sentence with S-M edge, specifically the edge connecting the mention node and the sentence node it belongs to. Intra-Mention-Mention Edge (M-M intra ) We connect distinct entity mentions in the same sentences with M-M intra edges. The co-occurrence of mentions in a sentence indicates those mentions are likely to be involved in the same event. We explicitly model this indication by M-M intra edges. Inter-Mention-Mention Edge (M-M inter ) The entity mentions that corresponds to the same entity are fully connected with each other by M-M inter edges. As in document EE, an entity usually corresponds to multiple mentions across sentences, we thus use M-M inter edge to track all the appearances of a specific entity, which facilitates the long distance event extraction from a global perspective. In Section. 4.5, experiments show that all of these four kinds of edges play an important role in event detection, and the performance would decrease without any of them. After heterogeneous graph construction * , we apply multi-layer Graph Convolution Network (Kipf and Welling, 2017) to model the global interactions inspired by Zeng et al. (2020). Given node u at the l -th layer, the graph convolutional operation is defined as follows: h ( l +1) u = ReLU  (cid:88) k K (cid:88) v N k ( u ) (cid:83) { u } 1 c u,k W ( l ) k h ( l ) v   where K represents different types of edges, W ( l ) k  R d m  d m is trainable parameters. N k ( u ) denotes the neighbors for node u connected in k -th type edge and c u,k is a normalization constant. We then derive the final hidden state h u for node u , h u = W a [ h (0) u ; h (1) u ; . . . ; h ( L ) u ] where h (0) u is the initial node embedding of node u , and L is the number of GCN layers. Finally, we obtain the sentence embedding matrix S = [ h (cid:62) 1 h (cid:62) 2 . . . h (cid:62)|D| ]  R d m |D| and entity embedding matrix E  R d m |E| . The i -th entity may have many mentions, where we simply use string matching to detect entity coreference following Zheng et al. (2019) , and the entity embedding E i is computed by the average of its mention node embedding, E i = Mean ( { h j } j  Mention( i ) ) . In this way, the sentences and entities are interactively represented in a context-aware way. where Q  R d m  T and W t  R d m are trainable parameters, and T denotes the number of possible event types. MultiHead refers to the standard multi-head attention mechanism with Query/Key/Value. Therefore, we derive the event types detection loss with golden label (cid:98) R  RT : L detect =  T (cid:88) t =1 I (cid:16) (cid:98) R t = 1 (cid:17) log P ( R t |D ) + I (cid:16) (cid:98) R t = 0 (cid:17) log (1  P ( R t |D )) (2) 3.4 Event Records Extraction Since a document is likely to express multiple event records and the number of records cannot be known in advance, we decode records by expanding a tree orderly as previous methods did (Zheng et al., 2019). However, they treat each record independently. Instead, to incorporate the interdependency among event records , we propose a Tracker module, which improves the model performance. we extract event records of a specific event type. The arguments extraction order is predefined so that the extraction is modeled as a constrained tree expanding task  . Taking Equity Freeze records as an example, as shown in Figure 3, we firstly extract EquityHolder , followed by FrozeShares and others. Starting from a virtual root node, the tree expands by predicting arguments in a sequential order. As there may exist multiple eligible entities for the event argument role, the current node will expand several branches during extraction, with different entities assigned to the current role. This branching operation is formulated as multi-label classification task. In this way, each path from the root node to the leaf node is identified as a unique event record. Interdependency exists extensively among different event records. For example, as shown in Figure 1, an Equity Underweight event record is closely related to an Equity Overweight event record, and they may share some key arguments or provide useful reasoning information. To take advantage of such interdependency, we propose a novel Tracker module inspired by memory network (Weston et al., 2015). Intuitively, the Tracker continually tracks the extracted records on-the-fly and store the information into a global memory. When predicting arguments for current record, the model will query the global memory and therefore make use of useful interdependency information of other records. In detail, for the i -th record path consisting of a sequence of entities, the Tracker encodes the corresponding entity representation sequence U i = [ E i 1 , E i 2 , ... ] into an vector G i with an LSTM (last hidden state) and add event type embedding. Then the compressed record information is stored in the global memory G , which is shared across different event types as shown in Figure 3. For extraction, given a record path U i  R d m  ( J  1) with the first J  1 arguments roles, we predict the J -th role by injecting role-specific information into entity representations, E = E + Role J , where Role J is the role embedding for the J -th role. Then we concatenate E , sentences feature S , current entities path U i , and the global memory G , followed by a transformer to obtain new entity feature matrix (cid:101) E  R d m |E| , which contains global role-specific  We simply adopt the order used by Zheng et al. (2019). We treat the path expansion as a multi-label classification problem with a binary classifier over (cid:101) E i , i.e., predicts whether the i -th entity is the next argument role for the current record and expand the path accordingly as shown in Figure 3. where ND denotes the nodes set in the event records tree, and y nt is the golden label. If the t -th entity is validate for the next argument in node n , then y nt = 1 , otherwise y nt = 0 . More training details are shown in Appendix A. We evaluate our model on a public dataset proposed by Zheng et al. (2019)  , which is constructed from Chinese financial documents. It consists of up to 32 , 040 documents which is the largest document-level EE dataset by far. It focuses on five event types: Equity Freeze (EF), Equity Repurchase (ER), Equity Underweight (EU), Equity Overweight (EO) and Equity Pledge (EP), with 35 different kinds of argument roles in total. We follow the standard split of the dataset, 25 , 632 / 3 , 204 / 3 , 204 documents for training/dev/test set. The dataset is quite challenging, as a document has 20 sentences and consists of 912 tokens on average. Besides, there are roughly 6 sentences involved for an event record, and 29% documents express multiple events. In our implementation of GIT , we use 8 and 4 layers Transformer (Vaswani et al., 2017) in encoding and decoding module respectively. The dimensions in hidden layers and feed-forward layers are the same as previous work (Zheng et al., 2019), i.e., 768 and 1 , 024 . We also use L = 3 layers of GCN, and set dropout rate to 0 . 1 , batch size to 64 . GIT is trained using Adam (Kingma and Ba, 2015) as optimizer with 1 e  4 learning rate for 100 epochs. We set  1 = 0 . 05 ,  2 =  3 = 1 for the loss function. Yang et al. (2018) proposes DCFEE that extracts arguments from the identified central sentence and queries surrounding sentences for missing arguments. The model has two variants, DCFEE-S and DCFEE-M . DCFEE-S produces one record at a time, while DCFEE-M produces multiple possible argument combinations by the closest distance from the central sentence. Besides, Doc2EDAG (Zheng et al., 2019) uses transformer encoder to obtain sentence and entity embeddings, followed by another transformer to fuse cross-sentence context. Then multiple events are extracted simultaneously. Greedy-Dec is a variant of Doc2EDAG, which produces only one record greedily. Three sub-tasks of the document-level EE are all evaluated by F1 score. Due to limited space, we leave the results of entity extraction and event types detection in Appendix B, which shows GIT only slightly outperform Doc2EDAG, because we mainly focus on event record extraction and the methods are similar to Doc2EDAG for these two sub-tasks. In the following, we mainly report and analyze the results of event record extraction . Overall performance . The results of the overall performance on the document-level EE dataset is illustrated in Table 1. As Table 1 shows, our GIT consistently outperforms other baselines, thanks to better modelling of global interactions and interdependency. Specifically, GIT improves 2 . 8 micro F1 compared with the previous state-of-the-art, Doc2EDAG, especially 4 . 5 improvement in Equity Underweight (EU) event type. Cross-sentence records scenario. There are more than 99 . 5% records of the test set are cross-sentence event records, and the extraction becomes gradually more difficult as the number of their involved sentences grows. To verifies the effectiveness of GIT to capture cross-sentence information, we first calculate the average number of sentences that the records involve for each document, and sort them in ascending order. Then we divide them into four sets I/II/III/IV with equal size. Documents in Set. IV is considered to be the most challenging as it requires the most number of sentences to successfully extract records. As Table 2 shows, GIT consistently outperforms Doc2EDAG, especially on the most challenging Set. IV that involves the most sentences, by 3 . 7 F1 score. It suggests that GIT can well capture global context and mitigate the arguments-scattering challenge, with the help of the heterogeneous graph interaction network. Multiple records scenario. GIT introduces the tracker to make use of global interdependency among event records, which is important in multiple records scenario. To illustrate its effectiveness, we divide the test set into single-record set (S.) containing documents with one record, and multi-record set (M.) containing those with multiple records. As shown in Table. 3, F1 score on M. Model EF ER EU EO EP Overall S. M. S. M. S. M. S. M. S. M. S. M. DCFEE-S 55.7 38.1 83.0 55.5 52.3 41.4 49.2 43.6 62.4 52.2 69.0 50.3 DCFEE-M 45.3 40.5 76.1 50.6 48.3 43.1 45.7 43.3 58.1 51.2 63.2 49.4 Greedy-Dec 74.0 40.7 82.2 50.0 61.5 35.6 63.4 29.4 78.6 36.5 77.8 37.0 Doc2EDAG 79.7 63.3 90.4 70.7 74.7 63.3 76.1 70.2 84.3 69.3 81.0 67.4 GIT (ours) 81.9 65.9 93.0 71.7 82.0 64.1 80.9 70.6 85.0 73.5 87.6 72.3 Table 3: F1 scores on single-record (S.) and multi-record (M.) sets. is much lower than that on S., indicating it is challenging to extract multiple records. However, GIT still surpasses other strong baselines by 4 . 9  35 . 3 on multi-record set (M.). This is because GIT is aware of other records through the T racker module, and leverage the interdependency information to improve the performance  . Nguyen et al. (2016) maintain three binary matrices to memorize entities and events states. Although they aim at sentence-level EE that contains fewer entities and event records, it would be also interesting to compare with them and we leave it as future work. We conduct further experiments to analyze the key modules in GIT more deeply. On the effect of heterogeneous graph interaction network . The heterogeneous graph we constructed contains four types of edges. To explore their functions, we remove one type of edges at a time, and remove the whole graph network finally. Results are shown in Table 4, including micro F1 and F1 on the four sets, which are divided by the number of involved sentences for records as we did before. The micro F1 would decreases 1 . 0  1 . 4 without a certainty type of edge. Besides, removing the whole graph causes an significant drop by 2 . 0 F1, especially for Set IV by 2 . 5 , which requires the most number of sentences to extract the event record. It demonstrates that the graph interaction network helps improve the performance, especially on records involving many sentences, and all kinds of edges play an important role for extraction. On the effect of Tracker module . GIT can leverage interdependency among records based on the information of other event records tracked by Tracker . To explore its effect, firstly, we remove the global interdependency information between records of different event types, by clearing the global memory whenever we extract events for an[5] The shareholder of the company, Quanlie Chen , pledged 52.4 million to GDZQ Co., Ltd. in 2018, and supplemented the pledge recently because of the decline of the share price. Next, we remove all the tracking information except the own path for a record, to explore whether the tracking of other records makes effect indeed (GITO wn P ath). Finally, we remove the whole Tracker module (GITN o T racker). As Table 5 shows, the F1 in GIT -OT/G IT-OP decreases by 0 . 5 / 1 . 2 , suggesting the interdependency among records of both the same and different event types do play an essential role. Besides, their F1 decrease in M. by 0 . 7 / 1 . 5 are more than those in S. by 0 . 8 / 1 . 0 , verifying the effectiveness of the Tracker in multi-event scenarios. Moreover, the performances are similar between GIT-OP and GIT-NT, which also provides evidence that other records do help. We also reveal F1 on documents with different number of records in Figure 4. The gap between models with or without Tracker raises as the number of records increases, which validates the effectiveness of our Tracker . Figure 5 demonstrates a case of the predictions of Doc2EDAG and GIT for Equity Pledge (EP) event types. The TotalHoldingShares and TotalPledgedShares information lies in Sentence 8 , while the PledgedShares and Pledgee information for Record 2 lies in Sentence 5 . Though Doc2EDAG fails to extract these arguments in Record 2 (colored in red), GIT succeeds because it can capture interactions between long-distance sentences, and utilize the information of Record 1 ( 325.4 million and 218.6 million ) thanks to the Tracker model. Sentence-level Event Extraction . Previous approaches mainly focus on sentence-level event extraction. Chen et al. (2015) propose a neural pipeline model that identifies triggers first and then extracts argument roles. Nguyen et al. (2016) use a joint model to extract triggers and argument roles simultaneously. Some studies also utilize dependency tree information (Liu et al., 2018; Yan et al., 2019). To utilize more knowledge, some studies leverage document context (Chen et al., 2018; Zhao et al., 2018), pre-trained language model (Yang et al., 2019), and explicit external knowledge (Liu et al., 2019a; Tong et al., 2020) such as WordNet (Miller, 1995). Du and Cardie (2020b) also try to extract events in a Question-Answer way. These studies usually conduct experiments on sentence-level event extraction dataset, ACE05 (Walker et al., 2006). However, it is hard for the sentence-level models to extract multiple qualified events spanning across sentences, which is more common in real-world scenarios. Document-level Event Extraction . Document-level EE has attracted more and more attention recently. Yang and Mitchell (2016) use well-defined features to handle the event-argument relations across sentences, which is, unfortunately, quite nontrivial. Yang et al. (2018) extract events from a central sentence and find other arguments from neighboring sentences separately. Although Zheng et al. (2019) use Transformer to fuse sentences and entities, interdependency among events is neglected. Du and Cardie (2020a) try to encode the sentences in a multi-granularity way and Du et al. (2020) leverage a seq2seq model. They conduct experiments on MUC-4 (Sundheim, 1992) dataset with 1 , 700 documents and 5 kinds of entity-based arguments, and it is formulated as a table-filling task, coping with single event record of single event type. However, our work is different from these studies in that a) we utilize heterogeneous graph to model the global interactions among sentences and mentions to capture cross-sentence context, b) and we leverage the global interdependency through Tracker to extract multiple event records of multiple event types. Although promising in practical application, document-level EE still faces some challenges such as arguments-scattering phenomenon and multiple correlated events expressed by a single document. To tackle the challenges, we introduce Heterogeneous G raph-based I nteraction Model with a T racker (GIT ). GIT uses a heterogeneous graph interaction network to model global interactions among sentences and entity mentions. GIT also uses a Tracker to track the extracted records to consider global interdependency during extraction. Experiments on large-scale public dataset (Zheng et al., 2019) show GIT outperforms previous state-of-the-art by 2 . 8 F1. Further analysis verifies the effectiveness of GIT especially in cross-sentence events extraction and multi-event scenarios. The authors would like to thank Changzhi Sun, Mingxuan Wang, and the anonymous reviewers for their thoughtful and constructive comments. This paper is supported in part by the National Key R&D Program of China under Grand No.2018AAA0102003, the National Science Foundation of China under Grant No.61936012 and 61876004.
abstain abstain abstain abstain objective abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective method method abstain abstain objective abstain objective objective objective objective objective objective abstain abstain abstain abstain abstain method other abstain abstain abstain abstain abstain method method abstain method method method abstain abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method other abstain method abstain method abstain abstain abstain other abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain other other other other other other other other other other other other other other other other abstain method method abstain method abstain abstain abstain abstain abstain other other