Most existing joint neural models for Information Extraction (IE) use local task-specific classifiers to predict labels for individual instances (e.g., trigger, relation) regardless of their interactions. For example, a VICTIM of a DIE event is likely to be a VICTIM of an ATTACK event in the same sentence. In order to capture such cross-subtask and cross-instance inter-dependencies, we propose a joint neural framework, ONEIE, that aims to extract the globally optimal IE result as a graph from an input sentence. ONEIE performs end-to-end IE in four stages: (1) Encoding a given sentence as contextualized word representations; (2) Identifying entity mentions and event triggers as nodes; (3) Computing label scores for all nodes and their pairwise links using local classifiers; (4) Searching for the globally optimal graph with a beam decoder. At the decoding stage, we incorporate global features to capture the cross-subtask and cross-instance interactions. Experiments show that adding global features improves the performance of our model and achieves new state-of-the-art on all subtasks. As ONEIE does not use any language-specific feature, we prove it can be easily applied to new languages or trained in a multilingual manner. Our code and models for English, Spanish and Chinese are publicly available for research purpose 1 . Information Extraction (IE) aims to extract structured information from unstructured texts. It is a complex task comprised of a wide range of subtasks, such as named, nominal, and pronominal mention extraction, entity linking, entity coreference resolution, relation extraction, event extraction, and event coreference resolution. Early efforts typically perform IE in a pipelined fashion, 1 http://blender.cs.illinois.edu/software/ oneie which leads to the error propagation problem and disallows interactions among components in the pipeline. As a solution, some researchers propose joint inference and joint modeling methods to improve local prediction (Roth and Yih, 2004; Ji and Grishman, 2005; Ji et al., 2005; Sil and Yates, 2013; Li et al., 2014; Durrett and Klein, 2014; Miwa and Sasaki, 2014; Lu and Roth, 2015; Yang and Mitchell, 2016; Kirschnick et al., 2016). Due to the success of deep learning, neural models have been widely applied to various IE subtasks (Col-lobert et al., 2011; Chiu and Nichols, 2016; Chen et al., 2015; Lin et al., 2016). Recently, some efforts (Wadden et al., 2019; Luan et al., 2019) revisit global inference approaches by designing neural networks with embedding features to jointly model multiple subtasks. However, these methods use separate local task-specific classifiers in the final layer and do not explicitly model the interdependencies among tasks and instances. Figure 1 shows a real example where the local argument role classifier predicts a redundant PERSON edge. The model should be able to avoid such mistakes if it is capable of learning and leveraging the fact that it is unusual for an ELECT event to have two PERSON arguments. Example: Prime Minister Abdullah Gul resigned earlier Tuesday to make way for Erdogan , who won a parliamentary seat in by-elections Sunday. To address this issue, we propose a joint neu-earthquake killed The 19 people and injured 300 in Kashmir region , India Identification Classification Trigger Entity Role Relation Decoding Encoding Die PER victim Injure Die PER victim Injure Die PER victim Injure ORG PER victim victim  Injure-victimORG  Injure-victimPER ... ... ral framework, ONEIE, to perform end-to-end IE with global constraints. As Figure 2 shows, instead of predicting separate knowledge elements using local classifiers, ONEIE aims to extract a globally optimal information network for the input sentence. When comparing candidate information networks during the decoding process, we not only consider individual label scores for each knowledge element, but evaluate cross-subtask and cross-instance interactions in the network. In this example, a graph with the INJURE-VICTIM-ORG (the VICTIM of an INJURE event is an ORG entity) structure is demoted. Experiments show that our framework achieves comparable or better results compared to the state-of-the-art end-to-end architecture (Wadden et al., 2019). To the best of our knowledge, ONEIE is the first end-to-end neural IE framework that explicitly models cross-subtask and cross-instance interdependencies and predicts the result as a unified graph instead of isolated knowledge elements. Because ONEIE does not rely on language-specific features, it can be rapidly applied to new languages. Furthermore, global features in our framework are highly explainable and can be explicitly analyzed. Given a sentence, our ONEIE framework aims to extract an information network representation (Li et al., 2014), where entity mentions and event triggers are represented as nodes, and relations and event-argument links are represented as edges. In other words, we perform entity, relation, and event extraction within a unified framework. In this section, we will elaborate these tasks and involved terminologies. Entity Extraction aims to identify entity mentions in text and classify them into pre-defined entity types. A mention can be a name, nominal, or pronoun. For example, Kashmir region should be recognized as a location ( LOC ) named entity mention in Figure 2. Relation Extraction is the task of assigning a relation type to an ordered pair of entity mentions. For example, there is a PART-WHOLE relation between Kashmir region and India. Event Extraction entails identifying event triggers (the words or phrases that most clearly express event occurrences) and their arguments (the words or phrases for participants in those events) in unstructured texts and classifying these phrases, respectively, for their types and roles. An argument can be an entity, time expression, or value (e.g., MONEY , JOB-TITLE , CRIME ). For example, in Figure 2, the word injured triggers an INJURE event and 300 is the VICTIM argument. We formulate the task of extracting information networks as follows. Given an input sentence, our goal is to predict a graph G = ( V, E ) , where V and E are the node and edge sets respectively. Each node v i = (cid:104) a i , b i , l i (cid:105)  V represents an entity mention or event trigger, where a and b are the start and end word indices, and l is the node type label. Each edge e ij = (cid:104) i, j, l ij (cid:105)  E is represented similarly, whereas i and j denote the indices of involved nodes. For example, in Figure 2, the trigger injured is represented as (cid:104) 7, 7, INJURE (cid:105) , the entity mention Kashmir region is represented as (cid:104) 10, 11, LOC (cid:105) , and the event-argument edge between them is (cid:104) 2, 3, PLACE (cid:105) . As Figure 2 illustrates, our ONEIE framework extracts the information network from a given sentence in four steps: encoding, identification, classification, and decoding. We encode the input sentence using a pre-trained BERT encoder (Devlin et al., 2019) and identify entity mentions and event triggers in the sentence. After that, we compute the type label scores for all nodes and pairwise edges among them. During decoding, we explore possible information networks for the input sentence using beam search and return the one with the highest global score. Given an input sentence of L words, we obtain the contextualized representation x i for each word using a pre-trained BERT encoder. If a word is split into multiple word pieces (e.g., Mondrian  Mon , ##dr , ##ian ), we use the average of all piece vectors as its word representation. While previous methods typically use the output of the last layer of BERT, our preliminary study shows that enriching word representations using the output of the third last layer of BERT can substantially improve the performance on most subtasks. At this stage, we identify entity mentions and event triggers in the sentence, which will act as nodes in the information network. We use a feed-forward network FFN to compute a score vector  y i = FFN( x i ) for each word, where each value in  y i represents the score for a tag in a target tag set 2 . After that, we use a conditional random fields (CRFs) layer to capture the dependencies between predicted tags (e.g., an I-PER tag should not follow a B-GPE tag). Similar to (Chiu and Nichols, 2016), we calculate the score of a tag path  z = {  z 1 , ...,  z L } as s ( X ,  z ) = L (cid:88) i =1  y i,  z i + L +1 (cid:88) i =1 A  z i  1 ,  z i , where X = { x 1 , ..., x L } is the contextualized representations of the input sequence,  y i,  z i is the  z i -th 2 We use the BIO tag scheme, in which the prefix Bmarks the beginning of a mention, and Imeans inside of a mention. A token not belonging to any mention is tagged with O . component of the score vector  y i , and A  z i  1 ,  z i is the ( z i  1 ,  z i ) entry in matrix A that indicates the transition score from tag  z i  1 to  z i . The weights in A are learned during training. We append two special tags <start> and <end> to the tag path as  z 0 and  z L +1 to denote the start and end of the sequence. At the training stage, we maximize the log-likelihood of the gold-standard tag path as log p ( z | X ) = s ( X , z )  log (cid:88)  z  Z e s ( X ,  z ) , where Z is the set of all possible tag paths for a given sentence. Thus, we define the identification loss as LI =  log p ( z | X ) . In our implementation, we use separate taggers to extract entity mentions and event triggers. Note that we do not use types predicted by the taggers. Instead, we make a joint decision for all knowledge elements at the decoding stage to prevent error propagation and utilize their interactions to improve the prediction of node type. We represent each identified node as v i by averaging its word representations. After that, we use separate task-specific feed-forward networks to calculate label scores for each node as  y ti = FFN t ( v i ) , where t indicates a specific task. To obtain the label score vector for the edge between the i -th and j -th nodes, we concatenate their span representations and calculate the vector as  y tk = FFN t ( v i , v j ) . For each task, the training objective is to minimize the following cross-entropy loss L t =  1 N t N t (cid:88) i =1 y ti log  y ti , where y ti is the true label vector and N t is the number of instances for task t . If we ignore the inter-dependencies between nodes and edges, we can simply predict the label with the highest score for each knowledge element and thus generate the locally best graph  G . The score of  G can be calculated as s (cid:48) (  G ) = (cid:88) t  T N t (cid:88) i =1 max  y ti , where T is the set of tasks. We refer to s (cid:48) (  G ) as the local score of  G . A limitation of local classifiers is that they are incapable of capturing inter-dependencies between knowledge elements in an information network. We consider two types of inter-dependencies in our framework. The first type of inter-dependency is Cross-subtask interactions between entities, relations, and events. Consider the following sentence. A civilian aid worker from San Francisco was killed in an attack in Afghanistan. A local classifier may predict San Francisco as a VICTIM argument because an entity mention preceding was killed is usually the victim despite the fact that a GPE is unlikely to be a VICTIM . To impose such constraints, we design a global feature as shown in Figure 3(a) to evaluate whether the structure DIE-VICTIM-GPE exists in a candidate graph. Another type of inter-dependency is Cross-instance interactions between multiple event and/or relation instances in the sentence. Take the following sentence as an example. South Carolina boy , 9, dies during hunting trip after his father accidentally shot him on Thanksgiving Day. It can be challenging for a local classifier to predict boy as the VICTIM of the ATTACK event triggered by shot due to the long distance between these two words. However, as shown in Figure 3(b), if an entity is the VICTIM of a DIE event, it is also likely to be the VICTIM of an ATTACK event in the same sentence. Motivated by these observations, we design a set of global feature templates (event schemas) as listed in Table 1 to capture cross-subtask and cross-instance interactions, while the model fills in all possible types to generate features and learns the (a) Cross-subtask Interaction (b) Cross-instance Interactions PER dies Die Attack boy victim victim shot Die San Francisco killed victim GPE Figure 3: Examples of inter-dependencies between elements in information networks. weight of each feature during training. Given a graph G , we represent its global feature vector as f G = { f 1 ( G ) , ..., f M ( G ) } , where M is the number of global features and f i (  ) is a function that evaluates a certain feature and returns a scalar. For example, f i ( G ) = (cid:40) 1 , G has multiple ATTACK events 0 , otherwise . Next, ONEIE learns a weight vector u  RM and calculates the global feature score of G as the dot product of f G and u . We define the global score of G as the sum of its local score and global feature score, namely s ( G ) = s (cid:48) ( G ) + uf G , We make the assumption that the gold-standard graph for a sentence should achieve the highest global score. Therefore, we minimize the following loss function LG = s (  G )  s ( G ) , where  G is the graph predicted by local classifiers and G is the gold-standard graph. Finally, we optimize the following joint objective function during training L = LI + (cid:88) t  TL t + LG 3.5 Decoding As we have discussed, because local classifiers ignore interactions among elements in an information network, they may predict contradictory results or fail to predict difficult edges that require information from other elements. In order to address these issues, ONEIE makes a joint decision for all nodes and their pairwise edges to obtain the globally optimal graph. The basic idea is to calculate the global score for each candidate graph and select the one with the highest score. However, exhaustive search is infeasible in many cases as the size of search space grows exponentially with the number of nodes. Therefore, we design a beam search-based decoder as Figure 4 depicts. Given a set of identified nodes V and the label scores for all nodes and their pairwise links, we perform decoding with an initial beam set B = { K 0 } , where K 0 is an order-zero graph. At each step i , we expand each candidate in B in node step and edge step as follows. Node step : We select v i  V and define its candidate set as V i = {(cid:104) a i , b i , l ( k ) i (cid:105)| 1  k   v } , where l ( k ) i denotes the label with the k -th highest local score for v i , and  v is a hyper-parameter that controls the number of candidate labels to consider. We update the beam set by B  { G + v | ( G, v )  B  V i } , Edge step : We iteratively select a previous node v j  V, j < i and add possible edges between v j and v i . Note that if v i is a trigger, we skip v j if it is also a trigger. At each iteration, we construct a candidate edge set as E ij = {(cid:104) j, i, l ( k ) ij (cid:105)| 1  k   e } , where l ( k ) ij is the label with k -th highest score for e ij and  e is a threshold for the number of candidate labels. Next, we update the beam set by B  { G + e | ( G, e )  B  E ij } , At the end of each edge step, if | B | is larger than the beam width  , we rank all candidates by global score in descending order and keep the top  ones. We perform our experiments on the Automatic Content Extraction (ACE) 2005 dataset 3 , which provides entity, value, time, relation, and event annotations for English, Chinese, and Arabic. Following Wadden et al. (2019)'s pre-processing 4 , we conduct experiments on two datasets, ACE05-R that includes named entity and relation annotations, and ACE05-E that includes entity, relation, and event annotations. We keep 7 entity types, 6 coarse-grained relation types, 33 event types, and 22 argument roles. In order to reinstate some important elements absent from ACE05-R and ACE05-E, we create a new dataset, ACE05-E + , by adding back the order of relation arguments, pronouns, and multi-token event triggers, which have been largely ignored in previous work. We also skip lines before the <text> tag (e.g., headline, datetime) as they are not annotated. In addition to ACE, we derive another dataset, ERE-EN, from the Entities, Relations and Events (ERE) annotation task created under the Deep Exploration and Filtering of Test (DEFT) program because it covers more recent articles. Specifi-cally, we extract 458 documents and 16,516 sentences from three ERE datasets, LDC2015E29, LDC2015E68, and LDC2015E78. For ERE-EN, we keep 7 entity types, 5 relation types, 38 event types, and 20 argument roles. To evaluate the portability of our model, we also develop a Chinese dataset from ACE2005 and a Spanish dataset from ERE (LDC2015E107). We refer to these datasets as ACE05-CN and ERE-ES respectively. We optimize our model with BertAdam for 80 epochs with a learning rate of 5e-5 and weight decay of 1e-5 for BERT, and a learning rate of 1e-3 and weight decay of 1e-3 for other parameters. We use use the bert-base-multilingual-cased 3 https://www.ldc.upenn.edu/collaborations/ past-projects/ace 4 https://github.com/dwadden/dygiepp Node Step E1 1 Candidate 1 of node E1 E1 2 Candidate 2 of node E1 Node Step E1 1 E1 1 E1 2 E1 2 T1 1 T1 2 T1 1 T1 2 E1 1 E1 1 E1 2 E1 2 T1 1 T1 2 T1 1 T1 2 E1 1 E1 1 E1 2 E1 2 T1 1 T1 2 T1 1 T1 2 Edge Step Add v 1 Add v 2 Add e 1,2 Sort FAC Fine Campbell fines PER Fine Campbell fines entity entity Example: He also brought a check from Campbell to pay the fines and fees. model 5 for ACE05-CN and ERE-ES, and use the bert-large-cased model for other datasets. Following (Wadden et al., 2019), we use two-layer FFNs with a dropout rate of 0.4 for local classifiers. We use 150 hidden units for entity and relation extraction, and 600 hidden units for event extraction. For global features, we set  v and  e to 2 and set  to 10. In our experiments, we use random seeds and report averaged scores across runs. We use the same criteria as (Zhang et al., 2019; Wadden et al., 2019) for evaluation as follows. Relation : A relation is correct if its relation type 5 https://huggingface.co/transformers/ pretrained_models.html is correct and the offsets of the related entity mentions are correct. Trigger : A trigger is correctly identified (Trig-I) if its offsets match a reference trigger. It is correctly classified (Trig-C) if its event type also matches the reference trigger. Argument : An argument is correctly identified (Arg-I) if its offsets and event type match a reference argument mention. It is correctly classified (Arg-C) if its role label also matches the reference argument mention. In Table 3, we compare our results with two models: (1) DY GIE++ (Wadden et al., 2019), the state-of-the-art end-to-end IE model that utilizes multi-sentence BERT encodings and span graph propagation; (2) BASELINE that follows the architecture of ONEIE but only uses the output of the last layer of BERT and local classifiers. We can see that our model consistently outperforms DY GIE++ and BASELINE on ACE05-R and ACE05-E. In (Wadden et al., 2019), the authors show that combining triggers predicted by a four-model ensemble optimized for trigger detection can improve the performance of event extraction. While we also report our results using a four-model ensemble in Table 4 for fair comparison, we hold the opinion that the single-model scores in Table 3 better reflect the actual performance of ONEIE and should be used for future comparison. In Table 6 we list salient global features learned by the model. Take feature #9 as an example, if a Dataset Task DY GIE++ BASELINEONEIE ACE05-R Entity 88.6 -88.8 Relation 63.4 -67.5 ACE05-E Entity 89.7 90.2 90.2 Trig-I -76.6 78.2 Trig-C 69.7 73.5 74.7 Arg-I 53.0 56.4 59.2 Arg-C 48.8 53.9 56.8 Table 3: Results on ACE2005 datasets (F-score, %). candidate graph contains multiple ORG-AFF edges incident to the same node, the model will demote this graph by adding a negative value into its global score. We also observe that the weights of about 9% global features are almost not updated, which indicates that they are barely found in both gold-standard and predicted graphs. In Table 8, we perform qualitative analysis on concrete examples. As Table 7, we evaluate the proposed framework on ACE05-CN and ERE-ES. The results show that ONEIE works well on Chinese and Spanish data without any special design for the new language. We also observe that adding English training data can improve the performance on Chinese and Spanish. We have analyzed 75 of the remaining errors and in Figure 5 we present the distribution of various error types which need more features and knowledge acquisition to address in the future. In this section, we will discuss some main categories with examples. Need background knowledge . Most of current IE methods ignore external knowledge such as entity attributes and scenario models. For exam-Positive Feature Weight 1 ATRANSPORT event has only one DESTINATION argument 2.61 2 An ATTACK event has only one PLACE argument 2.31 3 ATRANSPORT event has only one ORIGIN argument 2.01 4 An END-POSITION event has only one PERSON argument 1.51 5 APER-SOC relation exists between two PER entities 1.08 6 AGEN-AFF relation exists between ORG and LOC entities 0.96 7 ABENEFICIARY argument is a PER entity 0.93 8 AGEN-AFF relation exists between ORG and GPE entities 0.90 Negative Feature Weight 9 An entity has an ORG-AFF relation with multiple entities -3.21 10 An entity has an PART-WHOLE relation with multiple entities -2.49 11 An event has two PLACE arguments -2.47 12 ATRANSPORT event has multiple DESTINATION arguments -2.25 13 An entity has a GEN-AFF relation with multiple entities -2.02 14 An ATTACK event has multiple PLACE arguments -1.86 15 An entity has a PHYS relation with multiple entities -1.69 16 An event has multiple VICTIM arguments -1.61 Table 6: Salient positive and negative global features. ple, in the following sentence,  And Putin's media aide, Sergei Yastrzhembsky, told Kommersant Russia would not forgive the Iraqi debt , our model mistakenly identifies Kommersan as a person instead of organization. With entity linking, we Sentence & Analysis Baseline +Global Features #1: Russia 's foreign minister expressed outrage at suggestions from a top Washington official last week that Moscow should forgive the eight billion dollars in Soviet-era debt that Baghdad owes it, as a gesture of good will. can correct this error based on the first sentence in its Wikipedia page  Kommersant is a nationally distributed daily newspaper published in Russia mostly devoted to politics and business . Rare words . The second challenge is the famous long-tail problem: many triggers, entity mentions (e.g., caretaker, Gazeta.ru) and contextual phrases in the test data rarely appear in the training data. While most event triggers are verbs or nouns, some adverbs and multi-word expressions can also serve as triggers. Multiple types per trigger . Some trigger words may indicate both the procedure and the result status of an action. For example, named may indicate both NOMINATE and START-POSITION events; killed and eliminate may indicate both ATTACK and DIE events. In these cases the human ground truth usually only annotates the procedure types, whereas our system produces the resultant event types. Need syntactic structure . Our model may benefit from deeper syntactic analysis. For example, in the following sentence  As well as previously holding senior positions at Barclays Bank, BZW and Kleinwort Benson, McCarthy was formerly a top civil servant at the Department of Trade and Industry , our model misses all of the employers Barclays Bank, BZW and Kleinwort Benson for McCarthy probably because they appear in a previous sub-sentence. Uncertain events and metaphors . Our model mistakenly labels some future planned events as specific events because its lacking of tense prediction and metaphor recognition. For example, START-ORG triggered by formation does not happen in the following sentence  The statement did not give any reason for the move, but said Lahoud would begin consultations Wednesday aimed at the formation of a new government . Our model also mistakenly identifies camp as a facility, and a DIE event triggered by dying in the following sentence  Russia hints peace camp' alliance with Germany and France is dying by Dmitry Zaks. . The IE community is lacking of newer data sets with end-to-end annotations. Unfortunately, the annotation quality of the ACE data set is not perfect due to some long-term debates on the annotation guideline; e.g., Should government be tagged as a GPE or an ORG ? Should dead be both an entity and event trigger? Should we consider designator word as part of the entity mention or not? Previous work (Roth and Yih, 2004; Li et al., 2011) encodes inter-dependency among knowledge elements as global constraints in an integer linear programming framework to effectively remove extraction errors. Such integrity verification results can be used to find knowledge elements that violate the constraints and identify possible instances of detector errors or failures. Inspired by these previous efforts, we propose a joint neural framework with global features in which the weights are learned during training. Similar to (Li et al., 2014)'s method, ONEIE also uses global features to capture cross-subtask and cross-instance interdependencies, while our features are language-independent and do not rely on other NLP tools such as dependency parsers. Our methods also differ in local features, optimization methods, and decoding procedures. Some recent efforts develop joint neural models to perform extraction of two IE subtasks, such as entity and relation extraction (Zheng et al., 2017; Katiyar and Cardie, 2017; Bekoulis et al., 2018; Fu et al., 2019; Luan et al., 2019; Sun et al., 2019) and event and temporal relation extraction (Han et al., 2019). Wadden et al. (2019) design a joint model to extract entities, relations and events based on BERT and dynamic span graphs. Our framework extends (Wadden et al., 2019) by incorporating global features based on cross-subtask and cross-instance constraints. Unlike (Wadden et al., 2019) that uses a span-based method to extract mentions, we adopt a CRF-based tagger in our framework because it can extract mentions of any length, not restricted by the maximum span width. We propose a joint end-to-end IE framework that incorporates global features to capture the interdependency interdependency between knowledge elements. Experiments show that our framework achieves better or comparable performance compared to the state of the art and prove the effectiveness of global features. Our framework is also proved to be language-independent and can be applied to other languages, and it can benefit from multi-lingual training. In the future, we plan to incorporate more comprehensive event schemas that are automatically induced from multilingual multimedia data and external knowledge to further improve the quality of IE. We also plan to extend our framework to more IE subtasks such as document-level entity coreference resolution and event coreference resolution. This research is based upon work supported in part by U.S. DARPA KAIROS Program No. FA8750-19-2-1004, U.S. DARPA AIDA Program No. FA8750-18-2-0014, Air Force No. FA8650-17-C-7715, the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via contract No. FA8650-17-C-9116. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
abstain abstain objective abstain abstain objective objective objective abstain abstain other other abstain abstain abstain abstain abstain abstain objective abstain abstain method abstain result objective abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method method method method method method method method method method method abstain abstain abstain method method method method method method method method method abstain method abstain method abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain method method method abstain abstain abstain method method method method method method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method other abstain method method method method other abstain abstain abstain abstain result result abstain result abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other objective method objective other other abstain method objective abstain result method result objective other other other other other other other