Existing works on information extraction (IE) have mainly solved the four main tasks separately (entity mention recognition, relation extraction, event trigger detection, and argument extraction), thus failing to benefit from inter-dependencies between tasks. This paper presents a novel deep learning model to simultaneously solve the four tasks of IE in a single model (called FourIE). Compared to few prior work on jointly performing four IE tasks, FourIE features two novel contributions to capture inter-dependencies between tasks. First, at the representation level, we introduce an interaction graph between instances of the four tasks that is used to enrich the prediction representation for one instance with those from related instances of other tasks. Second, at the label level, we propose a dependency graph for the information types in the four IE tasks that captures the connections between the types expressed in an input sentence. A new regularization mechanism is introduced to enforce the consistency between the golden and predicted type dependency graphs to improve representation learning. We show that the proposed model achieves the state-of-the-art performance for joint IE on both monolingual and multilingual learning settings with three different languages. Information Extraction (IE) is an important and challenging task in Natural Language Processing (NLP) that aims to extract structured information from unstructured texts. Following the terminology for IE in the popular ACE 2005 program (Walker et al., 2006), we focus on four major IE tasks in this work: entity mention extraction (EME), relation extraction (RE), event trigger detection (ETD), and event argument extraction (EAE). Given an input sentence, a vast majority of prior work has solved the four tasks in IE independently at both instance and task levels (called independent Person Vehicle Transport Facility A man driving what appeared to be a taxicab came to the checkpoint , Person waved soldiers over , appeared to be having mechanical problems of some kind . PHYS ART 00 Artifact Destination PHYS Figure 1: A sentence example with the annotations for the four IE tasks. Blue words corresponds to entity mentions while red words are event triggers. Also, or-ange edges represent relations while green edges indicate argument roles. prediction models). First, at the instance level, each IE task often requires predictions/classifications for multiple instances in a single input sentence. For instance, in RE, one often needs to predict relations for every pair of entity mentions (called relation instances) in the sentence while multiple word spans in the sentence can be viewed as multiple instances where event type predictions have to be made in ETD (trigger instances). As such, most prior work on IE has performed predictions for instances in a sentence separately by treating each instance as one example in the dataset (Zhou et al., 2005; Nguyen and Grishman, 2015a; Santos and Guimaraes, 2015; Chen et al., 2015; Nguyen and Grishman, 2015b; Lai et al., 2020). Second, at the task level, prior work on IE tends to perform the four tasks in a pipelined approach where outputs from one task are used as inputs for other tasks (e.g., EAE is followed by EME and ETD) (Li et al., 2013; Chen et al., 2015; Veyseh et al., 2020c). Despite its popularity, the main issue of the independent prediction models is that they suffer from the error propagation between tasks and the failure to exploit the cross-task and cross-instance interdependencies within an input sentence to improve the performance for IE tasks. For instance, such systems are unable to benefit from the dependency that the Victim of a Die event has a high chance to Span Detection Mention Trigger A man driving what appeared to be a taxicab came to the checkpoint , waved soldiers over ,  BERT Encoder + Two Conditional Random Fields for event trigger and entity mention sequence labeling Instance Interaction A man driving what appeared to be a taxicab came to the checkpoint , waved soldiers over ,  came man taxicab checkpoint soldiers Event trigger Entity mention Event argument Relation (Candidates) Type Prediction & Regularization Instance representations: Soft predicted labels: Gold labels: Gumbel-Softmax One-hot samples: Figure 2: Overall architecture of our proposed model. also be the Victim of an Attack event in the same sentence (i.e., type or label dependencies). To address these issues, some prior work has explored joint inference models where multiple tasks of IE are performed simultaneously for all task instances in a sentence, using both feature-based models (Roth and Yih, 2004; Li et al., 2013; Miwa and Sasaki, 2014; Yang and Mitchell, 2016) and recent deep learning models (Miwa and Bansal, 2016; Zhang et al., 2019). However, such prior work has mostly considered joint models for a subset of the four IE tasks (e.g., EME+RE or ETD+EAE), thus still suffering from the error propagation issue (with the missing tasks) and failing to fully exploit potential inter-dependencies between the four tasks. To this end, this work aims to design a single model to simultaneously solve the four IE tasks for each input sentence (joint four-task IE) to address the aforementioned issues of prior joint IE work. Few recent work has considered joint four-task IE, using deep learning to produce state-of-the-art (SOTA) performance for the tasks (Wadden et al., 2019; Lin et al., 2020). However, there are still two problems that hinder further improvement of such models. First, at the instance level, an important component of deep learning models for joint IE involves the representation vectors of the instances that are used to perform the corresponding prediction tasks for IE in an input sentence (called predictive instance representations). For joint four-task IE, we argue that there are inter-dependencies between predictive representation vectors of related instances for the four tasks that should be modeled to improve the performance for IE. For instance, the entity type information encoded in the predictive representation vector for an entity mention can constrain the argument role that the representation vector for a related EAE instance (e.g., involving the same entity mention and some event trigger in the same sentence) should capture and vice versa. As such, prior work for joint four-task IE has only computed predictive representation vectors for instances of the tasks independently using shared hidden vectors from some deep learning layer (Wad-den et al., 2019; Lin et al., 2020). Although this shared mechanism helps capture the interaction of predictive representation vectors to some extent, it fails to explicitly present the connections between related instances of different tasks and encode them into the representation learning process. Consequently, to overcome this issue, we propose a novel deep learning model for joint four-task IE (called FourIE ) that creates a graph structure to explicitly capture the interactions between related instances of the four IE tasks in a sentence. This graph will then be consumed by a graph convolutional network (GCN) (Kipf and Welling, 2017; Nguyen and Grishman, 2018) to enrich the representation vector for an instance with those from the related (neigh-boring) instances for IE. Second, at the task level, existing joint four-task models for IE have only exploited the cross-task type dependencies in the decoding step to constrain predictions for the input sentence (by manually converting the type dependency graphs of the input sentence into global feature vectors for scoring the predictions in the beam search-based decoding) (Lin et al., 2020). The knowledge from cross-task type dependencies thus cannot contribute to the training process of the IE models. This is unfortunate as we expect that deeper integration of this knowledge into the training process could provide useful information to enhance representation learning for IE tasks. To this end, we propose to use the knowledge from cross-task type dependencies to obtain an additional training signal for each sentence to directly supervise our joint four-task IE model. In particular, our motivation is that the types expressed in a sentence for the four IE tasks can be organized into a dependency graph between the types (global type dependencies for the sentence). As such, in order for a joint model to perform well, the type dependency graph generated by its predictions for a sentence should be similar to the dependency graph obtained from the golden types (i.e., a global type constraint on the predictions in the training step). A novel regularization term is thus introduced into the training loss of our joint model to encode this constraint, employing another GCN to learn representation vectors for the predicted and golden dependency graphs to facilitate the graph similarity promotion. To our knowledge, this is the first work that employs global type dependencies to regularize joint models for IE. Finally, our extensive experiments demonstrate the effectiveness of the proposed model on benchmark datasets in three different languages (e.g., English, Chinese, and Spanish), leading to state-of-the-art performance on different settings. Problem Statement : The joint four-task IE problem in this work takes a sentence as the input and aims to jointly solve four tasks EAE, ETD, RE, and EAE using an unified model. As such, the goal of EME is to detect and classify entity mentions (names, nominals, pronouns) according to a set of predefined (semantic) entity types (e.g., Person ). Similarly, ETD seeks to identify and classify event triggers (verbs or normalization) that clearly evoke an event in some predefined set of event types (e.g., Attack ). Note that event triggers can involve multiple words. For RE, its concern is to predict the semantic relationship between two entity mentions in the sentence. Here, the set of relations of interest is also predefined and includes a special type of None to indicate no-relation . Finally, in EAE, given an event trigger, the systems need to predict the roles (also in a predefined set with a special type None ) that each entity mention plays in the corresponding event. Entity mentions are thus also called event argument candidates in this work. Figure 1 presents a sentence example where the expected outputs for each IE task are illustrated. Graph Convolutional Networks (GCN) : As GCNs are used extensively in our model, we present their computation process in this section to facilitate the discussion. Given a graph G = ( V , E ) where V = { v 1 , . . . , v u } is the node set (with u nodes) and E is the edge set. In GCN, the edges in G are often captured via the adjacency matrix A  R u  u . Also, each node v i  V is associated with an initial hidden vector v 0 i . As such, a GCN model involves multiple layers of abstraction in which the hidden vector v li for the node v i  V at the l -th layer is computed by ( l  1 ): v li = ReLU ( (cid:80) uj =1 A ij W l v l  1 j + b l (cid:80) uj =1 A ij ) where W l and b l are trainable weight and bias at the l -th layer. Assuming NGCN layers, the hidden vectors for the nodes in V at the last layer v N 1 , . . . , v Nu would capture richer and more abstract information for the nodes, serving as the outputs of the GCN model. This process is denoted by: v N 1 , . . . , v Nu = GCN ( A ; v 01 , . . . , v 0 u ; N ) . Given an input sentence w = [ w 1 , w 2 , . . . , w n ] (with n words), our model for joint four-task IE on w involves three major components: (i) Span Detection, (ii) Instance Interaction, and (iii) Type Dependency-based Regularization. This component aims to identify spans of entity mentions and event triggers in w that would be used to form the nodes in the interaction graph between different instances of our four IE tasks for w . As such, we formulate the span detection problems as sequence labeling tasks where each word w i in w is associated with two BIO tags to capture the span information for entity mentions and event triggers in w . Note that we do not predict entity types and event types at this step, leading to only three possible values (i.e., B, I, and O) for the tags of the words. In particular, following (Lin et al., 2020), we first feed w into the pre-trained BERT encoder (Devlin et al., 2019) to obtain a sequence of vectors X = [ x 1 , x 2 , . . . , x n ] to represent w . Here, each vector x i serves as the representation vector for the word w i  w that is obtained by averaging the hidden vectors of the word-pieces of w i returned by BERT. Afterward, X is fed into two conditional random field (CRF) layers to determine the best BIO tag sequences for event mentions and event triggers for w , following (Chiu and Nichols, 2016). As such, the Viterbi algorithm is used to decode the input sentence while the negative log-likelihood losses are employed as the training objectives for the span detection component of the model. For convenience, let L entspan and L trgspan be the negative log-likelihoods of the gold tag sequences for entity mentions and event triggers (respectively) for w . These terms will be included in the overall loss function of the model later. Based on the tag sequences for w from the previous component, we can obtain two separate span sets for the entity mentions and event triggers in w (the golden spans are used in the training phase to avoid noise). For the next computation, we first compute a representation vector for each span ( i, j ) ( 1  i  j  n ) in these two sets by averaging the BERT-based representation vectors for the words in this span (i.e., x i , . . . , x j ). For convenience, let R ent = { e 1 , e 2 , . . . , e n ent } ( n ent = | R ent | ) and R trg = { t 1 , t 2 , . . . , t n trg } ( n trg = | R trg | ) be the sets of span representation vectors for the entity mentions and event triggers in w 1 . The goal of this component is to leverage such span representation vectors to form instance representations and enrich them with instance interactions to perform necessary predictions in IE. Instance Representation . Prediction instances in our model amount to the specific objects that we need to predict a type for one of the four IE 1 We will also refer to entity mentions and event triggers interchangeably with their span representations e i and t i in this work. tasks. As such, the prediction instances for EME and ETD, called entity and trigger instances, correspond directly to the entity mentions and event triggers in R ent and R trg respectively (as we need to predict the entity types for e i  R ent and the event types for t i  R trg in this step). Thus, we also use R ent and R trg as the sets of initial representation vectors for the entity/event instances for EME and ETD in the following. Next, for RE, the prediction instances (called relation instances) involve pairs of entity mentions in R ent . To obtain the initial representation vector for a relation instance, we concatenate the representation vectors of the two corresponding entity mentions, leading to the set of representation vectors rel ij for relation instances: R rel = { rel ij = [ e i , e j ] | e i , e j  R ent , i < j } ( | R rel | = n ent ( n ent  1) / 2 ). Finally, for EAE, we form the prediction instances (called argument instances) by pairing each event trigger in R trg with each entity mention in R ent (for the argument role predictions of the entity mentions with respect to the event triggers/mentions). By concatenating the representation vectors of the paired entity mentions and event triggers, we generate the initial representation vectors arg ij for the corresponding argument instances: R arg = { arg ij = [ t i , e j ] | t i  R trg , e j  R ent } ( | R arg | = n trg n ent ) 2 . We also use the prediction instances and their representation vectors interchangeably in this work. Instance Interaction . The initial representation vectors for the instances so far do not explicitly consider beneficial interactions between related instances. To address this issue, we explicitly create an interaction graph between the prediction instances for the four IE tasks to connect related instances to each other. This graph will be consumed by a GCN model to enrich instance representations with interaction information afterward. In particular, the node set N inst in our instance interaction graph G inst = { N inst , E inst } involves all prediction instances for the four IE tasks, i.e., N inst = R ent  R trg  R rel  R arg . The edge set E inst then captures instance interactions by connecting the instance nodes in N inst that involve the same entity mentions or event triggers (i.e., two instances are related if they concern the same entity mention or event trigger). As such, the edges in E inst are created as follows: 2 In our implementation, R rel and R arg are transformed into vectors of the same size with those in R ent and R trg (us-ing one-layer feed forward networks) for future computation. (i) An entity instance node e i is connected to all relation instance nodes of the forms rel ij = [ e i , e j ] and rel ki = [ e k , e i ] (sharing entity mention e i ). (ii) An entity instance node e j is connected to all argument instance nodes of the form arg ij = [ t i , e j ] (sharing entity mention e j ). (iii) A trigger node t i is connected to all argument instance nodes of the form arg ij = [ t i , e j ] (i.e., sharing event trigger t i ). GCN . To enrich the representation vector for an instance in N inst with the information from the related (neighboring) nodes, we feed G inst into a GCN model (called GCN inst ). For convenience, we rename the initial representation vectors of all the instance nodes in N inst by: N inst = { r 1 , . . . , r n i } ( n i = | N inst | ). Also, let A inst  { 0 , 1 } n i  n i be the adjacency matrix of the interaction graph G inst where A instij = 1 if the instance nodes r i and r j are connected in G inst or i = j (for self-connections). The interaction-enriched representation vectors for the instances in N inst are then computed by the GCN inst model: r inst 1 , . . . , r instn i = GCN inst ( A inst ; r 1 , . . . , r n i ; N i ) where N i is the number of layers for the GCN inst model. Type Embedding and Prediction . Finally, the enriched instance representation vectors r inst 1 , . . . , r instn i will be used to perform the predictions for the four IE tasks. In particular, let t k  { ent, trg, rel, arg } be the corresponding task index and y k be the ground-truth type (of the task t k ) for the prediction instance r k in N inst . Also, let T = T ent  T trg  T rel  T arg be the union of the possible entity types (in T ent for EME), event types (in T trg for ETD), relations (in T rel for RE), and argument roles (in T arg for EAE) in our problem ( y k  T t k ). Note that T rel and T arg contain the special types None . To prepare for the type predictions and the type dependency modeling in the next steps, we associate each type in T with an embedding vector (of the same size as e i and t i ) that is initialized randomly and updated during our training process. For convenience, let T = [ t 1 , . . . ,  t n t ] where  t i is used interchangeably for both a type and its embedding vector in T ( n t is the total number of types). As such, to perform the prediction for an instance r k in N inst , we compute the dot products between r instk and each type embedding vectors in T t k  T to estimate the possibilities that r k has a type in T t k . Afterward, these scores are normalized by the softmax function to obtain the probability distribution  y k over the possible types in T t k for r k :  y k = softmax ( r instk  t T |  t  T t k  T ) . In the decoding phase, the predicted type  y k for r k is obtained via the argmax function (greedy decoding):  y k = argmax  y k . The negative log-likelihood over all the prediction instances is used to train the model: L type =  (cid:80) n i k =1 log  y k [ y k ] . In this section, we aim to obtain the type dependencies across tasks and use them to supervise the model in the training process (to improve the representation vectors for IE). As presented in the introduction, our motivation is to generate global dependency graphs between types of different IE tasks for each input sentence whose representations are leveraged to regularize the model during training. In particular, starting with the golden types y = y 1 , y 2 , . . . , y n i and the predicted types  y =  y 1 ,  y 2 , . . . ,  y n i for the instance nodes in N inst , we build two dependency graphs G gold and G pred to capture the global type dependencies for the tasks (called the golden and predicted dependency graphs respectively). Afterward, to supervise the training process, we seek to constrain the model so the predicted dependency graph G pred is similar to the golden graph G gold (i.e., using the dependency graphs as the bridges to inject the global type dependency knowledge in G gold into the model). Dependency Graph Construction . Both G gold and G pred involve the types of all the four IE tasks in T as the nodes. To encode the type dependencies, the connections/edges in G gold are computed based on the golden types y = y 1 , y 2 , . . . , y n i for the instance nodes in N inst as follows: (i) For each relation instance node r k = [ e i , e j ]  N inst that has the golden type y k (cid:54) = None , the relation type node y k is connected to the nodes of the golden entity types for the corresponding entity mentions e i and e j (called en-tity_relation type edges ). (ii) For each argument instance node r k = [ t i , e j ] that has the role type y k (cid:54) = None , the role type node y k is connected to both the node for the golden event type of t i (called event_argument type edges ) and the node for the golden entity type of e j (called entity_argument type edges ). The same procedure can be applied to build the predicted dependency graph G pred based on the predicted types  y =  y 1 ,  y 2 , . . . ,  y n i . Also, for convenience, let A gold and A pred (of size n t  n t ) be the binary adjacency matrices of G gold and G pred (including the self-loops) respectively. Regularization . In the next step, we obtain the representation vectors for the dependency graphs G gold and G pred by feeding them into a GCN model (called GCN type ). This GCN model has N t layers and uses the initial type embeddings T = [ t 1 , . . . ,  t n t ] as the inputs. In particular, the outputs of GCN type for the two graphs involve  t gold 1 , . . . ,  t goldn t = GCN type ( A gold ; t 1 , . . . ,  t n t ; N t ) and  t pred 1 , . . . ,  t predn t = GCN type ( A pred ; t 1 , . . . ,  t n t ; N t ) that encode the underlying information for the type dependencies presented in G gold and G pred . Finally, to promote the similarity of the type dependencies in G gold and G pred , we introduce the mean square difference between their GCN type -induced representation vectors into the overall loss function for minimization: L dep = (cid:80) n t i =1 ||  t goldi   t predi || 22 . Our final training loss is thus: L = L entspan + L trgspan + L type + L dep (  is a trade-off parameter). Approximating A pred . We distinguish two types of parameters in our model so far, i.e., the parameters used to compute instance representations, e.g., those in BERT and G inst (called  inst ), and the parameters for type dependency regularization, i.e., those for the type embeddings  t 1 , . . . ,  t n t and G type (called  dep ). As such, the current implementation only enables the training signal from L dep to back-propagate to the parameters  dep and disallows L dep to influence the instance representation-related parameters  inst . To enrich the instance representation vectors with type dependency information, we expect L dep to be deeper integrated into the model by also contributing to  inst . To achieve this goal, we note that the block of back-propagation between L dep and  inst is due to their only connection in the model via the adjacency matrix A pred , whose values are either one or zero. As such, the values in A pred are not directly dependent on any parameter in  inst , making it impossible for the back-propagation to flow. To this end, we propose to approximate A pred with a new matrix  A pred that directly involves  inst in its values. In particular, let I inst be the index set of the non-zero cells in A pred : I inst = { ( i, j ) | A predij = 1 } . As the elements in I inst are determined by the indexes i 1 , . . . , i n i in T of the predicted types  y 1 ,  y 2 , . . . ,  y n i (respectively), we also seek to compute the values for the approximated matrix  A pred based on such indexes. Accordingly, we first define the matrix B = { b ij } i,j =1 ..n t where the element b ij at the i -th row and j -th column is set to b ij = i  n t + j . Here,  > 0 is a large constant. For each element ( i, j )  I inst , all the elements in the matrix ( B  in t  j ) 2 are strictly positive, except for the element at ( i, j ) , which is zero. Thus, with a large value for  , the matrix exp(   ( B  in t  j ) 2 ) has the value of one at cell ( i, j ) and nearly zero at other cells. Consequently, the values of  A pred at the positions in I inst are close to one while those at other positions are close to zero, thus approximating our expected matrix A pred and still directly depending on the indexes i 1 , . . . , i n t . Addressing the Discreteness of Indexes . Even with the approximation  A pred , the back-propagation still cannot flow from L dep to  inst due to the block of the discrete and non-differentiable index variables i 1 , . . . , i n t . To address this issue, we propose to apply the Gumbel-Softmax distribution (Jang et al., 2017) that enables the optimization of models with discrete random variables, by providing a method to approximate one-hot vectors sampled from a categorical distribution with continuous ones. In particular, we first rewrite each index i k by: i k = h k c Tk , where c k is a vector whose each dimension contains the index of a type in T t k in the joint type set T , and h k is the binary one-hot vector whose dimensions correspond to the types in T t k . h k is only turned on at the position corresponding to the predicted type  y k  T t k (indexed at i k in T ). In our current implementation,  y k (thus the index i k and the one-hot vector h k ) is obtained via the argmax function:  y k = argmax  y k , which causes the discreteness. As such, the Gumbel-Softmax distribution method helps to relax argmax by approximating h k with a sample  h k =  h k, 1 , . . . ,  h k, |T tk | from the Gumbel-Softmax distribution:  h k,j = exp (( log (  k,j ) + g j ) / ) (cid:80) |T tk | j (cid:48) =1 exp (( log (  k,j (cid:48) ) + g j (cid:48) ) / ) (2) where  k,j =  y k,j = softmax j ( r instk  t T |  t  T t k T ) , g 1 , . . . , g |T tk | are the i.i.d samples drawn from Gumbel(0,1) distribution (Gumbel, 1948): g j =  log (  log ( u j )) ( u j  Uniform (0 , 1) ), and  is the temperature parameter. As   0 , the sample  h k would become close to our expected one-hot vector h k . Finally, we replace h k with the approximation  h k in the computation for i k : i k =  h k c Tk that directly depends on r instk and is applied in  A pred . This allows the gradients to flow from L dep to the parameters  inst and completes the description of our model. Datasets . Following the prior work on joint four-task IE (Wadden et al., 2019; Lin et al., 2020), we evaluate our joint IE model (FourIE) on the ACE 2005 (Walker et al., 2006) and ERE datasets that provide annotation for entity mentions, event triggers, relations, and argument roles. In particular, we use three different versions of the ACE 2005 dataset that focus on three major joint inference settings for IE: (i) ACE05-R for joint inference of EME and RE, (ii) ACE05-E for joint inference of EME, ETD and EAE, and (iii) ACE05-E+ for joint inference of the four tasks EME, ETD, RE, and EAE. ACE05-E+ is our main evaluation setting as it fits to our model design with the four IE tasks of interest. For ERE, following (Lin et al., 2020), we combine the data from three datasets for English (i.e., LDC2015E29, LDC2015E68, and LDC2015E78) that are created under the Deep Exploration and Filtering of Test (DEFT) program (called ERE-EN ). Similar to ACE05-E+, ERE-EN is also used to evaluate the joint models on four IE tasks. four-IE datasets on Chinese and Spanish. Following (Lin et al., 2020), we use the ACE 2005 dataset for the evaluation on Chinese (called ACE05-CN ) and the ERE dataset (LDC2015E107) for Spanish (called ERE-ES ). To ensure a fair comparison, we adopt the same data pre-processing and splits (train/dev/test) in prior work (Lin et al., 2020) for all the datasets. As such, ACE05-R, ACE05-E, ACE05-E+, and AC05-CN involve 7 entity types, 6 relation types, 33 event types, and 22 argument roles while ERE-ES and ERE-EN include 7 entity types, 5 relation types, 38 event types, and 20 argument roles. The statistics for the datasets are shown in Table 1. Hyper-parameters and Evaluation Criteria . We fine-tune the hyper-parameters for our model using the development data. The suggested values are shown in the appendix. To achieve a fair comparison with (Lin et al., 2020), we employ the bert-large-cased model for the English datasets and bert-multilingual-cased model for the Chinese and Spanish datasets. Finally, we follow the same evaluation script and correctness criteria for entity mentions, event triggers, relations, and argument as in prior work (Lin et al., 2020). The reported results are the average performance of 5 model runs using different random seeds. Performance Comparison . We compare the proposed model FourIE with two prior models for joint four-task IE: (i) DyGIE++ (Wadden et al., 2019): a BERT-based model with span graph propagation, and (ii) OneIE (Lin et al., 2020): the current state-of-the-art (SOTA) model for joint four-task IE based on BERT and type dependency constraint at the decoding step. Table 2 presents the performance (F1 scores) of the models on the test data of the English datasets. Note that in the tables, the prefixes Ent, Trg, Rel, and Arg represent the extraction tasks for entity mentions, event triggers, relations, and arguments respectively while the suffixes -I and -C correspond to the identification performance (only concerning the offset correctness) and identification+classification performance (evaluating both offsets and types). As can be seen from the table, FourIE is consistently better than the two baseline models (Dy-GIE++ and OneIE) across different datasets and tasks. The performance improvement is significant for almost all the cases and clearly demonstrates the effectiveness of the proposed model. FourIE and OneIE on the Chinese and Spanish datasets (i.e., ACE05-CN and ERE-ES). In addition to the monolingual setting (i.e., trained and evaluated on the same languages), following (Lin et al., 2020), we also evaluate the models on the multilingual training settings where ACE05-CN and ERE-ES are combined with their corresponding English datasets ACE05-E+ and EAE-EN (respectively) to train the models (for the four IE tasks), and the performance is then evaluated on the test sets of the corresponding languages (i.e., ACE05-CN and ERE-ES). It is clear from the table that FourIE also significantly outperforms OneIE across nearly all the different setting combinations for languages, datasets and tasks. This further illustrates the portability of FourIE to different languages. Effects of GCN inst and GCN type . This section evaluates the contributions of the two important components in our proposed model FourIE, i.e., the instance interaction graph with GCN inst and the type dependency graph with GCN type . In particular, we examine the following ablated/varied models for FourIE: (i)  FourIE -GCN inst : this model excludes the instance interaction graph and the GCN model GCN inst from FourIE so the initial instance representations r k are directly used to predict the types for the instances (replacing the enriched vectors r instk ), (ii)  FourIE -GCN type : this model eliminates the type dependency graph and the GCN model GCN type (thus the loss term L dep as well) from FourIE, (iii)  FourIE -GCN inst GCN type : this model removes both the instance interaction and type dependency graphs from FourIE, (iv)  FourIE -GCN type +TDDecode: this model also excludes GCN type ; however, it additionally applies the global type dependencies features to score the joint predictions for the beam search in the decoding step (the implementation for this beam search is inherited from (Lin et al., 2020) for a fair comparison), and (v)  FourIE  A pred : instead of employing the approximation matrix  A pred in FourIE, this model directly uses the adjacency matrix A pred in the L dep regularizer ( L dep thus does not influence the instance representation-related parameters  inst ). Table 4 shows the performance of the models on the development dataset of ACE05-E+ for four IE tasks. The most important observation from the table is that both GCN inst and GCN type are necessary for FourIE to achieve the highest performance for the four IE tasks. Importantly, replacing GCN type in FourIE with the global type dependency features for decoding (i.e.,  FourIE GCN type +TDDecode) as in (Lin et al., 2020) or eliminating the approximation  A pred for L dep produces inferior performance, especially for relation and argument extraction. This clearly demonstrates the benefits for deeply integrating knowledge from type dependencies to influence representation learning parameters with L dep for joint four-task IE. Contributions of Type Dependency Edges . Our type dependency graphs G gold and G pred involves three categories of edges, i.e., entity_relation, en-tity_argument, and event_argument type edges. Table 5 presents the performance of FourIE (on the development data of ACE05-E+) when each of these edge categories is excluded from our type dependency graph construction. dependency edges on the ACE05-E+ dev data. The table clearly shows the importance of different categories of type dependency edges for FourIE as the elimination of any category would generally hurt the performance of the model. In addition, we see that the contribution level of the type dependency edges intuitively varies according to the tasks of consideration. For instance, entity_relation type edges are helpful mainly for entity mention, relation and argument extraction. Finally, an error analysis is conducted in the appendix to provide insights about the benefits of the type dependency graphs G gold and G pred for FourIE (i.e., by comparing the outputs of FourIE and  FourIE -GCN type ). The early joint methods for IE have employed feature engineering to capture the dependencies between IE tasks, including Integer Linear Programming for Global Constraints (Roth and Yih, 2004; Li et al., 2011), Markov Logic Networks (Riedel et al., 2009; Venugopal et al., 2014), Structured Perceptron (Li et al., 2013, 2014; Miwa and Sasaki, 2014; Judea and Strube, 2016), and Graphical Models (Yu and Lam, 2010; Yang and Mitchell, 2016). Recently, the application of deep learning has facilitated the joint modeling for IE via shared parameter mechanisms across tasks. These joint models have focused on different subsets of the IE tasks, including EME and RE (Zheng et al., 2017; Katiyar and Cardie, 2017; Bekoulis et al., 2018; Fu et al., 2019; Luan et al., 2019; Sun et al., 2019; Veyseh et al., 2020b,a), event and temporal RE (Han et al., 2019), and ETD and EAE (Nguyen et al., 2016; Zhang et al., 2019; Nguyen and Nguyen, 2019). However, none of these work has explored joint inference for four IE tasks EME, ETD, RE, and EAE as we do. The two most related works to ours include (Wadden et al., 2019) that leverages the BERT-based information propagation via dynamic span graphs, and (Lin et al., 2020) that exploits BERT and global type dependency features to constrain the decoding step. Our model is different from these works in that we introduce a novel interaction graph for instance representations for four IE tasks and a global type dependency graph to directly inject the knowledge into the training process. We present a novel deep learning framework to jointly solve four IE tasks (EME, ETD, RE, and EAE). Our model attempts to capture the interdependencies between instances of the four tasks and their types based on instance interaction and type dependency graphs. GCN models are employed to induce representation vectors to perform type predictions for task instances and regularize the training process. The experiments demonstrate the effectiveness of the proposed model, leading to SOTA performance over multiple datasets on English, Chinese, and Spanish. In the future, we plan to extend the model to include more IE tasks (e.g., coreference resolution). This research has been supported by the Army Research Office (ARO) grant W911NF-21-1-0112. This research is also based upon work supported by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA Contract No. 2019-19051600006 under the Better Extraction from Text Towards Enhanced Retrieval (BETTER) Program. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ARO, ODNI, IARPA, the Department of Defense, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein. This document does not contain technology or technical data controlled under either the U.S. International Traffic in Arms Regulations or the U.S. Export Administration Regulations.
abstain objective abstain objective objective abstain objective abstain method other abstain abstain abstain abstain abstain other abstain other abstain objective abstain abstain abstain result abstain abstain abstain objective abstain abstain abstain method objective objective abstain objective objective objective objective other other other other other other abstain other method other other other other other other abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain result objective abstain abstain abstain method abstain method method abstain result method method method abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain other abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other objective other objective objective abstain abstain abstain objective other other other other other