Event extraction for the biomedical domain is more challenging than that in the general news domain since it requires broader acquisition of domain-specific knowledge and deeper understanding of complex contexts. To better encode contextual information and external background knowledge, we propose a novel knowledge base (KB)-driven tree-structured long short-term memory networks (Tree-LSTM) framework, incorporating two new types of features: (1) dependency structures to capture wide contexts; (2) entity properties (types and category descriptions) from external ontologies via entity linking. We evaluate our approach on the BioNLP shared task with Genia dataset and achieve a new state-of-the-art result. In addition, both quantitative and qualitative studies demonstrate the advancement of the Tree-LSTM and the external knowledge representation for biomedical event extraction. Biomedical information extraction is widely used to assist the biology community on knowledge acquisition and ontology construction. Biomedical events generally refer to a change of status, particularly on proteins or genes. The goal of event extraction is to identify triggers and their arguments from biomedical text, and then assign an event type to each trigger and a role to each argument. For example, in the sentence shown in Figure 1, it includes a gene expression and a positive regulation event mention, both triggered by the word transduced . Tax is the Theme argument of the gene expression event. An event could also serve as an argument of another event, leading to a nested structure. For instance, the gene expression event triggered by transduced is also a Theme argument of the positive regulation event as shown in Figure 1. Earlier studies on biomedical event extraction rely on kernel classification methods like the support vector machines (SVMs) (Bjorne and Salakoski, 2011; Venugopal et al., 2014) using hand-crafted features, which require high engineering effort and domain-specific knowledge. Recent distributional representation based approaches (Rao et al., 2017; Bjorne and Salakoski, 2018) explore deep neural networks which only require distributed semantic features. However, different from event extraction in the general news domain, biomedical event extraction requires broad acquisition of domain-specific knowledge and deep understanding of complex contexts. For example, in Genia event extracton of BioNLP shared task 2011 (Kim et al., 2011), about 80% of entity mentions are abbreviations of genes, proteins and diseases while more than 36% of event triggers and arguments are separated with more than 10 words. In order to efficiently capture indicative information from broad contexts, we first adopt tree structure based long short-term memory (Tree-LSTM) networks. Compared to the linear chain structured LSTM, the Tree-LSTM takes tree-structured network topology into consideration. As shown in the top frame of Figure 1, Tree-LSTM takes the dependency tree structure of each sentence as input and gradually incorporates the information from the whole subtree into each node. Dependency tree structure can connect semantically related concepts, and thus shorten the distance between a trigger and its arguments significantly. For instance, in the following sentence  ... , which binds to the enhancer A located in the promoter of the mouse MHC class I gene H-2Kb , ... , when determining the trigger type of binds , we need to carefully select its contextual words, such as H-2Kb , which indicates the object of binds . However, binds and H-2Kb are sepa-Characterization of peripheral blood T-lymphocytes transduced with HTLV-I Tax mutants with different trans activating phenotypes . rated with 16 words which is difficult for a chain-structured LSTM to capture their long distance dependency, while within dependency tree structure, their distance is significantly shortened to 7. Moreover, to better capture domain-specific knowledge, we further propose to leverage the external knowledge bases (KBs) to acquire properties of all the biomedical entities. The KB properties are extremely beneficial for our model to learn patterns more explicitly. Take the entity Tax in Figure 1 as an example, it's a protein often involved in the biological process of positive regulation of transcription referred to Gene Ontology (Ashburner et al., 2000). This function description provides crucial clues to determine the type of transduced as positive regulation . Therefore, to capture such knowledge from external KBs, for each entity, we first learn a KB concept embedding from its properties, and then automatically incorporate the KB representation into its Tree-LSTM hidden state with a gate function. Our contributions are twofold: First, to the best of our knowledge, it's the first time to adopt Tree-LSTM for biomedical event extraction to effectively capture the wide contexts. Second, we further incorporate external knowledge from domain-specific KBs into the Tree-LSTM, which yields state-of-the-art performance on Genia event extraction shared task. In this section, we present our KB-driven Tree-LSTM approach for biomedical event extraction. We first introduce the Tree-LSTM framework, and then describe the construction of KB concept embedding for each entity. Finally we incorporate the KB concept embedding into a Tree-LSTM and apply it for event trigger and argument extraction. The Tree-LSTM (Tai et al., 2015) is a variation of LSTM (Hochreiter and Schmidhuber, 1997) to a tree-structured network topology. It shows improvement in representing sentence semantic meaning compared to sequential LSTM such as Bidirectional LSTM (BiLSTM) (Graves et al., 2013). The main difference between sequential LSTM and Tree-LSTM is, at each time step, the former calculates its hidden state from the input at the current time step and the hidden state from previous step, while Tree-LSTM computes its hidden state from the input token and the hidden states of all its children nodes from the tree structure. A Tree-LSTM reduces to sequential LSTM when each node in the tree only has one child. Figure 2 (A) shows a Tree-LSTM unit. In order to obtain the hidden state h j of an input token x j , the unit calculates all of its children hidden states ( h j  1 , h j  2 ) through depth-first traversal. For the biomedical event extraction, we mainly explore the Gene Ontology as our external KB since it provides detailed descriptions for each gene and gene product attributes across all species. It consists of two types of information: (1) the gene ontology (GO) defines all the gene functions, relations between these gene functions, and aspects used to describe the gene functions, including molecular function, cellular component and biological process. (2) the gene product annotations (GO Anno) provide all entity related attributes, such as the full entity name, entity type, as well as the gene functions it is related to. For example, in Figure 1, given the entity tax , from the gene product annotations, we can get its full entity name as tax protein which is a type of proteins and it's related to a function about biological process . From the gene ontology, we can further determine the specific function that tax is related to positive regulation of transcription in terms of biological process aspect. In order to leverage the external KB information, we first apply QuickGO API (Binns et al., 2009) to link each entity mention to the Gene Ontology and retrieve all the KB annotations. For each entity, we carefully select two types of properties which are beneficial for event extraction task: the entity type (e.g., protein for tax ) and the gene ontology function it is related to (e.g., positive regulation of transcription for tax ). The entity type can facilitate the explicit pattern learning for argument role labeling, for example, the gene expression event pattern (Theme: Protein, Trigger: transduced) is more popular than (Theme: Tax, Trigger: transduced) in Figure 1. The gene ontology function can provide implicit clues to determine the trigger type as aforementioned in Section 1. As shown in Figure 1, we assign a word embedding which pretrained on PubMed and PMC texts (Moen and Ananiadou, 2013) to represent each entity type. For each gene ontology function which is usually a long phrase, we use a state-of-the-art sentence embedding approach (Conneau et al., 2017) to automatically learn a vector representation. We then concatenate these two types of KB property representations as the final KB concept embedding. After obtaining the KB concept embeddings, we further incorporate them into the Tree-LSTM to leverage the domain-specific knowledge. Given a sentence, for example the sentence shown in Figure 3, we first perform the dependency parsing with the Stanford dependency parser (Chen and Manning) and obtain a dependency tree structure. For each node j in the tree structure, C ( j ) is the set of children nodes of node j and  k is the KB concept embedding of node k . We set  k to 0 if node k is not a biomedical entity. (cid:101)  j denotes the sum of the KB concept embeddings of j 's children nodes and (cid:101) h j is the sum of the hidden states of j 's children nodes: (cid:101) h j = (cid:88) k  C ( j ) h k (cid:101)  j = (cid:88) k  C ( j )  k where h k is the hidden state of node k . Then we incorporate the KB concept embeddings into the input, forget, and output gates of the Tree-LSTM: i j =  ( W i [ x j , (cid:101) h j , (cid:101)  j ] + b i ) f jk =  ( W f [ x j , h k , (cid:101)  k ] + b f ) o j =  ( W o [ x j , (cid:101) h j , (cid:101)  j ] + b o ) where i j and o j are the input gate and the output gate for node j respectively. f jk is the forget gate for node j in terms of its child node k . W i , W f , and W o are learnable parameters, b i , b f and b o are bias terms. Thus, for each node j , the input gate gathers all KB information from its children nodes, and the output gate balances the meaningful information from its local contexts and the KB concept embeddings of its children nodes. Besides adding the KB concept embeddings into the three gates to select useful KB formation implicitly, similar to Ma et al. (2018), we also introduce a knowledge specific output gate g j to explicitly incorporate knowledge information into each node's hidden state. While different from Ma et al. (2018) which only considers the knowledge concept embedding of each node itself, we use the sum of the KB concept embeddings of the whole subtree instead: g j =  ( W g [ x j , (cid:101) h j , (cid:101)  j ] + b g ) where W g is a weight matrix to be learned, b g is the bias term. As demonstrated in Figure 2 (B), we eventually combine the implicit way of incorporating KB information into the input, output and forget gates and an explicit way of directly incorporating the KB information into a node's hidden state: (cid:101) c j = tanh( W c [ x j , (cid:101) h j ] + b c ) c j = (cid:88) k  C ( j ) f jk (cid:12) c k + i j (cid:12) (cid:101) c j h j = o j (cid:12) tanh( c j ) + g j (cid:12) tanh( W  (cid:101)  j ) where c j is the memory cell, W c and W  are weight matrices to be learned. After getting the hidden state h j of each node j , we use a softmax classifier to predict a label for each node, and optimize the parameters by minimizing a negative log-likelihood loss. After detecting all candidate triggers, we further extract arguments for each trigger. The Genia event extraction shared task provides the annotations of all entity mentions. Thus, for each trigger, we use all the entity mentions that occur in the same sentence as its candidate arguments, and then assign an argument role or None . Different from trigger extraction, we use the shortest dependency path (SDP) within the dependency tree structure instead of the surface contexts to better capture the dependency between the trigger and each argument. Taking the sentence in Figure 3 as an example, given a trigger transcription and a candidate argument OBF-1 , we first perform dependency parsing and extract the shortest dependency path between transcription and OBF-1 with the Dijkstra's algorithm (Johnson, 1973) and obtain the shortest dependency path transcription  of  genes  OBF-1 . We use the same KB-driven Tree-LSTM architecture as introduced in Section 2.3 to encode each node into a new hidden state representation. We use the hidden state of the root node h 0 as the overall vector representation of the whole dependency path. Finally, we feed the concatenation of h 0 with the hidden state of the trigger and argument as input to another softmax to predict the argument role. We also optimize the model by minimizing a negative log-likelihood loss. The Genia Event Extraction task is the main task in the BioNLP Shared Task series (Kim et al., 2009, 2011; Nedellec et al., 2013). The Genia task defines 9 fine-grained event types as shown in Table 1. Note that a Binding event may take more than one protein as its Theme arguments. A Regulation event may take one protein or event as its Theme argument and also optionally take one protein or event as its Cause argument. A Regulation event taking an event as its argument will lead to a nested structure. 37.2% nested events are observed in Genia 2011 corpus (Bjorne and Salakoski, 2011). There are 6.0% inter-sentence events while our model only focuses on sentence-level event extraction. We apply our KB-driven Tree-LSTM model on Genia 2011 data set. The entities in Genia data set are manually annotated and given as part of the input. We evaluate our results on the test set using the official online tool provided by the Genia task organizers. 1 Following previous studies (Bj  orne and Salakoski, 2011; Venugopal et al., 2014; Rao et al., 2017; Bjorne and Salakoski, 2018), we report scores obtained by the approximate span (al-lowing trigger spans to differ from gold spans by single words). As we only focus on matching core arguments, we use recursive matching criterion for evaluation which not requires matching of additional arguments for events referred from other events (Kim et al., 2011). We use the word embedding pretrained on PubMed and PMC texts (Moen and Ananiadou, 1 http://bionlp-st.dbcls.jp/GE/2011/eval-test/ 2013) for word and type embeddings. The hyper-parameters are tuned on the development set and listed in Table 2. Word representations are updated during training with an initial learning rate of 0.1. Table 3 shows the final event extraction results of applying our KB-driven Tree-LSTM model on Genia 2011 dataset with the comparison of only using Tree-LSTM and a standard BiLSTM model. Tree-LSTM outperforms the BiLSTM baseline which indicates the power of Tree-LSTM in dealing with long-distance dependency structure in biomedical literature. By incorporating external KB information, our approach achieves about 2.12% F-score gain comparing to Tree-LSTM, which demonstrates the effectiveness of the KB properties for biomedical event extraction. We will show detailed analysis in Section 3.4. Table 4 presents the previous event extraction results from the BioNLP shared task using the same corpus. Our approach outperforms all previous methods. Among them, the systems TEES (Bjorne and Salakoski, 2011), EventMine-CR (Miwa et al., 2012) and Stacked Generalization (Majumder et al., 2016) are based on SVMs with well designed features. FAUST (Riedel and McCallum, 2011) and BioMLN (Venugopal et al., 2014) use jointed inference models. Bjorne and Salakoski (2018) adopts a convolutional neural networks (CNNs) with abundant features derived from TEES system. In our work, instead of using high-dimensional features with manual effort as in these previous models, our approach only requires pretrained distributed word representations as input features. We notice that our approach achieves high scores on Simple event types but get relatively low scores on Binding event and Regulation event types. We analyze the results and find that Bind-System Event Type Rec Prec F1 KB-drivenTree-LSTM Gene expression 74.35 87.24 80.28 Transcription 69.54 82.31 75.39 Protein catabolism 46.67 87.50 60.87 Phosphorylation 81.62 87.28 84.36 Localization 59.69 80.28 68.47 Simple total 72.62 85.95 78.73 Binding 37.68 53.16 44.10 Regulation 36.62 53.61 43.52 Positive regulation 41.37 57.90 48.26 Negative regulation 46.06 52.39 49.02 Regulation total 41.73 55.73 47.72 Event total 52.14 67.01 58.65 Tree-LSTM Simple total 71.22 83.41 76.83 Binding 34.83 48.72 40.62 Regulation total 39.78 53.54 45.64 Event total 50.28 64.56 56.53 BiLSTM Simple total 68.09 78.75 73.03 Binding 38.49 43.05 40.65 Regulation total 37.64 53.81 44.30 Event total 48.44 62.18 54.46 Table 3: Precision (Prec), recall (Rec) and F-score (F1) results achieved by the KB-driven Tree-LSTM model on the test set of BioNLP Genia 2011, evaluated on approximate span and recursive criteria. ing event extraction is more challenging since it usually has multiple arguments. For example, Figure 4 shows two sentences which are chosen from the output of the development data set. There are two Binding event mentions in the first sentence: E1 (Trigger: interacting, Type: Binding, Theme: RUNX1, Theme2: p3000) and E2 (Trigger: binding, Type: Binding, Theme: CREB). Our model mistakenly extracts CREB as a Theme of E1 since CREB is highly related to protein p300 in the dependency tree structure. Regulation events are considered as the most challenging event type because they usually have an optional Cause argument and are involved in nested structures, which are not handled well by most of current event extraction approaches. In addition, intuitively, most trigger words are verbs or nouns. We rank all the trigger words in the train-... the EBNA-1 gene in infected thymocytes was transcribed from the Fp promoter, rather than from the Cp / Wp promoter ... [protein] E1:[transcription] E2:[positive_regulation] E3:[positive_regulation] E1:Theme E3:Theme RUNX1 alone, or together with its interacting partners p300 and CREB binding protein, ... [protein] E1:[binding] [protein] [protein] E2:[binding] E1:Theme1 E1:Theme2 E2:Theme E2:Theme Figure 4: Case study on binding event and regulation event types. ing data set according to their frequency, and find that most of spurious errors for Regulation event trigger extraction occur when the trigger words are prepositions or conjunctions. For instance, in Figure 4, the second sentence contains two positive Regulation events triggered by a preposition from and a conjunction rather than . Such function words are rarely annotated as triggers and our KB-aware Tree-LSTM cannot well collect meaningful contexts from their subtrees. As shown in Table 3, we achieve about 3.5% and 2.1% F1 score gain on Binding and Regulation event types by leveraging external KB information into the Tree-LSTM. In order to show the effect of KB concept embeddings, we visualize the probabilities of word transcription to be predicted for each event type. As Figure 5 shows, by adding KB concept embeddings, the function description positive regulation of transcription, DNA-templated provided by the biomedical entity OBF-1 significantly enhances the probability of transcription being predicted to a Transcription event type. Similarly, Figure 6 visualizes the probabilities of the E1 event mention (Trigger: trans-transduced [gene_expression]/[positive_regulation] with mutants Tax [protein] E1 : (Type:gene expression, Theme: Tax , Trigger: transduced) E2 : (Type:positive regulation, Theme: E1 , Trigger: transduced) E2 : (Theme: None ) E2 : (Theme: E1 ) Theme Theme Figure 6: Visualization of the effect of KB concept embedding on argument role labeling for a Positive Regulation event triggered by transduced and a Gene Expression event E1 (Theme: Tax, Trigger: transduced). duced, Type: gene expression, Theme: Tax) to be predicted as an argument of E2 event mention (Trigger: transduced, Type: positive regulation, Theme: E1). We can see that, without using KB information, the Tree-LSTM mistakenly predict the argument role of E1 as None . In contrast, by incorporating KB concept embeddings, especially the information from the function description positive regulation of transcription, DNA-templated for Tax , our approach successfully promotes the probability of E1 being predicted as the Theme of E2. As a crucial task in information extraction, event extraction has gained a lot of interest. In general news domain, previous work on event extraction can be divided into two main categories. The first is feature-based methods which mainly focus on feature design, leveraging local features (Grishman et al., 2005; Ahn, 2006) and global features (Ji and Grishman, 2008; Liao and Gr-ishman, 2011; Huang and Riloff, 2012) to improve the performance. Some studies proposed joint models to overcome the error propagation problem (Poon and Vanderwende, 2010; Riedel et al., 2009; Li et al., 2013; Venugopal et al., 2014; Li et al., 2014). The second category includes distributional representation based methods which have been applied into event extraction extensively. Most of these approaches are based on the standard Convolutional Neural Networks (CNNs) (Chen et al., 2015; Nguyen and Grishman, 2015, 2016), Recurrent Neural Networks (RNNs) (Nguyen et al., 2016), generative adversarial networks (Hong et al., 2018), zero-shot learning (Huang et al., 2017) and advanced attention mechanisms (Liu et al., 2018b; Chen et al., 2018). Our work is also related to the studies which leverage the external knowledge base for information extraction. Liu et al. (2017) takes advantage of external resources, such as FrameNet, to label events while Chen et al. (2017) adopts distance supervision to augment the training data. Liu et al. (2018a) develops an attention-based model for event extraction. What's more, shortest dependency path is broadly explored for information extraction, especially for relation classification (Xu et al., 2015; Miwa and Bansal, 2016) and shows promising benefits. Biomedical event extraction task part of the BioNLP Shared Task series (Kim et al., 2009, 2011; Nedellec et al., 2013). Previous studies mainly explore local and global features with SVM model (Miwa et al., 2010, 2012; Bjorne and Salakoski, 2013; Majumder et al., 2016). Riedel and McCallum (2011) develop a joint model with dual decomposition. Cohen et al. (2009), Kilicoglu and Bergler (2011) and Bui et al. (2013) develop rule-based methods and achieve high precision. Venugopal et al. (2014) leverage Markov logic networks for joint inference. Rao et al. (2017) uses the Abstract Meaning Representations (AMR) to extract events based on the assump-tion that an event structure can be derived from an AMR subgraph. Recently, some representation-based models (Jagannatha and Yu, 2016; Rao et al., 2017; Bjorne and Salakoski, 2018) have been proposed while most of them adopt the widely used CNNs and RNNs with features derived from the biomedical text. Lim et al. (2018) implements a binary Tree-LSTM architecture for biomedical relation extraction. Compared with these methods, our approach only requires pretrained distributed word representations as input features and incorporates meaningful KB information into a Tree-LSTM. In this paper, we show the effectiveness of using a KB-driven tree-structured LSTM for event extraction in biomedical domain. The Tree-LSTM can efficiently capture semantically related concepts for each node within the tree structure. By leveraging the external KB concept properties including the entity type and the function description, our approach is able to perform deep understanding of domain-specific expressions and connections. Without using manually designed high-dimensional features, our approach significantly outperforms all previous methods. In the future, we plan to explore a broader range of properties from KB to facilitate biomedical information extraction tasks. This work was supported by the U.S. NSF No. 1741634, Air Force No. FA8650-17-C-7715 and ARL NS-CTA No. W911NF-09-2-0053. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation here on.
abstain objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain method abstain abstain objective abstain abstain abstain objective objective abstain method objective method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain other other abstain abstain abstain method method other method method abstain method method abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain method abstain method method method other abstain abstain result abstain abstain result abstain result abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain other abstain abstain abstain result result abstain other abstain result abstain other other other other other other abstain other other other other other other other other other other other method result abstain method result objective other other other other other