It has been shown that named entity recognition (NER) could benefit from incorporating the long-distance structured information captured by dependency trees. We believe this is because both types of features  the contextual information captured by the linear sequences and the structured information captured by the dependency trees may complement each other. However, existing approaches largely focused on stacking the LSTM and graph neural networks such as graph convolutional networks (GCNs) for building improved NER models, where the exact interaction mechanism between the two different types of features is not very clear, and the performance gain does not appear to be significant. In this work, we propose a simple and robust solution to incorporate both types of features with our Synergized-LSTM (Syn-LSTM), which clearly captures how the two types of features interact. We conduct extensive experiments on several standard datasets across four languages. The results demonstrate that the proposed model achieves better performance than previous approaches while requiring fewer parameters. Our further analysis demonstrates that our model can capture longer dependencies compared with strong baselines. 1 1 Introduction Named entity recognition (NER) is one of the most fundamental and important tasks in natural language processing (NLP). While the literature (Peters et al., 2018; Akbik et al., 2018; Devlin et al., 2019) largely focuses on training deep language models to improve the contextualized word representations, previous studies show that  Lu Xu is under the Joint PhD Program between Alibaba and Singapore University of Technology and Design. The work was done when Zhanming Jie was a PhD student in Singapore University of Technology and Design. 1 We make our code publicly available at https:// github.com/xuuuluuu/SynLSTM-for-NER . the structured information such as interactions between non-adjacent words can also be important for NER (Finkel et al., 2005; Jie et al., 2017; Aguilar and Solorio, 2019). However, sequence models such as bidirectional LSTM (Hochreiter and Schmidhuber, 1997) are not able to fully capture the long-range dependencies (Bengio, 2009). For instance, Figure 1 (top) shows one type of structured information in NER. The words  Precision Castparts Corp.  can be easily inferred as ORGANIZATION by its context (i.e., Corp. ). However, the second entity  PCP  could be misclassified as a PRODUCT entity if a model relies more on the context  begin trading with  but ignores the hidden information that  PCP  is the symbol of  Precision Castparts Corp. . Previous research works (Li et al., 2017; Jie and Lu, 2019; Wang et al., 2019) have been using the parse trees (Chomsky, 1956, 1969; Sandra and Taft, 2014) to incorporate such structured information. Figure 1 (Dependency Path) shows that the first entity can be connected to the second entity following the dependency tree with 5 hops. Incorporating the dependency information can be done with graph neural networks (GNNs) such as graph convolutional networks (GCNs) (Kipf and Welling, 2017). However, simply stacking the LSTM and GCN architectures for NER can only provide us with modest improvements; sometimes, it decreases performance (Jie and Lu, 2019). Based on the dependency path in Figure 1, it requires a 5-layer GCN to capture the connections between these two entities. However, deep GCN architectures often face training difficulties, which cause a performance drop (Hamilton et al., 2017b; Kipf and Welling, 2017). Directly stacking GCN and LSTM has diffi-culties in modeling the interaction between dependency trees and contextual information. To address the above limitations, we propose the Synergized-LSTM (Syn-LSTM), a new recurrent neural network architecture that considers an additional graph-encoded representation to update the memory and hidden states, as shown in Figure 2. More specifically, the graph-encoded representation for each word can be obtained with GCNs. Our proposed Syn-LSTM allows the cell to receive the structured information from the graph-encoded representation. With the newly designed gating mechanism, our model is able to make independent assessments on the amounts of information to be retrieved from the word representation and the graph-encoded representation respectively. Such a mechanism allows for better integration of both contextual and structured information. Our contributions can be summarized as:  We propose a simple and robust Syn-LSTM model to better incorporate the structured information conveyed by dependency trees. The output of the Syn-LSTM cell is jointly determined by both contextual and structured information. We adopt the classic conditional random fields (CRF) (Lafferty et al., 2001) on top of the Syn-LSTM for NER. We conduct extensive experiments on several standard datasets across four languages. The proposed model significantly outperforms previous approaches on these datasets. We show that the proposed model can capture long-distance interactions between entities. Our further analysis statistically demonstrates the proposed gating mechanism is able to aggregate the structured information selectively. To incorporate the long-range dependencies, we consider an additional graph-encoded representation g t (Figure 2) as the model input to integrate contextual and structured information. The graph-encoded representation g t can be derived from Graph Neural Networks (GNNs) such as GCN (Kipf and Welling, 2017), which are capable of bringing in structured information through graph structure (Hamilton et al., 2017a). However, structured information sometimes is hard to encode, as we can see from the example in Figure 1. One naive approach is to use a deep GNN to capture such information along multiple dependency arcs between two words, which could mess up information and lead to training difficul-ties. A straightforward solution is to integrate both structured and contextual information via LSTM. As shown in Figure 1 (Hybrid Paths), the structured information can be passed to neighbors or context, which allows a model to use less number of GNN layers and alleviate such issues for long-range dependencies. The input to the LSTM can simply be the concatenation of word representation x t and g t at each position (Jie and Lu, 2019) 2 . However, because such an approach requires both x t and g t to decide the value of the input gate jointly, it could be a potential victim of two sources of uncertainties: 1) the uncertainty of the quality of graph-encoded representation g t , and 2) the uncertainty of the exact interaction mechanism between the two types of features. These may lead to sub-optimal performance, especially if the graph-encoded representation g t is unsatisfactory. Thus, we need to design a new approach to incorporate both types of information from x t and g t with a more explicit interaction mechanism, with which we hope to alleviate the above issues. We propose the Synergized-LSTM (Syn-LSTM) to better integrate the contextual and structured information to address the above limitations. The inputs of the Syn-LSTM cell include previous cell state c t  1 , previous hidden state h t  1 , current cell input x t , and an additional graph-encoded representation g t . The outputs of the Syn-LSTM cell include current cell state c t and current hidden state h t . Within the cell, there are four gates: input gate i t , forget gate f t , output gate o t , and an additional new gate m t to control the flow of information. Note that the forget gate f t and output gate o t are not just looking at h t  1 and x t ; they are also affected by the graph-encoded representation g t . The cell state c t and hidden state h t are computed as follows: f t =  ( W ( f ) x t + U ( f ) h t  1 + Q ( f ) g t + b ( f ) ) (1) o t =  ( W ( o ) x t + U ( o ) h t  1 + Q ( o ) g t + b ( o ) ) (2) i t =  ( W ( i ) x t + U ( i ) h t  1 + b ( i ) ) (3) m t =  ( W ( m ) g t + U ( m ) h t  1 + b ( m ) ) (4)  c t = tanh( W ( u ) x t + U ( u ) h t  1 + b ( u ) ) (5)  s t = tanh( W ( n ) g t + U ( n ) h t  1 + b ( n ) ) (6) c t = f t (cid:12) c t  1 + i t (cid:12)  c t + m t (cid:12)  s t (7) h t = o t (cid:12) tanh( c t ) (8) where  is the sigmoid function, W (  ) , U (  ) , Q (  ) and b (  ) are learnable parameters. The additional new gate m t is used to control the information from the graph-encoded representation directly. Such a design allows the original input gates i t and our new gate m t to make independent assessments on the amounts of information to be retrieved from the word representation x t and the graph-encoded representation g t respectively. On the other hand, we also have a different candidate state  s t to represent the cell state that corresponds to the graph-encoded representation separately. With the proposed Syn-LSTM, the structured information captured by the dependency trees can be passed to each cell, and the additional gate m t is able to control how much structured information can be incorporated. The additional gate enables the model to feed the contextual and structured information into the LSTM cell separately. Such a mechanism allows our model to aggregate the information from linear sequence and dependency trees selectively. Similar to the previous work (Levy et al., 2018), it is also possible to show that the cell state c t implicitly computes the element-wise weighted sum of the previous states by expanding Equation 7: c t = f t (cid:12) c t  1 + i t (cid:12)  c t + m t (cid:12)  s t (9) = t (cid:88) j =0 ( i j (cid:12) t (cid:89) k = j +1 f k ) (cid:12)  c j + t (cid:88) j =0 ( m j (cid:12) t (cid:89) k = j +1 f k ) (cid:12)  s j (10) = t (cid:88) j =0 a tj (cid:12)  c j + t (cid:88) j =0 q tj (cid:12)  s j (11) Note that the two terms, a tj and q tj , are the product of gates. The value of the two terms are in the range from 0 to 1. Since the  c t and  s t represent contextual and structured features, the corresponding weights control the flow of information. The goal of named entity recognition is to predict the label sequence y = { y 1 , y 2 , ..., y n } given the input sequence w = { w 1 , w 2 , ..., w n } , where w t represents the t -th word and n is the number of words. Our model is mainly constructed with three layers: input representation layer, bi-directional Syn-LSTM layer, and CRF layer. The architecture of our Syn-LSTM-CRF is shown in Figure 3. Input Representation Layer Similar to the work by Lample et al. (2016), our input representation also includes the character embeddings, which are the hidden states of character-based BiLSTM. Jie and Lu (2019) highlight that the dependency relation helps to enhance the input representation. Furthermore, previous methods (Wang et al., 2018; Wang and Lu, 2018) use embeddings of part-of-speech (POS) tags as additional input representation. The input representation x t of our model is the concatenation of the word embedding v t , the character representation e t , the dependency relation embedding r t , and the POS embedding p t : x t = [ v t ; e t ; r t ; p t ] (12) where both r t and p t embeddings are randomly initialized and are fine-tuned during training. For experiments with the contextualized representations (e.g., BERT (Devlin et al., 2019)), we further concatenate the contextual word representation to x t . For our task, we employ the graph convolutional network (Kipf and Welling, 2017; Zhang et al., 2018b) to get the graph-encoded representation g t . Given a graph, an adjacency matrix A of size n  n is able to represent the graph structure, where n is the number of nodes; A i,j = 1 indicates that node i and node j are connected. We transform dependency tree into its corresponding adjacency matrix 3 A , and A i,j = 1 denotes that node i and node j have dependency relation. Note that the purpose of graph-encoded representation g t is to incorporate the dependency information from neighbor nodes. The input and output representations of the l -th layer GCN at t -th position are denoted as g l  1 t and g lt respectively. Similar to the work by Zhang et al. (2018b), we use d t = (cid:80) nj =1 A t,j , which is the total number of neighbors of node t , to normalize the representation before going through the nonlinear function. The GCN operation is defined as: g lt = ReLU( n (cid:88) j =1 A t,j W l g l  1 t /d t + b l ) (13) where W l is a linear transformation and b l is a bias. The initial g 0 t is the concatenation of word embedding v t , character embedding e t , and dependency relation embedding r t : g 0 t = [ v t ; e t ; r t ] . Bi-directional Syn-LSTM Layer With the word representation x t and the graph-encoded representation g t , a bi-directional Syn-LSTM is applied to generate contextual representation. The forward and backward Syn-LSTM enable the model to integrate the contextual and structured information from both directions. We concatenate the hidden state  h t from forward Syn-LSTM and hidden state 3 We treat the dependency edge as undirected and add a self-loop for each node: A i,j = A j,i and A i,i = 1 . CRF Layer The CRF (Lafferty et al., 2001) is widely used in NER tasks as it is capable of capturing the structured correlations between adjacent output labels. Given the sentence w and dependency tree  , the probability of the label sequence y is defined as: P ( y | w ,  ) = exp( score ( w , , y )) (cid:80) y (cid:48) exp( score ( w , , y (cid:48) )) (14) The score function is defined as: score ( w , , y ) = n (cid:88) t =0 T y t ,y t +1 + n (cid:88) t =1 E y t (15) where T y t ,y t +1 denotes the transition score from label y t to y t +1 , E y t denotes the score of label y t at the t -th position and the scores are computed using the hidden state h t . We learn the model parameters by minimizing the negative log-likelihood and employ the Viterbi algorithm to obtain the best label sequence during evaluation. Datasets The proposed model is evaluated on four benchmark datasets: SemEval 2010 Task 1 (Recasens et al., 2010) Catalan and Spanish datasets, and OntoNotes 5.0 (Weischedel et al., 2013) English and Chinese datasets. We choose these four datasets as they have explicit dependency annotations which allow us to evaluate the effectiveness of our approach when dependency trees of different qualities are used. For SemEval 2010 Task 1 datasets, there are 4 entity types: PER , LOC and ORG and MISC . For OntoNotes 5.0 datasets, there are 18 entity types in total. Following the work by Jie and Lu (2019), we transform the parse trees into the Stanford dependency trees (De Marneffe and Manning, 2008) by using Stanford CoreNLP (Manning et al., 2014). Detailed statistics of each dataset can be found in Table 1. Intuitively, longer sentences would require the model to capture more long-distance interactions in the sentences. We present the number of entities in terms of different sentence lengths to show that these datasets have a modest amount of entities in long sentences. Experimental Setup For Catalan, Spanish, and Chinese, we use the FastText (Grave et al., 2018) 300 dimensional embeddings to initialize the word embeddings. For OntoNotes 5.0 English, we adopt the publicly available GloVE (Pennington et al., 2014) 100 dimensional embeddings to initialize the word embeddings. For experiments with the contextualized representation, we adopt the pre-trained language model BERT (Devlin et al., 2019) for the four datasets. Specifically, we use bert-as-service (Xiao, 2018) to generate the contextualized word representation without fine-tuning. Following Luo et al. (2020), we use the cased version of BERT large model for the experiments on the OntoNotes 5.0 English data. We use the cased version of BERT base model for the experiments on the other three datasets. For the character embedding, we randomly initialize the character embeddings and set the dimension as 30, and set the hidden size of character-level BiLSTM as 50. The hidden size of GCN and Syn-LSTM is set as 200, the number of GCN layer is 2. We adopt stochastic gradient descent (SGD) to optimize our model with batch size 100, L2 regularization 10  8 , initial learning rate lr 0.2 and the learning rate is decayed 4 with respect to the number of epoch. We select the best model based on the performance on the dev set 5 and apply it to the test set. We use the bootstrapping t-test to compare the results. Baselines We compare our model with several baselines with or without dependency tree information. The first one is BERT-CRF, where we apply a CRF layer on top of BERT (Devlin et al., 2019). Secondly, we compare with the BERT implementation by HuggingFace (Wolf et al., 2019). For models with dependency trees, we take the models BiLSTM-GCN-CRF and dependency-4 We set the decay as 0.1 and the learning rate for each epoch equals to lr/ (1 + decay  ( epoch  1)) . guided LSTM-CRF (DGLSTM-CRF) proposed by Jie and Lu (2019), and our implemented GCN-BiLSTM-CRF. The BiLSTM-GCN-CRF model simply stacks the GCN on top of the BiLSTM to incorporate the dependency trees. The GCN-BiLSTM-CRF model takes the concatenation of the graph-encoded representation from GCN and word embedding as input into BiLSTM. The DGLSTM-CRF takes the concatenation of the head word representation and word embedding as input into BiLSTM. Note that the original implementation of DGLSTM-CRF uses ELMo (Peters et al., 2018), but we also implement it with BERT. Besides, we compare our model with previous works that have results on these datasets. SemEval 2010 Task 1 Table 2 shows comparisons of our model with baseline models on the SemEval 2010 Task 1 Catalan and Spanish datasets. Our Syn-LSTM-CRF model outperforms all existing models with F 1 82.76 and 85.09 ( p < 10  5 ) compared to DGLSTM-CRF on Catalan and Spanish datasets when FastText word embeddings are used. Our model outperforms the BiLSTM-CRF model by 13.25 and 11.22 F 1 points, and outperforms BiLSTM-GCN-CRF (Jie and Lu, 2019) model by 4.64 and 3.16 on Catalan and Spanish. The large performance gap between BiLSTM-GCN-CRF and our model indicates that Syn-LSTM-CRF shows better compatibility with GCN, and this confirms that simply stacking GCN on top of the BiLSTM does not perform well. Our method outperforms GCN-BiLSTM-CRF model by 5.33 and 3.24 F 1 points on Catalan and Spanish. This shows that our proposed model demonstrates a better integration of contextual information and structured information. Furthermore, our proposed method brings 1.12 and 1.62 F 1 points improvement on Catalan and Spanish datasets compare to the DGLSTM-CRF (Jie and Lu, 2019). The DGLSTM-CRF employs 2-layer dependency guided BiLSTM to capture grandchild dependencies, which leads to longer training time and more model parameters. However, our Syn-LSTM-CRF is able to get better performance with fewer model parameters and shorter training time because of the fewer LSTM layers. Such results demonstrate that our proposed Syn-LSTM-CRF manages to capture structured information effectively. Furthermore, with the contextualized word representation, the Syn-LSTM-CRF +BERT achieves much higher performance improvement than any other method. Our model outperforms the strong baseline model DGLSTM-CRF +ELMO by 4.83 and 2.54 in terms of F 1 ( p < 10  5 ) on Catalan and Spanish, respectively. OntoNotes 5.0 English To understand the generalizability of our model, we evaluate the proposed Syn-LSTM-CRF model on large scale OntoNotes 5.0 datasets. Table 3 shows comparisons of our model with baseline models on English. Our Syn-LSTM-CRF model outperforms all existing methods with 89.04 in terms of F 1 score ( p < 0 . 01 ) compared to DGLSTM-CRF, when GloVE word embeddings are used. Our model outperforms the BiLSTM-CRF model by 1.97 in F 1 , BiLSTM-GCN-CRF (Jie and Lu, 2019) model by 0.86. Note that our implemented GCN-BiLSTM-CRF outperforms the previous DGLSTM-CRF (Jie and Lu, 2019) by 0.14 in F 1 . Our Syn-LSTM-CRF further brings the improvement to 0.52. Moreover, with the contextualized word representation BERT, our method achieves an F 1 score of 90.85 ( p < 10  5 ) compared to DGLSTM-CRF +ELMO . Our method outperforms the previous model (Luo et al., 2020), which relies on document-level information, by 0.55 in F 1 . Furthermore, the performance improvement on recall is more prominent as compared to precision. This shows that the proposed Syn-LSTM-CRF is able to extract more entities. experimental results on the OntoNotes 5.0 Chinese test set in Table 4. Our model still consistently outper-Models P. R. F 1 Chiu and Nichols (2016a) 86.04 86.53 86.28 Li et al. (2017) 88.00 86.50 87.21 Strubell et al. (2017) -86.84 Ghaddar and Langlais (2018) -87.95 BiLSTM-CRF  87.21 86.93 87.07 BiLSTM-GCN-CRF  88.30 88.06 88.18 GCN-BiLSTM-CRF  88.56 88.76 88.66 DGLSTM-CRF (2019) 88.53 88.50 88.52 Luo et al. (2020) -87.98 Syn-LSTM-CRF (Ours) 88.96 89.13 89.04 + Contextualized Word Representation Akbik et al. (2018) -89.30 BERT-CRF  88.42 88.33 88.37 Wolf et al. (2019)  88.39 90.29 89.33 BiLSTM-CRF +ELMO  89.14 88.59 88.87 BiLSTM-CRF +BERT  89.32 90.02 89.67 BiLSTM-GCN-CRF +ELMO  89.40 89.71 89.55 GCN-BiLSTM-CRF +BERT  89.34 91.26 90.29 DGLSTM-CRF (2019) +ELMO 89.59 90.17 89.88 DGLSTM-CRF +BERT  89.63 89.87 89.75 Luo et al. (2020) +BERT -90.30 Syn-LSTM-CRF +BERT (Ours) 90.14 91.58 90.85 Table 3: Experimental results [%] on OntoNotes 5.0 English test set. forms the baseline models, specifically by 2.04 in F 1 compared to BiLSTM-CRF, by 2.39 compared to BiLSTM-GCN-CRF, by 1.86 compared to GCN-BILSTM-CRF and by 1.11 ( p < 10  5 ) compared to DGLSTM-CRF when FastText is used. Note that the baseline BiLSTM-GCN-CRF model is 0.35 points worse than BiLSTM-CRF. Such results further confirm the effectiveness of our proposed Syn-LSTM-CRF for incorporating structured information. We find a similar behavior when the contextualized word representation BERT is used. With the contextualized word representation, we achieve a higher F 1 score of 80.20. Robustness Analysis To study the robustness of our model and check whether our model can regulate the flow of information from the graph-encoded representation, we analyze the influence of the quality of dependency trees. We train and evaluate an additional dependency parser (Dozat and Manning, 2017). Specifically, we train the Models P. R. F 1 Pradhan et al. (2013) 78.20 66.45 71.85 Lattice LSTM (2018) 76.34 77.01 76.67 BiLSTM-CRF  78.45 74.59 76.47 BiLSTM-GCN-CRF  76.35 75.89 76.12 GCN-BiLSTM-CRF  78.30 75.07 76.65 DGLSTM-CRF (2019) 77.40 77.41 77.40 Syn-LSTM-CRF (Ours) 77.95 79.07 78.51 + Contextualized Word Representation BERT-CRF  79.83 79.68 79.75 Wolf et al. (2019)  77.35 81.74 79.49 BiLSTM-CRF +ELMO  79.20 79.21 79.20 BiLSTM-CRF +BERT  78.45 81.24 79.82 BiLSTM-GCN-CRF +ELMO  78.71 79.29 79.00 GCN-BiLSTM-CRF +BERT  79.03 80.98 80.00 DGLSTM-CRF (2019) +ELMO 78.86 81.00 79.92 DGLSTM-CRF +BERT  77.79 81.65 79.67 Syn-LSTM-CRF +BERT (Ours) 78.66 81.80 80.20 Table 4: Experimental results [%] on OntoNotes 5.0 Chinese test set. dependency parser 6 on the given training datasets and select the best model based on the dev sets. Then we apply the best model to the test sets to obtain dependency trees. We also train and evaluate our model with random dependency trees. Table 8 presents the comparisons between Syn-LSTM-CRF +BERT and DGLSTM-CRF +ELMO with given, predicted and random dependency trees. We observe that both models encounter a performance drop when we use the predicted parse trees and random trees. Our performance differences with the given parse trees are relatively smaller than the corresponding differences in DGLSTM-CRF +ELMO . Such an observation demonstrates the robustness of our proposed model against structured information from the trees of different quality. It is worthwhile to note that, with the predicted dependencies, our proposed Syn-LSTM-CRF +BERT is still able to outperform the strong baseline DGLSTM-CRF +ELMO even with the given parse trees on Catalan, English, and Chinese datasets. To further study the robustness, we conduct an analysis to investigate if the gate m t (Figure 2) has the ability to regulate the flow of information from the graph-encoded representation. Intuitively, the gate m t should tend to have a small value when 6 The performance of the dependency parser can be found in the Appendix. We statistically plot the number of words with respect to different gate value ranges ( m t ). Figure 4 shows the comparison between the models of using random trees and given trees on Catalan and Spanish 7 . We observe that the gate m t is more likely to open (the value is higher) when we use the given parse trees compared with random parse trees. Such behavior demonstrates that our proposed model can selectively aggregate the information from the graph-encoded representation. Effect of Sentence Length We compare the performance of our Syn-LSTM-CRF +BERT with BiLSTM-CRF +BERT and DGLSTM-CRF +ELMO models with respect to sentence length, and the results are shown in Figure 5. We observe that the Syn-LSTM-CRF +BERT model consistently outperforms the two baseline models on the four languages 8 . In particular, although the performance tends to drop as the sentence length increases, our proposed model shows relatively better performance when the sentence length is  60 . This confirms that the proposed Syn-LSTM-CRF +BERT is able to effectively incorporate structured information. Note that our 2-layer GCN is computed based on the 7 We found a similar behavior for OntoNotes 5.0 English and Chinese datasets, and the detailed result can be found in the Appendix. 8 See the Appendix for the results on OntoNotes 5.0 English and Chinese datasets. With the graph-encoded representation and the proposed Syn-LSTM-CRF +BERT , the individual word representation is enhanced by both contextual and structured information. Therefore, for the sentences with length of  14 , we can still observe obvious improvements. The significant performance improvements on the four datasets show the capability of our Syn-LSTM-CRF to capture the structured information despite the sentence length. Effect of Entity Length We conduct another evaluation on BiLSTM-CRF +BERT , DGLSTM-CRF +ELMO , and Syn-LSTM-CRF +BERT models with respect to entity length  { 1 , 2 , 3 , 4 , 5 ,  6 } on the four languages. Table 6 shows the performance comparison of two models with respect to entity length. With the structured information, both DGLSTM-CRF +ELMO and Syn-LSTM-CRF +BERT achieve better performance compared to BiLSTM-CRF +BERT . When the length of entity is  3, Syn-LSTM-CRF +BERT achieves better results compared to DGLSTM-CRF +ELMO . This confirms that our proposed method can effectively incorporate the structured information. Our model consistently outperforms BiLSTM-CRF +BERT , and the performance tends to have more improvements when entities are getting longer except on the Chinese dataset. We note there are some special characteristics of the Chinese language. As mentioned by Jie and Lu (2019), the percentage of entities that are able to perfectly form a sub-tree is only 92 . 9% for OntoNotes Chinese, as compared to 98 . 5% , 100% , 100% for OntoNotes English, SemEval Catalan and Spanish. Furthermore, the ratio of long entities is much higher for Catalan and Spanish compared Dataset Model Entity Length 1 2 3 4 5  6 Catalan BiLSTM-CRF +BERT 82.4 84.4 77.8 53.3 31.8 36.2 DGLSTM-CRF +ELMO 85.4 85.1 84.1 78.9 60.9 59.3 Syn-LSTM-CRF +BERT 90.5 91.1 87.2 77.8 63.8 60.6 Spanish BiLSTM-CRF +BERT 85.1 84.2 81.5 33.7 43.1 27.2 DGLSTM-CRF +ELMO 89.3 87.4 90.8 74.1 67.7 64.4 Syn-LSTM-CRF +BERT 92.7 90.9 91.1 73.0 75.4 58.5 English BiLSTM-CRF +BERT 92.9 88.3 83.1 85.5 80.5 77.9 DGLSTM-CRF +ELMO 91.8 90.1 85.4 87.0 80.8 78.7 Syn-LSTM-CRF +BERT 92.9 90.8 87.7 87.4 80.6 79.8 Chinese BiLSTM-CRF +BERT 82.5 74.6 71.4 65.0 69.8 52.5 DGLSTM-CRF +ELMO 82.2 75.5 71.8 64.1 58.5 41.1 Syn-LSTM-CRF +BERT 82.5 75.6 73.1 66.4 66.1 42.5 Table 6: F 1 score [%] based on entity length on Catalan, Spanish, English and Chinese datasets. to English and Chinese. The experimental results on Catalan and Spanish datasets show significant improvements for long entities. Such results show that the structured information conveyed by the dependency trees can be more crucial when entity length becomes longer. Number of GCN Layers To fully explore the impact of the number of GCN layers, we conduct another experiment on Syn-LSTM-CRF +BERT model with the number of GCN layers  { 1 , 2 , 3 } , and Figure 6 shows the performance on the dev set of the four languages. The last bar, indicated as AVG, is obtained by averaging the dev results on the four datasets. We observe that the overall performance is better when the number of GCN layers equals 2. Note that similar behavior can also be found in the work by Kipf and Welling (2017) for document classification and node classification. Therefore, we evaluate our proposed Syn-LSTM-CRF model with 2-layer GCN. Ablation Study To understand the contribution of each component, we conduct an ablation study on the OntoNotes 5.0 English dataset, and Table 7 presents the detailed results of our model with contextualized representation. We find that the performance drops by 0.24 F 1 score when we only use 1-layer GCN. Without GCN at all, the score drops by 1.13 F 1 . The original dependency contributes 0.27 F 1 score. Removing the dependency relation embedding also decreases the performance by 0.27 F 1 . When we remove the POS tags embedding, the result drops by 0.39 F 1 . LSTM LSTM has demonstrated its great effectiveness in many NLP tasks and becomes a standard module for many state-of-the-art models (Wen et al., 2015; Ma and Hovy, 2016; Dozat and Manning, 2017). However, the sequential nature of the LSTM makes it challenging to capture long-range dependencies. Zhang et al. (2018a) propose the S-LSTM model to include a sentence state to allow both local and global information exchange simultaneously. Mogrifier LSTM (Melis et al., 2020) mutually gates the current input and the previous output to enhance the interaction between the input and the context. These two works do not consider structured information for the LSTM design. Since natural language is usually structured, Shen et al. (2018) propose ON-LSTM to add a hierarchical bias to allow the neurons to be updated by following certain order. While the ON-LSTM is learning the latent constituency parse trees, we focus on incorporating the explicit structured information conveyed by the dependency parse trees. NER Early work (Sasano and Kurohashi, 2008) uses syntactic dependency features to improve the SVM performance on Japanese NER task. Liu et al. (2010) propose to construct skip-edges to link similar words or words having typed dependencies to capture long-range dependencies. The later works (Collobert et al., 2010; Lample et al., 2016; Chiu and Nichols, 2016b) focus on using neural networks to extract features and achieved the state-of-the-art performance. Jie et al. (2017) find that some relations between the dependency edges and the entities can be used to reduce the search space of their model, which significantly reduces the time complexity. Yu et al. (2020) employ pre-trained language model to encode document-level information to explore all spans with the graph-based dependency graph based ideas. The pre-trained language models (e.g., BERT (Devlin et al., 2019), ELMO (Peters et al., 2018)) further improve neural-based approaches with a good contextualized representation. However, previous works did not focus on investigating how to effectively integrate structured and contextual information well. In this paper, we propose a simple and robust Syn-LSTM model to better integrate the structured information leveraged from the long-range dependencies. Specifically, we introduce an additional graph-encoded representation to each recurrent unit. Such a graph-encoded representation can be obtained via GNNs. Through the newly designed gating mechanism, the hidden states are enhanced by contextual information captured by the linear sequence and structured information captured by the dependency trees. We present the Syn-LSTM-CRF for NER and adopt the GCN on dependency trees to obtain the graph-encoded representations. Our extensive experiments and analysis on the datasets with four languages demonstrate that the proposed Syn-LSTM is able to effectively incorporate both contextual and structured information. The robustness analysis demonstrates that our model is capable of selectively aggregating the information from the graph-encoded representation. We would like to thank the anonymous reviewers for their helpful comments. This research is partially supported by Ministry of Education, Singapore, under its Academic Research Fund (AcRF) Tier 2 Programme (MOE AcRF Tier 2 Award No: MOE2017-T2-1-156). Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not reflect the views of the Ministry of Education, Singapore.
abstain method abstain objective method abstain objective abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain objective abstain objective method abstain objective objective method method objective objective objective method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain method abstain abstain abstain other abstain abstain abstain method abstain abstain abstain abstain method abstain method abstain method abstain abstain method abstain abstain abstain abstain method abstain other abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result result result result result result abstain abstain abstain abstain abstain abstain result abstain result result result result abstain result result abstain abstain abstain result abstain abstain abstain result result method method other abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other other other method other other other other other other other objective method abstain abstain result objective objective other other other