External syntactic and semantic information has been largely ignored by existing neural coreference resolution models. In this paper, we present a heterogeneous graph-based model to incorporate syntactic and semantic structures of sentences. The proposed graph contains a syntactic sub-graph where tokens are connected based on a dependency tree, and a semantic sub-graph that contains arguments and predicates as nodes and semantic role labels as edges. By applying a graph attention network, we can obtain syntactically and semantically augmented word representation, which can be integrated using an attentive integration layer and gating mechanism. Experiments on the OntoNotes 5.0 benchmark show the effectiveness of our proposed model. 1 1 Introduction Coreference resolution is a core task in NLP, which aims to identify all mentions that refer to the same entity. Coreference encodes rich semantic information which has been successfully applied to improve many downstream NLP tasks (Luan et al., 2019; Wadden et al., 2019; Dasigi et al., 2019; Stojanovski and Fraser, 2018). Impressive progress has been made in recent years since the introduction of the first end-to-end neural coreference resolution model (Lee et al., 2017) by utilising contextualized embeddings from large pretrained language models (Joshi et al., 2019, 2020; Kantor and Globerson, 2019; Wu et al., 2020) such as ELMo (Peters et al., 2018) and BERT (De-vlin et al., 2019). Rich language knowledge encoded in these pretrained models has largely alleviated the need for syntactic and semantic features. However, such information has been shown to benefit BERT based models on other tasks (Nie et al., 2020a; Wang et al., 2020; Pouran Ben Veyseh et al., 1 https://github.com/Fantabulous-J/coref-HGAT 2020). Therefore, we believe such information could also benefit the coreference resolution task. In this paper, we propose a neural coreference resolution model based on Joshi et al. (2019), which we extend by incorporating external syntactic and semantic information. For syntactic information, we use dependency trees to capture the long-term dependency exists among mentions. Kong and Jian (2019) has successfully incorporated structural information into neural models, but their model still requires the design of complex hand-engineered features. In contrast, our model is more flexible, using a graph neural network to encode syntax in the form of dependency trees. For semantic information, we adopt semantic role labelling (SRL) structures. SRL labels capture who did what to whom and it is effective in providing document-level event description information, which allows us to better identify the relationship between event mentions. Previous statistical coreference systems have successfully integrated such information (Ponzetto and Strube, 2006; Kong et al., 2009), but their effectiveness has not been examined in neural models. Moreover, inspired by recent progress made in document-level relation extraction (Christopoulou et al., 2019), we encode both syntactic and semantic information in a heterogeneous graph. Nodes of different granularity are connected based on the feature structures. Node representations are updated iteratively through our defined message passing mechanism and incorporated into contextualized embeddings using an attentive integration module and gating mechanism. We conduct experiments on the OntoNotes 5.0 (Pradhan et al., 2012) benchmark, where the results show that our proposed model significantly outperforms a strong baseline. potential mentions and prunes unlikely spans aggressively. For each mention i , 2 the model learns a distribution over its possible antecedents Y ( i ) : P ( y ) = e s ( i,y ) (cid:80) y (cid:48) Y ( i ) e s ( i,y (cid:48) ) (1) where the scoring function s ( i, j ) measures how likely span i and j comprise valid mentions and corefer to one another: s ( i, j ) = s m ( i ) + s m ( j ) + s c ( i, j ) (2) s m ( i ) = FFNN m ( g i ) (3) s c ( i, j ) = FFNN c ( g i , g j ,  ( i, j )) (4) where g i and g j are span representations formed by the concatenation of contextualized embeddings of span endpoints and head vector using attention mechanism. FFNN represents the feedforward layer,  ( i, j ) are meta features including span distance and speaker identities, and s m and s c are the mention score and pairwise coreference score. Figure 2 shows the architecture of our proposed model, where the key components are presented in blue and orange backgrounds. Other parts follow Lee et al. (2018) (see 2) except that we use SpanBERT (Joshi et al., 2020) as the document encoder and discard the higher-order span refinement module as suggested by Xu and Choi (2020). There are three types of nodes in our heterogeneous graph: token nodes (T), argument nodes (A) and predicate nodes (P). The representation of token nodes and predicate nodes is the contextualized embeddings from the SpanBERT encoder, denoted as h w and h p respectively. The representation of an argument node is formed by averaging the embeddings of the tokens it contains, denoted as h a . Token-Token Edges are constructed according to dependency tree structures. Specifically, there will be a directed edge between two token nodes starting from head to dependent if they are connected, with edges being the corresponding dependency labels. A self-loop edge with cyclic label is 2 i is a span with one or more tokens. also added to each node in the graph. Besides, we also link the root nodes of two adjacent sentences to allow cross-sentence interaction. Token-Argument Argument nodes are linked to token nodes they contain. The edge is unlabelled but bidirectional to allow token-level information to augment the averaged representation of arguments and propagate semantic information back to tokens. Predicate-Argument Argument nodes are connected to predicate nodes they belong to with edges being the corresponding SRL labels. The edge is made bidirectional to allow mutual information propagation. Predicates can be regarded as intermediate nodes to allow each argument to aggregate information from other arguments with the same predicate. We use a Graph Attention Network (Velickovic et al., 2018) to propagate syntactic and semantic information to basic token nodes. For a node i , the attention mechanism allows it to selectively incorporate information from its neighbour nodes:  ij = softmax(  ( a T [ Wh i ; Wh j ; e ij ])) (5) h (cid:48) i = (cid:107) Kk =1 ReLU ( (cid:88) j  kij W k h j ) (6) where h i and h j are embeddings of node i and j , a T , W and W k are trainable parameters. e ij is the embedding of edge label type between node i and j based on graph structures,  is the LeakyReLU activation function. (cid:107) and [; ] represent the concatenation operation. Eqs. 5 and 6 are designated as an operation h (cid:48) i = GAT( h i , h j ) , where h i and h j are the embeddings of target and neighbour node and h (cid:48) i is the updated embedding of target node. To make each node embedding more informative, we update all nodes in the graph multiple times via our designed message passing path. First, we update token nodes using neighbour token nodes connected through dependency syntactic edges: h lw = GAT( h l  1 w , h l  1 w ) (7) where h l  1 w is the token representation in previous layer l  1 , h lw is the updated representation in current layer l and h 0 w is the SpanBERT encoding. In parallel, we update the argument using the token representation; then the updated argument is used to update the predicate features; after that, the updated predicate nodes propagate information back to their connected argument nodes; finally, the updated argument nodes distribute the representation to all connected basic token nodes: h la = GAT( h l  1 a , h l  1 w ) (8) h lp = GAT( h l  1 p , h la ) (9) h la = GAT( h la , h lp ) (10) h lw = GAT( h l  1 w , h la ) (11) After L iterations, we can get the final syntax and semantics-enhanced token representation, which can be denoted as h dw and h sw , respectively. 2020a,b), we use an attentive integration layer to selectively incorporate the syntactic and semantic information. For each type of information h cw  { h dw , h sw } , we concatenate it with initial token representation h 0 w and use the concatenation to compute the importance score of h cw to h 0 w :  c = softmax( FFNN c ([ h 0 w ; h cw ])) (12) where FFNN c is a one-layer feedforward network with sigmoid activation function for information type c (either Dep or SRL). After obtaining the valid attention weights using softmax function, we could compute the weighted average sum of both syntactic and semantic information: o = (cid:88) c { d,s }  c h cw (13) Since the extra syntactic and semantic information is not always useful, we use a gate to leverage such information dynamically: f =  ( W g  [ h 0 w ; o ] + b g ) (14) h (cid:48) w = f (cid:12) h 0 w + (1  f ) (cid:12) o (15) where W g and b g are trainable parameters, (cid:12) represents element-wise multiplication and  is the logistic sigmoid function. Finally, the augmented token representation h (cid:48) w can be used to form span representation and compute pairwise coreference score as in Section 2. 4 Experiments Dataset We evaluate our model on the English OnotoNotes 5.0 benchmark (Pradhan et al., 2012), which consists of 2802, 343 and 348 documents in the training, development and test data sets. Implementation Details We reimplement the c2f-coref+SpanBERT 3 baseline using PyTorch and use the Independent setup for long documents. For graph encoders, the number of heads of syntactic and semantic sub-graphs is 4 and 8 for base and large model, respectively. We set the size of edge label embeddings to 300 and use 2 GAT layers for both sub-graphs. More details are in Appendix A. Results The main evaluation is the average F1 of three metrics  MUC, B 3 and CEAF  4 on the test set using the official CoNLL-2012 evaluation scripts. 4 Table 1 shows the results of coref-HGAT 3 https://github.com/mandarjoshi90/coref 4 http://conll.cemantix.org/2012/software.html MUC B 3 CEAF  4 P R F1 P R F1 P R F1 Avg. +SpanBERT-base and large model compared with previous work. Our model consistently outperforms the SpanBERT baseline (Joshi et al., 2020) on all three metrics with an improvement of 1.4% and 1.5% on Avg. F1 score respectively, as well as our reimplemented baseline (+1.3% and +1.1%), which is a substantial improvement by considering the difficulty of this task. This demonstrates the effectiveness of our heterogeneous graph-based method in leveraging syntactic and semantic features and such features are indeed useful in neural methods. Note that we also show the current state-of-the-art CorefQA model (Wu et al., 2020), which uses span-prediction paradigm to compute pairwise coreference scores. The model is compatible with our method, i.e. adding our proposed graph attention and attentive integration layer on top of their document encoder with minor modification. The reason why we did not use it as a start baseline is due to hardware limitations since it requires 128G GPU memory for training. Ablation Study We perform ablation study on the test set to investigate the contribution of different features in our model, with results shown in Table 2. We can see that both dependency features and SRL labels individually contribute to the success of our final model with minor difference (+1.0% and 0.9%), and the gains are complementary to each other. Effect of #Graph Layers From Table 2, we can see that both using one layer and three layers hurt model performance. This indicates that first-order information is not effective in capturing long-range dependencies while third-order information may cause overfitting due to too much model capacity. Effect of Feature Quality To evaluate how the quality of features will affect the performance, we use the biaffine dependency parser (Dozat and Manning, 2017) and SRL parser (Shi and Lin, 2019) (denoted as CoNLL12-SRL ) implemented by Al-lenNLP (Gardner et al., 2018) as well as the Stanford Parser (Chen and Manning, 2014) to extract features. The biaffine parser has roughly 3% LAS improvements compared to the Stanford CoreNLP parser on Penn Treebank. Moreover, in order to Doc length #Docs Baseline Ours + F 1 0  128 57 82.9 85.4 +2.5 129  256 73 81.8 83.1 +1.3 257  512 78 82.2 83.2 +1.0 512  768 71 77.7 78.2 +0.5 769  1152 52 76.8 78.6 +1.8 1153+ 12 67.5 70.3 +2.8 All 343 77.8 79.2 +1.4 Table 4: The Avg. evaluate the impact of different SRL parsers, we also implemented the same model from Shi and Lin (2019) but trained on the CoNLL 2005 dataset (Car-reras and Mrquez, 2004) (denoted as CoNLL05-SRL ), which achieves an F1 of 81.9% on the out-of-domain setting. From Table 3, we observe that better parsers and parsers trained in closer domains result in higher Avg. F1 score, with improvements of up to 0.9%. Meanwhile, although our model suffers a performance drop from imperfect features, it can still achieve robust performance, outperforming the baseline with at least 0.6% improvement. Overall, high-quality features are important to good performance of the proposed model. Document Length In Table 4, we show the performance of our model against the baseline on the development set as a function of document lengths. As expected, our model consistently outperforms the baseline model on all document sizes, especially for documents with length larger than 765 tokens. This demonstrates that the incorporated external syntax and semantics are beneficial for modelling longer dependencies. However, our model has similar pattern as the baseline model, performing distinctly worse as document length increases. This shows that the sentence-level syntax and semantics used in this work are not sufficient enough to tackle the deficiency of modelling long-range dependency. One possible solution is to leverage document-level features such as hierarchical discourse structures. Graph Neural Networks (GNN) have long been used for integrating external features of graph structures into a range of NLP tasks, including semantic role labelling (Marcheggiani and Titov, 2017) and machine translation (Bastings et al., 2017). However, the application of GNN on coreference resolution task is less explored. Xu and Yang (2019) adopted dependency syntax to improve gendered pronoun resolution. However, they did not evaluate their model on larger datasets and identify whether syntax features are still useful for common coreference resolution. In this paper, we not only utilise syntax but also semantic features, and we show both of them contribute to significant improvement over a strong baseline on a large standard dataset. There are many GNN variants. Graph Convolutional Network (GCN) (Kipf and Welling, 2017) is the most widely-used one and has been shown to benefit a number of NLP tasks. However, it lacks the ability of modeling different edge labels including directions and edge types. Although Relational Graph Convolutional Network (RGCN) (Schlichtkrull et al., 2017) was proposed to tackle this problem, the way of representing edge information as label-wise parameters makes it suffer from over-parameteration problem even for small sized label vocabularies. In this work, we use a graph encoder improved based on Graph Attention Network (GAT) (Velickovic et al., 2018) to better capture structural syntax and semantics, as GAT is able to model different types of edges with few parameters. In this paper, we propose a heterogeneous-graph based model to enhance coreference resolution by effectively leveraging dependency tree structures and SRL semantic features. Particularly, nodes of different granularity in the graph propagate and aggregate information to and from neighbour nodes to obtain both syntactically and semantically augmented representation. Moreover, an attention-based mechanism is used to dynamically aggregate such augmented information. Experiments on the OntoNotes 5.0 benchmark confirm the effectiveness of our proposed model with significant improvement achieved against the strong baseline. Future work will focus on applying other features, such as constituent parsing trees and WordNet. We thank the anonymous reviewers for their helpful feedback. This research was undertaken using the LIEF HPC-GPGPU Facility hosted at the University of Melbourne. This Facility was established with the assistance of LIEF Grant LE170100200.
abstain method abstain objective objective abstain abstain other abstain other method abstain method abstain objective method method abstain method abstain method abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method other abstain abstain abstain abstain method abstain method method method other abstain method method abstain method abstain other abstain abstain method abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain other other other other objective other other other other abstain objective abstain abstain objective abstain other other other