Coreference resolution over semantic graphs like AMRs aims to group the graph nodes that represent the same entity. This is a crucial step for making document-level formal semantic representations. With annotated data on AMR coreference resolution, deep learning approaches have recently shown great potential for this task, yet they are usually data hungry and annotating data is costly. We propose a general pretraining method using variational graph autoencoder (VGAE) for AMR coreference resolution, which can leverage any general AMR corpus and even automatically parsed AMR data. Experiments on benchmarks show that the pretraining approach achieves performance gains of up to 6% absolute F1 points. Moreover, our model signifi-cantly improves on the previous state-of-the-art model by up to 11% F1 points. Abstract Meaning Representation (AMR) is a way to preserve the semantic meaning of a sentence in a graph (Banarescu et al., 2013). As shown in Figure 1, AMRs are directed and acyclic graphs where the nodes and edges indicate concepts and their semantic relations. As a sentence-level semantic representation, AMRs have been shown to be effective in many NLP tasks, including text summarization (Liu et al., 2015; Dohare et al., 2018), information extraction (Rao et al., 2017; Li et al., 2020b; Zhang and Ji, 2021), and machine translation (Song et al., 2019; Pham et al., 2020). More recently, the NLP tasks that are beyond the single-sentence level (Nallapati et al., 2016; Rajpurkar et al., 2016; Li et al., 2017; Chen et al., 2021) are attracting rising attention, and thus representing multiple sentences with AMR becomes important. To expand AMRs to represent multiple  Work done as an intern at Tencent AI Lab. sentences, the task of AMR coreference resolution (O'Gorman et al., 2018) has been proposed, aiming at recognizing the concepts from multiple AMRs that represent the same entity. Figure 1 illustrates the AMR graphs of two consecutive sentences in a news article. Given them as the input, an AMR coreference resolver needs to group police and they (colored with blue), as well as shop and the implicit mention shop (dashed and colored with pink). Unlike text-based coreference resolution, where dense textual information is available, AMR coreference resolution deals with sparsely connected graphs and implicit graph nodes. More importantly, only a handful of annotated data (around 8K AMRs) exists for AMR coreference resolution. Furthermore, annotating such coreference information and sentence AMRs requires linguists, making the annotation very costly . Both situations add extra difficulties to this task. Early attempts on AMR coreference resolution adopt rule-based methods. For instance, Liu et al. (2015) only consider the nodes that represent entities (e.g., police in Figure 1), and they rely on string match to detect coreference. This method can cause errors, as concepts with the same surface string may not point to the same entity. It also fails 2790 to recognize any situations that involve a pronoun (e.g., police and they ). Anikina et al. (2020) build a pipeline system that uses a textual coreference resolution model (Lee et al., 2017) and a text-to-AMR aligner (Flanigan et al., 2014). Though this system can theoretically resolve many situations, in fact, it suffers from severe error propagation (Fu et al., 2021). With the availability of recent human-annotated data (O'Gorman et al., 2018) on AMR coreference resolution, later work starts exploring data-driven models. Fu et al. (2021) extend a standard text-based coreference model (Lee et al., 2017) on AMRs by replacing the LSTM encoder with a graph neural network (GNN). They show a significant performance boost over previous rule-based methods, and their generated document-level AMRs can help a downstream neural summarization system, demonstrating the potential of this task. However, the performance is still far from satisfactory, and they find that the main reason is the lack of annotated data. This calls for approaches that can leverage cheap and/or existing supervision signals to make further improvements. In this paper, we propose a model and a corresponding pretraining method based on Variational Graph Autoencoder (VGAE) (Kipf and Welling, 2016b). Our model extends AMRCoref (Fu et al., 2021), the current state-of-the-art model, by replacing the core GNN encoder with an improved VGAE encoder. Our model can leverage the reconstruction loss and variational restriction from the VGAE module as additional supervision at no extra cost. Since the loss by our VGAE model can work on any AMR graphs, we also study pretraining our model on the full AMR bank 1 with gold or automatically parsed annotations. In this way, the training signal can be further enriched; thus, the data hunger issue can be alleviated. Though there exist some work applying VAEs and VGAEs on concept knowledge graphs (Li et al., 2020a), corpus-level graphs (Xie et al., 2021) and text (Su et al., 2018), we are the first to study VGAE on a graph-based formal semantic representation, to the best of our knowledge. Experiments on the MS-AMR benchmark (O'Gorman et al., 2018) show that our model outperforms the previous state-of-the-art system by 11 absolute F1-score points. Besides, we find that pretraining with a larger AMR bank is helpful re-1 https://catalog.ldc.upenn.edu/ LDC2020T02 DocumentAMR Graph FFNN & S o ft m a x FFNN & S o ft m a x Concept Identification CoreferenceClustering GraphEncoder GRNE n c ode r PredictClusters Input Representation Figure 2: AMRCoref framework (Fu et al., 2021). gardless of whether gold or silver AMR annotations are used. This indicates another potential boost on the performance if more automatically annotated data can be used. Code and pretrained models are made public 2 . We take the end-to-end AMR coreference resolution model ( AMRCoref , Fu et al. 2021) as our baseline system. Generally, it adapts a text-based end-to-end coreference model (Lee et al., 2017) on AMRs by clustering AMR nodes instead of text spans. Another major difference is that they also consider omitted AMR nodes (e.g., the dashed node shop in Figure 1), which are represented by their parent nodes and the corresponding relation (e.g., depart-01 and :ARG1 ). As illustrated in Figure 2, AMRCoref consists of four essential modules: input representation, graph encoding, node type identification, and antecedent prediction. As the first step of AMRCoref, it calculates the embedding h (0) i for each AMR node x i from its character-level embedding e ci , token-level embedding e ti and fixed embedding e berti generated by a pretrained BERT model: where W concept and b concept are model parameters. The character-level and token-level embeddings can be learned from scratch. One can choose to eliminate BERT embedding e berti as a simple base model. Next, the representations H (0) = [ h (0)1 , . . . , h (0) N ] of all AMR nodes X = [ x 1 , . . . , x N ] are sent to a graph encoder together with the AMR edges. Since the input AMRs are disconnected (each AMR alone represents a sentence), Fu et al. (2021) heuristically connect the root nodes of these sentence AMRs to make a connected graph G . Specifically, G = ( X, A ) , where the edge set A consists of both the original AMR edges and the added ones between pairs of roots. The graph encoder, f GRN , is based on the Graph Recurrent Network (GRN, Song et al. 2018; Beck et al. 2018). It utilizes the gated operations of an LSTM (Hochreiter and Schmidhuber, 1997) step to simultaneously update each node representation h i by exchanging information from its incoming N iin and outgoing neighbors N iout that can be easily obtained from the edge set A : m ( l  1) in = (cid:88) j  N iin [ h ( l  1) j ; r ij ] , m ( l  1) out = (cid:88) j  N iout [ h ( l  1) j ; r ij ] , h ( l ) i = LSTM( h ( l  1) i , [ m ( l  1) in ; m ( l  1) out ]) , (2) where each r ij represents the embedding of the edge from x i to x j . After L steps of information exchange, z i = [ h (0) i ; h ( L ) i ] is used as the representation of node x i for the next step. The concept identification subtask is to determine the type for each AMR node from 6 predefined candidate types. Taking Figure 1 as an example, these types are: func (functional node like and ), ent (entity node like police ), ver (regular verbal node like report-01 ), ver x ( x  [0 , 1 , 2] ) (verbal node with implicit argument like depart-01 ). Given the node representation z i from the graph encoder, a feed-forward network (FFNN type ) with softmax activation is adopted to calculate the probability distribution for its node type p typei : p typei = softmax(FFNN type ( z i )) . (3) This subtask is introduced for detecting implicit mentions as shown in Figure 1, and it can also provide additional supervision defined by cross-entropy loss: L type =  1 NN (cid:88) i =1 log p typei [ t i ] , (4) where  t i is the index of the correct node type for node x i . In the last step, coreference clusters are predicted by finding the antecedent for each AMR node. Taking node x i for example, the score of a precedent node x j being its antecedent is defined as: s ( x j , x i ) = f m ( x j ) + f m ( x i ) + f ant ( x j , x i ) , f m ( x i ) = FFNN m ([ z i ; p typei ]) , f an ( x j , x i ) = FFNN ant ([ z j , z i ]) , (5) where FFNN m classifies if the given node involves in a coreference link, and FFNN ant determines if the given node pair form a coreference relation. Next, the scores are normalized into a probability distribution via a softmax layer, and the probability p i,j for x j being the antecedent of x i is: p x j ,x i = e s ( x j ,x i ) (cid:80) x (cid:48) Y ( x i ) e s ( x (cid:48) ,x i ) , (6) where Y ( x i ) represents all precedents of x i . The antecedent loss is a marginal log-likelihood on all correct antecedents of all the nodes, given the gold clustering for node i is GOLD( x i ) : L ant =  log N (cid:89) i =1 (cid:88)  x Y ( x i )  GOLD( x i ) p  x,x i . Finally, the training loss is a combination of antecedent loss and node type prediction loss: This section describes our proposed model ( VG-AMRCoref ) that adopts Variational Graph Autoencoder (VGAE) to enable the cheap supervision of graph reconstruction. For fair comparison, we replace the original graph encoder of AMRCoref (Fig-ure 2) with our optimized VGAE module. By doing so, we make it possible to pretrain our model on other standard AMR data for stronger robustness and generalizability. We illustrate the model framework in Figure 3. (Sec.2.1), a VGAE graph encoder is applied to further encode the input graph nodes into the representations with more contextual information. VGAE consists of a local graph encoder and a a local graph decoder. Local Graph Encoder The local graph encoder functions as a typical graph neural network, where the node features in the l th layer are defines as: H ( l ) = f ( H ( l  1) , A ) . (9) A typical VGAE model usually applies a Graph Convolutional Network (GCN) (Kipf and Welling, 2016a) as its local graph encoder f GCN . Eq. 9 can be further defined as: f GCN (cid:0) H ( l ) , A (cid:1) =  (cid:16) (cid:101) D 12 (cid:101) A (cid:101) D  12 H ( l ) W ( l ) (cid:17) , (10) where  (  ) is the Sigmoid activation function, (cid:101) A = A + I , I is the identity matrix, and (cid:101) D is the diagonal node degree matrix of (cid:101) A . We study equipping the vanilla VGAE model with other major graph encoders, such as Graph Attention Network (GAT, Velickovic et al. 2017) and Graph Recurrent Network (GRN, Beck et al. 2018; Song et al. 2018), to better capture the contextual information of each node. The GAT encoder f GAT considers attention from the neighbors: f GAT (cid:16) H ( l ) , A (cid:17) =  ( (cid:88)  W ( l  1) H ( l  1) ) ,  = Attention( H ( l  1) ) , (11) and the definition of the GRN encoder f GRN is given in Eq. 2. This local graph encoder also takes L layers. Same with the baseline (Sec. 2.2), we choose the hidden layer features after encoding to be Z = [ H (0) ; H ( L ) ] for the next step. Besides, Z indicates the stochastic latent variable, and it is modeled by a Gaussian prior distribution (cid:81) i N ( z i , 0 , I ) . For z i  Z : q ( z i | X, A ) = N ( z i |  i , diag (  2 i )) , (12) we have  = f  ( X, A ) and log  = f  ( X, A ) . Local Graph Decoder The hidden layer representation Z is also fed into a local graph decoder of VGAE. This decoder reconstructs the edge set A from Z . Typically, it is calculated by dot-product: A (cid:48) =  ( ZZT ) , p ( A (cid:48) | Z ) = N (cid:89) i =1 N (cid:89) j =1 p (cid:0) A (cid:48) ij | z i , z j (cid:1) . (13) The loss from the VGAE module LV GAE is defined by the reconstruction loss on the edge set L edge and the variational restriction on the hidden parameters L var : LV GAE = L edge + L var = E q ( Z | X,A ) [log p ( A (cid:48) | Z )]  KL [ q ( Z | X, A ) || p ( Z )] , (14) where KL [ q (  ) || p (  )] is the Kullback-Leibler divergence between q and p . Next, the encoded AMR graph node Z from Eq. 12 is sent to the Concept Identification and Coreference Clustering step, which are described in Sec. 2.3 and 2.4. As shown in Figure 3, the overall loss L comes from three parts: VGAE loss LV GAE , concept type loss L type and the antecedent loss L ant , referring to Eq. 14, 4 and 7, respectively: L = LV GAE + L type + L ant . Eq. 14 shows that VGAE can be trained in a self-supervised way, which only needs node features X and the edge set A . So we propose to pretrain the VGAE graph encoder using AMR graphs when only AMR graphs are available. In this pretraining stage, the loss function L pt is defined as: L pt = LV GAE . After pretraining, the VGAE graph encoder will be fine-tuned on the coreference resolution downstream task. Datasets Following previous work, we choose the MS-AMR benchmark (O'Gorman et al., 2018), which has manually annotated coreference information over gold AMRs. It contains 273 documents for training, 9 for development and 9 for testing. In addition to the in-domain test set, we also evaluate on the Little Prince data ( LP ) that is annotated by (Fu et al., 2021) for out-of-domain evaluation. For pretraining, we choose the AMR bank 3.0 (LDC2020T02), the largest AMR corpus with only regular sentence-level AMRs. Please note that these AMRs are manually labeled and do not contain comprehensive document-level coreference annotations, thus they can not be utilized for task training. We consider this dataset as AMR-gold . To reduce the reliance on the annotated dataset, we conduct another setting, AMR-silver : we take the sentences of the AMR-gold dataset and apply a well-trained neural AMR parser (Van Noord and Bos, 2017) to generate silver AMR graphs. When doing this, a few documents failed because of postprocessing issues 3 , so one may notice that it has slight differences with AMR-gold, but we consider this to be acceptable. Smatch F1 score (Cai and Knight, 2013) on the silver results is 0.71, indicating an acceptable AMR parsing quality. We show the statistics in Table 1. Evaluation Metrics To be consistent with previous work (Fu et al., 2021), we apply three evaluation metrics and an average F1 of all: MUC F1 (Vilain et al., 1995), B 3 F1 (Bagga and Baldwin, 1998) and CEAF  4 F1 (Luo, 2005). Hyperparameters For all of the experiment, we follow Fu et al. (2021) to set hyperparameters for fair comparison. For instance, the character embedding and concept type dimension are 32; the concept embedding dimension is 256. The pretrained BERT-base-cased model is used. We choose the number of local graph encoder layer of VGAE to be 3, an empirical value following Fu et al. (2021), and provide more details in the Ablation Study later. The optimizer is Adam (Kingma and Ba, 2017). We report average results on 5 runs with different random seeds. Baselines We choose to compare with the following 4 models. Rule-based (Liu et al., 2015): it merges entity nodes with the same surface string to build document AMRs. Pipeline (Anikina et al., 2020): it combines a pretrained text-based coreference model and an AMR-to-text aligner into a pipeline, where the text-based coreference resolution results are projected onto AMRs via AMR-to-text alignments. AMRCoref and AM-RCoref+bert are the baselines (Section 2) without and with BERT features, respectively. Since the local graph decoder has multiple choices including GRN, GCN and GAT, as described in Eq. 9, so we compare the performance on the development set to select the best setting in Table 2. Results show that our model can get the best performance when applying GAT, so we choose this setting in the main experiments. beddings from scratch; VG-AMRCoref+pretrain first pretrains the VGAE encoder using AMR-gold, and then fine-tune on the task; VG-AMRCoref+pretrain+bert is a model that adds pretrained BERT embeddings further. These three models are using GAT as the graph encoder. To compare with Fu et al. (2021) that applies a GRN as the graph encoder, we also conducte the VG-AMRCoref (GRN) that applies the same encoder. Both VG-AMRCoref (GRN) and VG-AMRCoref can be fairly compared with AMRCoref, given that they use the same training data and are under the same setting (without BERT). When applying GRN, our model improves about 8.6% and 3.2% Average F1 gains on inand out-domain. When applying GAT, we could have a significant improvement, specifically, 20.7% and 9.1% Average F1 gains on inand out-domain. With pretraining, VG-AMRCoref+pretrain performs better than VG-AMRCoref, improving 1.8% and 5.8% on the Average F1 score. This shows that our graph pretraining approach that learns from external data is effective, especially on the out-domain. Finally, we can notice that small gains can be found in the two domains when integrating with BERT knowledge. A possible reason is that only fixed BERT embeddings are applied. Since AMRCoref is undertrained, we see BERT improves the F1 scores by a large margin there. Overall, our best model outperforms the best baseline by around 11.1% and 3.4% on inand out-domain. Besides, though there is a performance gap between the inand outdomain test sets, our model shows improvements on both two domains. One may notice a significant gap between the dev and test results when comparing Table 2 and 3, which is also reported by Fu et al. (2021). After a careful check on the data, we find that the average Model MUC B 3 CEAF  4 Avg. cluster sizes of the dev and test sets are 3.6 and 5.6, respectively. Since the model predicts as correct if the predicted ancestor is in the same cluster as the current mention, a larger cluster size gives better chances to make correct decisions. We also calculate the average distance between a mention to its closest ancestor, and the values for the dev and test sets are 7.1 and 5.8. This also indicates that the dev set is even more difficult. VGAE Loss We first study how the VGAE loss from Eq. 14 can affect model performance. We start with a basic setting: applying GAT as the graph encoder (GAT Encoder). Then we add variational restriction (+VGAE L var ), as well as the reconstruction loss of edge set (+VGAE L edge ). We show the results on the MS-AMR test set in Table 4. With variational loss L var , we see an improvement of about 2.4% of Average F1. And with the edge set reconstruction loss L edge , we see the Average F1 increases again by 1.4%. In total, we see an overall improvement of 3.6% with the VGAE loss. formance (Zhou et al., 2020; Fu et al., 2021), due to the over-smoothing issue led by message passing over multiple layers on the graph. We compare 1 to 5 graph layers in the VGAE encoder, and show the Average F1 score of two domains (test set) in Figure 4. When the number of layers is 3, the model achieves the best performance on both in-and out-domain. The performance increases from 1 to 3, and decreases from 3 to 5. This observation is consistent with the AMRCoref model. Pretraining Data Size Our main results have shown that pretraining on the AMR-gold dataset makes a significant difference, especially for out-domain. We further investigate if our model can benefit from silver AMR data. We compare the Average F1 score with different pretraining sizes of AMR-gold and AMR-silver in Figure 5. In both domains, the x-axis shows the number of pretraining data size. Gold and silver datasets have the same trend: more pretraining data leads to better performance. Though pretraining using the silver dataset is slightly worse than the gold dataset, our model can still improve. Specifically, while the AMR parser (Van Noord and Bos, 2017) is not the current state-of-the-art, results show that applying silver dataset is positive. In the future, we plan to optimize with better AMR parsers and larger datasets to see if the silver data may achieve even better results than the gold dataset. To further understand the predicted results of our model, we compare our best performed model (VG-AMRCoref+pretrain+bert) and the best baseline model (AMRCoref+bert) with two case studies. we keep a part of the content and highlight the coreference cluster tokens with different colors to indicate ground truth, base model prediction and our model prediction. Note that we illustrate both AMRs and original sentences to show the context better, while the sentences were not directly participated in the training and testing. This content piece shows a dialogue between two characters: me and little prince . In the ground truth, the coreference cluster is indicating little prince, and this can be easily recognized from the token prince in S1 and the token he in S5 and S6. However, to find out if the token I in S3 belongs to this cluster, one needs to read from S1. Because dialogues are going in turns, it is important to figure out which character said S3. Here, the answer should be little prince (token I means himself) and should be included in the cluster. This could be challenging due to the deep understanding of the previous content and also the difficulty of long dependency. Our model successfully recognized the coreference tokens in this situation. We illustrate another example from the MS-AMR test set in Figure 7. As can be observed form the ground truth, the highlighted tokens are indicating the coreference cluster of the main character in this article, I . The base model predicts a wrong answer in S1 ( who ), and misses the correct token I in that sentence. While both models ignore the token I in S2 and S3, compare with the base model, 2796 S1: For the little prince asked me abruptly -as if seized by a grave doubt -It is true , is n't it , that sheep eat little bushes ? (a / ask-01 :ARG0 (p / prince :mod (l2 / little)) :ARG1 (t / Consistent with the previous example case, both models tend to predict only a part of the ground truth that they are more confident with, in order to keep a reasonable good performance. While automatic evaluation only shows the overall performance, our case studies provide some interesting observations. The base and our model tend to predict fewer coreference nodes than the ground truth, but our model can capture larger and more accurate coreference clusters than the base model. Encoding AMRs using Graph Neural Networks Recently, graph neural networks (GNNs) have shown their simplicity and effectiveness in many NLP tasks, especially in encoding graph-structured input, such as knowledge graphs and other task-specific graphs (Li et al., 2020a; Xiong and Gao, 2019; Yin et al., 2019; Song et al., 2020b). Some methods are proposed to encode AMR graphs. For example, Graph Convolutional Networks (GCNs, Kipf and Welling 2016a) and some variations are well-studied for AMRs (Zhang et al., 2020; Cai and Lam, 2020). On the other S1: ...Well I might have signs of something on the autism spectrum but who does n't have one or two ? As a variant of GAT (Velickovic et al., 2017), relation-aware self-attention (Shaw et al., 2018) is recently proposed and has been shown more effective (Zhu et al., 2019; Song et al., 2020a) than other GNN variants on presenting AMRs for text generation. We have similar observations where GAT gives better results over GCN and GRN on encoding AMRs for AMR coreference resolution. Graph Pretraining Previous work shows that pretraining a model may bring better generalizability and performance gain, such as the pretrained language model, BERT (Devlin et al., 2018). There is limited research that focuses on pretraining graph neural networks. The work by Hu et al. (2019) proposes two methods to pretrain GNNs in both 2797 individual node level and the entire graph level. Though there are a few attempts to pretrain GNNs in a similar way with BERT, i.e., Graph Transformer (Dwivedi and Bresson, 2020), and Knowledge Graph Pretraining (Yu et al., 2020), there is still limited study in other NLP tasks. Our work fills this gap by taking advantage of knowledge learned from external data. Coreference Resolution Coreference resolution has long been an active research topic in NLP. Recently, Clark and Manning (2016) proposed a reinforcement learning approach to optimize a neural mention-ranking model for coreference. The first end-to-end neural coreference resolution method (Lee et al., 2017) targets span embeddings from context-dependent boundary representations using a head-finding attention mechanism. Then, Kantor and Globerson (2019) proposed the Entity Equalization mechanism to capture mentions in clusters using a neural network. Applying these textual coreference methods to AMR graphs requires extra AMR-to-text alignment, which can cause severe error propagation. To promote multi-sentence AMR coreference resolution, O'Gorman et al. (2018) annotated MS-AMR dataset, which considered coreferences, implicit role coreferences and bridging relations. Very recent work by Fu et al. (2021) is the first end-to-end AMR coreference resolution model for multi-sentence. This model achieves better and robust performance compared with selected baselines. This work proposed a new model (VG-AMRCoref) that is capable of self-supervised training for multi-sentence AMR coreference resolution. It applies VGAEs to encode document-level AMRs, significantly improving performance by up to 11% on the F1 score. We further proposed a simple but effective graph pretraining method using VGAEs, which can simultaneously boost in in-domain and out-domain performances. Analysis shows that potential boost performance may happen if more automatically parsed AMR data is available. One future work will focus on applying larger scale silver AMR datasets for pretraining to improve AMR coreference resolution. Another future direction is to investigate the generated document-level AMRs on more downstream tasks, like question answering and dialogue understanding.
abstain abstain abstain objective abstain result abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain result other abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain other other abstain abstain method method method method abstain abstain abstain abstain abstain other method abstain abstain method abstain method abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result result abstain result result abstain result result result abstain result abstain abstain abstain abstain abstain abstain abstain abstain result result result result abstain result abstain abstain abstain result abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method other other other other other abstain other other other other method other other other other other other other other objective abstain objective abstain abstain abstain