Abstract Meaning Representations (AMRs) capture sentence-level semantics structural representations to broad-coverage natural sentences. We investigate parsing AMR with explicit dependency structures and interpretable latent structures. We generate the latent soft structure without additional annotations, and fuse both dependency and latent structure via an extended graph neural networks. The fused structural information helps our experiments results to achieve the best reported results on both AMR 2.0 (77.5% Smatch F1 on LDC2017T10) and AMR 1.0 (71.8% Smatch F1 on LDC2014T12). Abstract Meaning Representations (AMRs) (Ba-narescu et al., 2013) model sentence level semantics as rooted, directed, acyclic graphs. Nodes in the graph are concepts which represent the events, objects and features of the input sentence, and edges between nodes represent semantic relations. AMR introduces re-entrance relation to depict the node reuse in the graphs. It has been adopted in downstream NLP tasks, including text summarization (Liu et al., 2015; Dohare and Karnick, 2017), question answering (Mitra and Baral, 2016) and machine translation (Jones et al., 2012; Song et al., 2019). AMR parsing aims to transform natural language sentences into AMR semantic graphs. Similar to constituent parsing and dependency parsing (Nivre, 2008; Dozat and Manning, 2017), AMR parsers mainly employ two parsing techniques: transition-based parsing (Wang et al., 2016; Damonte et al., 2017; Wang and Xue, 2017; Liu et al., 2018; Guo and Lu, 2018) use a sequence of transition actions  Part of work was done when the author was visiting Westlake University * Corresponding author. to incrementally construct the graph, while graph-based parsing (Flanigan et al., 2014; Lyu and Titov, 2018; Zhang et al., 2019a; Cai and Lam, 2019) divides the task into concept identification and relation extraction stages and then generate a full AMR graph with decoding algorithms such as greedy and maximum spanning tree (MST). Additionally, reinforcement learning (Naseem et al., 2019) and sequence-to-sequence (Konstas et al., 2017) have been exploited in AMR parsing as well. Previous works (Wang et al., 2016; Artzi et al., 2015) shows that structural information can bring benefit to AMR parsing. Illustrated by Figure 1, for example syntactic dependencies can convey the main predicate-argument structure. However, dependency structural information may be noisy due to the error propagation of external parsers. Moreover, AMR concentrates on semantic relations, which can be different from syntactic dependencies. For instance, in Figure 1, AMR prefers to select the coordination (i.e.  and ) as the root, which is different from syntactic dependencies (i.e.  came ). Given the above observations, we investigate the effectiveness of latent syntactic dependencies for AMR parsing. Different from existing work (Wang et al., 2016), which uses a dependency parser to provide explicit syntactic structures, we make use of a two-parameter distribution (Bastings et al., 2019) to induce latent graphs, which is differentiable under reparameterization (Kingma and Welling, 2014). We thus build a end-to-end model for AMR parsing with induced latent dependency structures as a middle layer, which is tuned in AMR training and thus can be more aligned to the need of AMR structure. For better investigating the correlation between induced and gold syntax, and better combine the strengths, we additionally consider fusing gold and induced structural dependencies into an align-free AMR parser (Zhang et al., 2019a). Specifically, we first obtain the input sentence's syntactic dependencies 1 and treat the input sentence as prior of the probabilistic graph generator for inferring the latent graph. Second, we propose an extended graph neural network (GNN) for encoding above structural information. Subsequently we feed the encoded structural information into a two stage align-free AMR parser (Zhang et al., 2019a) for promoting AMR parsing. To our knowledge, we are the first to incorporate syntactic latent structure in AMR parsing. Experimented results show that our model achieves 77.5% and 71.8% SMATCH F1 on standard AMR benchmarks LDC2017T10 and LDC2014T12, respectively, outperforming all previous best reported results. Beyond that, to some extent, our model can interpret the probabilistic relations between the input words in AMR parsing by generating the latent graph 2 . We adopt the parser of Zhang et al. (2019a) as our baseline, which treats AMR parsing as sequence-to-graph transduction. Our baseline splits AMR parsing into a two-stage procedure: concept identification and edge prediction . The first task aims to identify the concepts (nodes) in AMR graph from input tokens, and the second task is designed to predict semantic relations between identified concepts. We employ Stanford CoreNLP (Manning et al., 2014) to get the dependencies. 2 Our code will be available at: https://github. com/zhouqiji/ACL2020_AMR_Parsing . Formally, for a given input sequence of words w = (cid:104) w 1 , ..., w n (cid:105) , the goal of concept identification in our baseline is sequentially predicting the concept nodes u = (cid:104) u 1 , ..., u m (cid:105) in the output AMR graph, and deterministically assigning corresponding indices d = (cid:104) d 1 , ..., d m (cid:105) . Our baseline extends the pointer-generator network with self-copy mechanism for concept identification (See et al., 2017; Zhang et al., 2018a). The extended model can copy the nodes not only from the source text, but also from the previously generated list of nodes on the target side. The concept identifier firstly encodes the input sentence into concatenated vector embeddings with GloVe (Pennington et al., 2014), BERT (Devlin et al., 2019), POS (part-of-speech) and character-level (Kim et al., 2016) embeddings. Subsequently, we encode the embedded sentence by a two-layer bidirectional LSTM (Schuster and Paliwal, 1997; Hochreiter and Schmidhuber, 1997): h li = [  f l ( h l  1 i , h li  1 );  f l ( h l  1 i , h li +1 )] , where h li is the l -th layer encoded hidden state at the time step i and h 0 i is the embedded token w i . Different from the encoding stage, the decoder does not use pre-trained BERT embeddings, but employs a two-layer LSTM to generate the decoding hidden state s lt at each time step: s lt = f l ( s l  1 t , s lt  1 ) , where s l  1 t and s lt  1 are hidden states from last layer and previous time step respectively, and s l 0 is the concatenation of the last bi-directional encoding hidden states. In addition, s 0 t is generated from the concatenation of the previous node u t  1 embedding and the attention vector (cid:101) s t  1 , which combine both source and target information: (cid:101) s t = tanh ( W c [ c t ; s lt ] + b c ) , where W c and b c are trainable parameters, c t is the context vector calculated by the attention weighted encoding hidden states and the source attention distribution a t src following Bahdanau et al. (2015) The produced attention vector (cid:101) s is used to generate the vocabulary distribution: P vocab = softmax ( W vocab (cid:101) s t + b vocab ) , as well as the target attention distribution: e t tgt = v (cid:62) tgt tanh ( W tgt (cid:101) s 1: t  1 + U tgt (cid:101) s t + b tgt ) , a t tgt = softmax ( e t tgt ) , The source-side copy probability p src , target-side copy probability p tgt and generation probability p gen are calculated by (cid:101) s , which can be treated as generation switches: [ p src , p tgt , p gen ] = softmax ( W switch (cid:101) s t + b switch ) , The final distribution is defined below, if v t is copied from existing nodes: P (node) ( u t ) = p tgt t  1 (cid:88) i : u i = u t a t tgt [ i ] , otherwise: P (node) ( u t ) = p gen P vocab ( u t ) + p src n (cid:88) i : w i = u t a t src [ i ] , where a t [ i ] is the i -th element of a t , and then deterministically assigned the existing indices to the identified nodes based on whether the node is generated from the target-side distribution. Our baseline employs a deep biaffine attention classifier for semantic edge prediction (Dozat and Manning, 2017), which have been widely used in graph-based structure parsing (Peng et al., 2017; Lyu and Titov, 2018; Zhang et al., 2019a). The overall structure of our model is shown in Figure 2. First, we use an external dependency parser (Manning et al., 2014) to obtain the explicit structural information, and obtain the latent structural information via a probabilistic latent graph generator. We then combine both explicit and latent structural information by encoding the input sentence through an extended graph neural network. Finally, we incorporate our model with an align-free AMR parser for parsing AMR graphs with the benefit of structural information. We generate the latent graph of input sentence via the HardKuma distribution (Bastings et al., 2019), which has both continuous and discrete behaviours. HardKuma can generate samples from the closed interval [0 , 1] probabilisitcally . This feature allows us to predict soft connections probabilities between input words, which can be seen as a latent graph. Specifically, we treat embedded input words as a prior of a two-parameters distribution, and then sample a soft adjacency matrix between input words for representing a dependency. HardKuma Distribution The HardKuma distribution is derived from the Kumaraswamy distribution ( Kuma ) (Kumaraswamy, 1980), which is a two-parameters distribution over an open interval (0 , 1) , i.e., K  Kuma( a , b ) , where a  R > 0 and b  R > 0 . The Kuma distribution is similar to Beta distribution, but its CDF function has a simpler analytical solution and inverse of the CDF is: C  1 K ( u ; a , b ) = (cid:16) 1  (1  u ) 1 / b (cid:17) 1 / a , We can generate the samples by: C  1 K ( U ;  ,  )  Kuma(  ,  ) , where U  U (0 , 1) is the uniform distribution, and we can reconstruct this inverse CDF function by the reparameterizing fashion (Kingma and Welling, 2014; Nalisnick and Smyth, 2017). In order to include the two discrete points 0 and 1, HardKuma employs a stretch-and-rectify method with support (Louizos et al., 2017), which leads   The boy came and left Latent Graph Generator  1  2  3  4  5 Graph Encoder Align-Free Concept Identifier Explicit Graph Latent Graph 0.0 1.0 1.0 HardKuma  3  2  4  1  5  1  4 GELU Graph Fusion Layer  3  5  2 GELU GELU GELU GELU GELU GELU GELU GELU GELU       l1  l Figure 2: Stretch of the model which has four main components: (1) A latent graph generator for producing the soft-connected latent graph (3.1); (2) An extended syntactic graph convolutional network for encoding the structural information (3.2); (3) An align-free concept identification for concept node generation (2.2); (4) A deep biaffine classifier for relation edge prediction (2.3). the variable T  Kuma( a , b , l, r ) to be sampled from Kuma distribution with an open interval ( l, r ) where l < 0 and r > 0 . The new CDF is: CT ( t ; a , b , l, r ) = CK ( ( t  l ) / ( r  l ) ; a , b ) , We pass the stretched variable T  Kuma( a , b , l, r ) through a hard-sigmoid function (i.e., h = min(1 , max(0 , t )) ) to obtain the rectified variable H  HardKuma( a , b , l, r ) . Therefore, the rectified variable covers the closed interval [0 , 1] . Note that all negative values of t are deterministically mapped to 0 . In contrast, all samples t > 1 are mapped to 1 3 . Because the rectified variable is sampled based on Kuma distribution, HardKuma first sample a uniform variable over open interval (0 , 1) from uniform distribution U  U (0 , 1) , and then generate a Kuma variable through inverse CDF: k = C  1 K ( u ; a , b ) , Second, we transform the Kuma variable for covering the stretched support: t = l + ( r  l ) k, 3 Details of derivations can be found at (Bastings et al., 2019). Latent Graph We generate the latent graph of input words w by sampling from HardKuma distribution with trained parameters a and b . We first calculate the prior c of ( a , b ) by employing multihead self-attention (Vaswani et al., 2017): c a = Transfomer a ( v ) , c b = Transfomer b ( v ) , where v = (cid:104) v 1 , ..., v n (cid:105) is the embedded input words. Subsequently, we compute a and b as: a = Norm ( c a c Ta ) , b = Norm ( c b c Tb ) , where a i = (cid:104) a i 1 , ..., a in (cid:105) and b i = (cid:104) b i 1 , ..., b in (cid:105) , c a , c b  R n  n and Norm ( x ) is the normalization function. Hence, the latent graph L is sampled via learned parameters a and b : l ij  HardKuma( a ij , b ij , l, r ) . For a syntactic graph with n nodes, the cell A ij = 1 in the corresponding adjacent matrix represents that an edge connects word w i to word w j . An L-layer syntactic GCN of l -th layer can be used to represent A, where the hidden vector for each word w i at the l  th layer is: h ( l ) i =  ( n (cid:88) j =1  A ij W ( l ) h ( l  1) j /d i + b ( l ) ) , where  A = A + I with the n  n identity matrix I , d i = (cid:80) nj =1  A ij is the degree of word w i in the graph for normalizing the activation to avoid the word representation with significantly different magnitudes (Marcheggiani and Titov, 2017; Kipf and Welling, 2017), and  is a nonlinear activation function. In order to take benefits from both explicit and latent structural information in AMR parsing, we extend the Syntactic-GCN (Marcheggiani and Titov, 2017; Zhang et al., 2018b) with a graph fusion layer and omit labels in the graph (i.e. we only consider the connected relation in GCN). Specifically, we propose to merge the parsed syntactic dependencies and sampled latent graph through a graph fusion layer: F = L + (1   ) D where  is trainable gate variables are calculated via the sigmoid function, D and L are the parsed syntactic dependencies and generated latent graph respectively, and F represent the fused soft graph. Furthermore, F is a n  n adjacent matrix for the input words w , different from the sparse adjacent matrix A , F ij denote a soft connection degree from word w i to word w j . We adapt syntactic-GCN with a fused adjacent matrix F , and employ a gate mechanism: h ( l ) i = GELU ( L norm ( n (cid:88) j =1 G j ( F ij W ( l ) h ( l  1) j + b ( l ) ))) , We use GELU (Hendrycks and Gimpel, 2016) as the activation function, and apply layer normalization L norm (Ba et al., 2016) before passing the results into GELU. The scalar gate G j is calculated by each edge-node pair : G j =  ( h ( l  1) j   v ( l  1) +  b ( l  1) ) , where  is the logistic sigmoid function,  v and  b are trainable parameters. Similar to our baseline (Zhang et al., 2019a), we linearize the AMR concepts nodes by a pre-order traversal over the training dataset. We obtain gradient estimates of E ( ,  ) through Monte Carlo sampling from: E ( ,  ) = EU (0 ,I ) [log P ( node | u t , g  ( u , w ) ,  ) + log P t ( head | u k , g  ( u , w ) ,  ) + log P k,t ( label | l, g  ( u , w ) ,  )] +  covloss t where u t is the reference node at time step t with reference head u k and l is the reference edge label between u k and u j . The form g  ( u , w ) is short for the latent graph samples from uniform distribution to HardKuma distribution (3.1). Different from Bastings et al. (2019), we do not limit the sparsity of sampled latent graphs, i.e. we do not control the proportion of zeros in the latent graph, because we prefer to retain the probabilistic connection information of each word in w . Finally, we introduce coverage loss into our estimation due to reduce duplication of node generation (See et al., 2017). We directly generate the latent graph by the PDF function of HardKuma distribution with trained parameters a and b . In the concept identification stage, we decode the node from the final probability distribution P (node) ( u t ) at each time step, and apply beam search for sequentially generating the concept nodes u and deterministically assigning corresponding indices d . For edge prediction, we use a bi-affine classifier to calculate the edge scores under the generated nodes u and indices d : S = { score (edge) i,j | 0  i, j  m } . Similar to Zhang et al. (2019a), we apply a maximum spanning tree (MST) algorithm (Chu, 1965; Edmonds, 1967) to generate complete AMR graph and restore the re-entrance relations by merging the receptive nodes via their indices. We use two standard AMR corpora: AMR1.0 (LDC2014T12) and AMR 2.0 (LDC2017T10). AMR 1.0 contains 13051 sentences in total. AMR Data Parser F1(%) AMR 2.0 Cai and Lam (2019) 73.2 Lyu and Titov (2018) 74.4  0.2 Lindemann et al. (2019) 75.3  0.1 Naseem et al. (2019) 75.5 Zhang et al. (2019a) 76.3  0.1 w/o BERT 74.6 Zhang et al. (2019b) 77.0  0.1 Ours 77.5  0.2 w/o BERT 75.5  0.2 AMR 1.0 Flanigan et al. (2016) 66.0 Pust et al. (2015) 67.1 Wang and Xue (2017) 68.1 Guo and Lu (2018) 68.3  0.4 Zhang et al. (2019a) 70.2  0.1 w/o BERT 68.8 Zhang et al. (2019b) 71.3  0.1 Ours 71.8  0.2 w/o BERT 70.0  0.2 Table 1: Main results of SMATCH F1 on AMR 2.0 (LDC2017T10) and 1.0 (LDC2014T12) test sets. 2.0 is larger which is split into 36521, 1368 and 1371 sentences in training, development and testing sets respectively. We treat in AMR 2.0 as the main dataset in our experiments since it is larger. We tune hyperparameters on the development set, and store the checkpoints under best development results for evaluation. We employ the pre-processing and post-processing methods from Zhang et al. (2019a), and get the syntactic dependencies via Stanford Corenlp (Manning et al., 2014). We train our model jointly with the Adam optimizer (Kingma and Ba, 2015). The learning rate is decayed based on the results of development set in training. Training takes approximately 22 hours on two Nivida GeForce GTX 2080 Ti. Main Results We compare the SMATCH F1 scores (Cai and Knight, 2013) against previous best reported models and other recent AMR parsers. Table 1 summarizes the results on both AMR 1.0 and AMR 2.0 data sets. For AMR 2.0, with the benefit from the fused structural information, we improve our baseline (Zhang et al., 2019a) by 1.2% F1 in the full model, and 0.9% F1 Metric N'19 Z'19 a Z'19 b Ours SMATCH 75.5 76.3 77 77.5 Unlabeled 80 79 80 80.4 No WSD 76 77 78 78.2 Reentrancies 56 60 61 61.1 Concepts 86 85 86 85.9 Named Ent. In addition, our model outperform the best reported model (Zhang et al., 2019b) by 0.5% F1. On AMR 1.0, there are only about 10k sentences for training. We outperform the best results by 0.5% Smatch F1. We observe that for the smaller data set, our model has a greater improvement of 1.6% F1 than for the larger data set (1.2% F1 comparing with our baseline.) Fine-grained Results Table 2 shows fined-grained parsing results of each sub-tasks in AMR 2.0, which are evaluated by the enhance AMR evaluation tools (Damonte et al., 2017). We notice that our model brings more than 1% average improvement to our baseline (Zhang et al., 2019a) for most sub-tasks, in particular, the unlabeled is gained 1.4% F1 score increasing with the structural information, and the sub-task of no WSD, reentrancies, negation and SRL are all improved more than 1.0% score under our graph encoder. In addition, our model achieves comparable results to the best reported method (Zhang et al., 2019b) for each subtask. Ablation Study We investigate the impacts of different structural information in our model on AMR 2.0 with main sub-tasks 5 . Table 3 shows the fused structure perform better in most sub-task than explicit and latent structure. In particular, the model with explicit structures (i.e. both explicit and fused) 4 We use pre-tained bert-base embedings without fine-tuning. outperform the model with only latent structure by 0.5% F1 in Reentrancies sub-task, which demonstrates that the explicit dependencies information can improve the this sub-task. Latent structure perform better in concepts sub-task, and fused structure brings more information to the negation subtask which obtain 0.5% and 1.0% improvement than explicit and latent structure respectively. Additionally, we can notice that both latent and explicit models outperform the previous best reported Smatch F1 score, and fused model reach the best results. It shows that different types of structural information can help the AMR parsing, we discuss the connection tendencies of each structure in (4.3). Experiment results show that both the explicit structure and latent structure can improve the performance of AMR parsing, and latent structural information reduces the errors in sub-tasks such as concept and SRL. Different from the discrete relation of explicit structures, the internal latent structure holds soft connection probabilities between words in the input sentence, so that, each fully-connected word receive information from all the other words. Figure 3 depicts the latent and fused soft adjacent matrix of the input sentence The boy came and left respectively. It can be seen that the la-(a) Latent Matrix tent matrix (Figure 3a) tries to retain information from most word pairs, and the AMR root and holds high connection probabilities to each word in the sentence. In addition, the main predicates and arguments in the sentence tend to be connected with high probabilities. The fused matrix (Fig-ure 3b) holds similar connection probabilities to predicates and arguments in the sentence as well, and it reduces the connection degrees to the determiner The which does not appear in corresponding AMR graph. Moreover, the syntactic root came and semantic root and reserve most connection probabilistic to other words. We compare the connections in different structures in Figure 4. The latent graph (Figure 4a) prefers to connect most words, and the main predicates and arguments in the graph have higher connection probabilities. The fused graph (Figure 4c) shows that our model provides core structural information between interpretable relations. Specifically, it holds similar potential relations to annotated AMR graph, and tries to alleviate the connection information to the words which are not aligned in AMR concept nodes. Beyond that, we calculate the Unlabeled Attachment Score (UAS) for fused and latent graph in Table 4, the unsupervised latent graph captures less explicit edges than fused graph, and both fused and latent graph ignore some arcs on explicit graph. It shows that a lower UAS does not mean lower AMR parsing score and some arcs are more useful to AMR parsing but not in explicit gold trees. Consequently, we preserve the explicit and latent structure information simultaneously. The latent structure can not only improve AMR parsing, but also have ability to interpret the latent connections between input words. Transition-based AMR parsers (Wang et al., 2016; Damonte et al., 2017; Wang and Xue, 2017; Liu et al., 2018; Guo and Lu, 2018; Naseem et al., 2019) suffer from the lack of annotated alignments between words and concept notes is crucial in these models. Lyu and Titov (2018) treat the alignments as an latent variable for their probabilistic model, which jointly obtains the concepts, relations and alignments variables. Sequence-to-sequence AMR parsers transform AMR graphs into serialized sequences by external traversal rules, and then restore the generated the AMR sequence to avoid aligning issue (Konstas et al., 2017; van Noord and Bos, 2017). Moreover, Zhang et al. (2019a) extend a pointer generator (See et al., 2017), which can generate a node multiple times without alignment through the copy mechanism. With regards to latent structure, Naradowsky et al. (2012) couples syntactically-oriented NLP tasks to combinatorially constrained hidden syntactic representations. Bowman et al. (2016); Yo-gatama et al. (2017) and Choi et al. (2018) generate unsupervised constituent tree for text classification. The latent constituent trees are shallower than human annotated, and it can boost the performance of downstream NLP tasks (e.g., text classification). Guo et al. (2019) and Ji et al. (2019) employ self-attention and bi-affine attention mechanism respectively to generate soft connected graphs, and then adopt GNNs to encode the soft structure to take advantage from the structural information to their works. GCN and its variants are increasingly applied in embedding syntactic and semantic structures in NLP tasks (Kipf and Welling, 2017; Marcheggiani and Titov, 2017; Damonte and Cohen, 2019). Syntactic-GCN tries to alleviate the error propagation in external parsers with gates mechanism, it encodes both relations and labels with the gates, and filters the output of each GCN layer over the dependencies. (Marcheggiani and Titov, 2017; Bastings et al., 2017). Damonte and Cohen (2019) encodes AMR graphs via GCN to promote the AMR-to-text generation task. We investigate latent structure for AMR parsing, and we denote that the inferred latent graph can interpret the connection probabilities between input words. Experiment results show that the latent structural information improve the best reported parsing performance on both AMR 2.0 (LDC2017T10) and AMR 1.0 (LDC2014T12). We also propose to incorporate the latent graph into other multi-task learning problems (Chen et al., 2019; Kurita and Sgaard, 2019). We thank the anonymous reviewers for their detailed comments. We are grateful to Zhiyang Teng's discussions and suggestions. This work is supported by the National Natural Science Foundation of China (NSFC-61772378), the National Key Research and Development Program of China (No.2017YFC1200500) and the Major Projects of the National Social Science Foundation of China (No.11&ZD189). We also would like to acknowledge funding support from the Westlake University and Bright Dream Joint Institute for Intelligent Robotics.
abstain objective method result abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain objective method method abstain objective objective method objective result method method method abstain method other abstain other abstain abstain abstain method other other method abstain abstain method abstain method abstain abstain method abstain method other abstain other abstain abstain abstain other method abstain method abstain abstain other abstain abstain abstain other abstain abstain other abstain method abstain method method method method method abstain other abstain method method method method abstain abstain result abstain result result abstain result result result result abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain other other other other other other other other other other other other objective abstain abstain other other other other