Abstract meaning representation (AMR) highlights the core semantic information of text in a graph structure. Recently, pre-trained language models (PLMs) have advanced tasks of AMR parsing and AMR-to-text generation, respectively. However, PLMs are typically pretrained on textual data, thus are sub-optimal for modeling structural knowledge. To this end, we investigate graph self-supervised training to improve the structure awareness of PLMs over AMR graphs. In particular, we introduce two graph auto-encoding strategies for graph-to-graph pre-training and four tasks to integrate text and graph information during pre-training. We further design a unified framework to bridge the gap between pre-training and fine-tuning tasks. Experiments on both AMR parsing and AMR-to-text generation show the superiority of our model. To our knowledge, we are the first to consider pre-training on semantic graphs. Abstract meaning representation (AMR; Banarescu et al. (2013)) is a semantic structure formalism. It represents the meaning of a text in a rooted directed graph, where nodes represent basic semantic units such as entities and predicates, and edges represent their semantic relations, respectively. One example is shown in Figure 1(a), with the corresponding sentence in Figure 1(b). Serving as a structural representation, AMR has been shown useful for NLP tasks such as text summarization (Liu et al., 2015; Liao et al., 2018; Chen et al., 2021), machine translation (Song et al., 2019), information extraction (Huang et al., 2016; Zhang and Ji, 2021) and dialogue systems (Bai et al., 2021). There are two fundamental NLP tasks concerning AMR, namely AMR parsing (Flanigan et al., 2014; Konstas et al., 2017; Lyu and Titov, 2018; Guo and Lu, 2018; Zhang et al., 2019a; Cai and Lam, 2020; Bevilacqua et al., 2021) and AMR-to-text generation (Konstas et al., 2017; Song et al., possible :domain go-01 :polarity negative :arg0 boy 2018; Zhu et al., 2019; Zhao et al., 2020; Bai et al., 2020; Ribeiro et al., 2021a). As shown in Figure 1, the former transforms a textual input ( e.g. , a sentence) into a corresponding AMR structure, and the latter transforms an AMR input into a fluent and grammatical sentence that conveys the same meaning. A common challenge to both tasks is that AMR exists in the form of a graph structure, which is difficult for neural models to learn with limited human-curated data. Recently, large-scale pre-trained sequence-to-sequence (seq2seq) language models (Lewis et al., 2020; Raffel et al., 2020) have been shown useful for both tasks above. The basic idea is to linearize AMR structures into a sequence form, so that both AMR parsing and AMR-to-text generation can be solved as standard seq2seq tasks, using a pre-trained language model fine-tuned on task-specific data. In this way, semantic knowledge learned in self-supervised text-to-text ( t2t ) pretraining can benefit both text-to-graph ( t2g ) and graph-to-text ( g2t ) transformation. Intuitively, structural knowledge from AMR can be a useful complement to semantic knowledge from text. A natural question is whether similar self-supervision strategy can be useful for AMR graphs, so that graph-to-graph ( g2g ) denoise auto-encoder training can serve as effective addition to t2t pre-training, before a model is fine-tuned on t2g and g2t tasks. We investigate this problem in this paper. In particular, there are three questions of interest. First, as mentioned before, is g2g 6001 pre-training complementary to t2t pre-training? Second, what is the most effective way to combine t2t and g2g training? Third, is silver data useful for AMR self-supervision training, and what is the most effective way of making use of such data? Taking BART (Lewis et al., 2020) as the seq-to-seq model, we introduce two strategies for g2g pre-training and propose four tasks to combine t2t and g2g training. To reduce the gap among different pre-training tasks and between pre-training and fine-tuing, we unify all pre-training tasks and fine-tuning tasks in a general framework. Experimental results on standard benchmarks show that: 1) graph pre-training achieves significant improvements over the state-of-the-art systems; 2) silver data are useful for our pre-training framework; 3) our pre-training framework is a better way than fine-tuning to make use of silver data and; 4) our model is more robust than existing systems in unseen domains. Our final models give the best reported results on both AMR parsing and AMR-to-text generation tasks, with a large margin of improvement over the previous best results. To our knowledge, we are the first to consider graph-to-graph self-supervised training on semantic graphs. We release code at https://github.com/muyeby/AMRBART . AMR Parsing. Early AMR parsing systems use statistical methods (Flanigan et al., 2014, 2016; Wang et al., 2015a,b). With the advance in deep learning, various neural models are developed for AMR parsing. Those models can be categorized into: 1) neural transition-based parsers (Ballesteros and Al-Onaizan, 2017; Liu et al., 2018; Fernandez Astudillo et al., 2020; Zhou et al., 2021); 2) sequence-to-graph parsers (Zhang et al., 2019a; Lyu et al., 2020; Cai and Lam, 2020) and; 3) sequence-to-sequence parsers (Konstas et al., 2017; Peng et al., 2017, 2018; Zhang et al., 2019b; Xu et al., 2020; Bevilacqua et al., 2021). Recently, pretraining techniques have significantly boosted the performance of AMR parsing. For example, Lyu and Titov (2018), Zhang et al. (2019a,b) and Cai and Lam (2020) use BERT (Devlin et al., 2019) for sentence encoding; Bevilacqua et al. (2021) fine-tune BART for sequence-to-AMR generation. Xu et al. (2020) pre-train a model on relevant seq2seq learning tasks (e.g., machine translation (Bahdanau et al., 2015), syntactic parsing (Zhu et al., 2013)) before fine-tuning on AMR parsing. Similar to those methods, we consider using pre-trained models to improve the model capacity. However, previous studies focus on fine-tuning language models trained on text data for AMR parsing task, in contract, we focus on integrating structural information into the pre-training. In addition, our method does not require information from auxiliary tasks. AMR-to-Text Generation. On a coarse-grained level, we categorize existing AMR-to-text generation approaches into two main classes: Graph-to-sequence models that adopt a graph encoder to process an AMR graph and use a sequence decoder for generation (Beck et al., 2018; Damonte and Cohen, 2019; Zhu et al., 2019), and sequence-to-sequence models that linearize an AMR graph into a sequence and solve it as a seq2seq problem using randomly initialized (Konstas et al., 2017) or pretrained models (Mager et al., 2020; Ribeiro et al., 2021a; Bevilacqua et al., 2021). This work follows a seq2seq manner, but we use an encoder that integrates AMR and text information. The closest to our work, Ribeiro et al. (2021b) integrate AMR structures into pre-trained T5 (Raffel et al., 2020) using adapters (Houlsby et al., 2019) for AMR-to-text generation. However, they do not pre-train on AMR graphs, and their method cannot solve both AMR parsing and AMR-to-text generation tasks as they require the full AMR structure as the input. Graph Self-supervised Learning. Kipf and Welling (2016) introduce a variational graph auto-encoder to allow self-supervised learning on graph data. Hu et al. (2020a,b) propose local and global learning strategies to pre-train a graph neural network on large-scale protein ego-networks, aca-demic graphs and recommendation data. Lu et al. (2021) enhance the graph learning strategies of Hu et al. (2020b) with dual adaptations. While existing work considers graph neural networks, we pre-train a seq2seq model on AMR graphs. In addition, we jointly pre-train on graphs and text for graph-text correlation modeling. In contrast, existing work pre-trains models on graphs and in isolation with text pre-training. To our knowledge, we are the first to consider AMR as a graph pre-training target. We take BART (Lewis et al., 2020) as the basic seq2seq model (Section 3.1), and introduce graph pre-training strategies (Section 3.2) and an unified pre-training framework (Section 3.3) for both AMR parsing and AMR-to-text generation. BART (Lewis et al., 2020) is a pre-trained denoising auto-encoder, which is implemented as a seq2seq model based on standard Transformer (Vaswani et al., 2017) architecture. Typically, BART is trained to reconstruct original text based on a corrupted text generated by 5 noising functions: 1) Token Masking. Tokens are randomly replaced by [mask] elements; 2) Token Deletion. Tokens are randomly deleted from the input; 3) Text Infilling. Text spans are randomly replaced by a single [mask] token; 4) Sentence Permutation. Text is divided into segments and then shuffled; 5) Document Rotation. A document is rotated to start with a random token. In the fine-tuning, BART takes a complete text as input and maps it into a task-specific output sequence. We linearize an AMR graph into a sequence, so that both AMR parsing and AMR-to-text generation can be performed using a seq2seq model. In addition, it allows pre-training on AMR structures using BART. Following Konstas et al. (2017), we adopt the depth-first search (DFS) algorithm, which is closely related to the linearized natural language syntactic trees (Bevilacqua et al., 2021). For instance, the AMR graph in Figure 1 is linearized into: ( <Z0> possible :domain ( <Z1> go :arg0 ( <Z2> boy ) ) :polarity ( <Z3> negative ) ) , where <Z0> , <Z1> and <Z2> are special tokens to handle co-referring nodes. To deal with such AMR symbols, we follow previous work (Bevilac-qua et al., 2021) and expand the vocabulary by adding all relations and frames. In addition, to distinguish between texts and AMR graphs, we add two special tokens, <g> and < / g> , to mark the beginning and end of AMR graphs, respectively. We introduce two self-supervised training strategies to further pre-train a BART model on AMR graphs. As shown in Figure 2(a), the node/edge level denoising strategy encourages the model to capture local knowledge about nodes and edges. The graph level denoising strategy (Figure 2(c)) enforces the model to predict a sub-graph, thus facilitating the graph-level learning. 1) Node/edge level denoising. We apply a noise function on AMR nodes/edges to construct a noisy input graph. In particular, the noise function is implemented by masking 15% nodes and 15% edges in each graph. As shown in Figure 2(a), the node [go-01] and edge [:arg0] are replaced with two [mask] tokens. 2) Sub-graph level denoising. This task aims to recover the complete graph when given part of the graph. We randomly remove a sub-graph 1 from the graph and replace it with a [mask] token ( cf. Figure 2(c)). The masking probability is 0 . 35 . The above standard pre-training and fine-tuning strategies are shown in Table 1(a), by using <s> and <g> for differentiating text and graphic information, respectively. However, the model does not fully learn the interaction between textual and AMR information during pre-training. To further address this issue, we consider a unified pretraining framework, which combines text and AMR sequences as input to the denoise auto-encoder. In this way, dynamic masking can be carried out on the text, AMR or both ends, so that the model can learn to make use of one source of information for inferring the other. This can benefit both a parser and a generation model by enforcing the learning of correspondence between text and AMR structures. In addition, as shown in Table 1, there is a gap between standard pre-training and fine-tuning for AMR from/to text transduction. Specifically, the input and output formats are same in the pre-training ( i.e. ,  t2t and  g2g ) but different in the fine-tuning 1 We define a sub-graph has at least one edge and one node. ( i.e. , t2g and g2t ). This gap restrains models to make the best use of pre-trained knowledge in the fine-tuning phase. The unified pre-training framework can also benefit task-specific fine-tuning by eliminating the difference of input and output formats between pre-training and fine-tuning. Formally, denoting the text and linearized graph sequence as t and g , where t = { x 1 , x 2 , ..., x n } and g = { g 1 , g 2 , ..., g n } . t and  g represent the noisy text and graph, respectively, and t and g refer to the empty text and graph, respectively. As shown in Table 1(b), we unify the input format for both pre-training and fine-tuning to tg . For consistency, all input sequences start with a text sequence and end with a graph sequence. Joint Text and Graph Pre-training. We introduce 4 auxiliary pre-training tasks to encourage information exchanging between graphs and text. As shown in Table 1(b), the auxiliary tasks are: 1) Graph augmented text denoising (  tg2t ), where an AMR graph is taken as additional input to help masked text reconstruction; 2) Text augmented graph denoising ( t  g2g ), where text helps masked graph reconstruction; 3) Noisy graph augmented text denoising (  t  g2t ), where the target text is generated based on a pair of masked text and masked graph; 4) Noisy text augmented graph denoising (  t  g2g ), where a target graph is generated based on a pair of masked text and masked graph. Dynamic masking rate. Different from standard masking (Devlin et al., 2019) that uses a static masking rate, we adopt a dynamic masking rate p for task  tg2t and t  g2g . Formally, at step t , we calculate the masking probability p as: p = 0 . where 0 . 1 is the initial masking rate, T denotes the total training step. p increases as t grows, as t approaches to T , the pre-training tasks  tg2t and t  g2g are closer to fine-tuning tasks. Unified Pre-training and Fine-tuning. In our unified framework, fine-tuning tasks can be viewed as having an empty text/graph in the original input, resulting in an input format of tg2t for AMR-to-text generation and tg2g for AMR parsing. In this way, pre-training and fine-tuning tasks share the same input format, thus facilitating knowledge transfer from pre-training to fine-tuning. To pre-train our model, we optimize the total loss ( L total ) which is calculated as: L  t2t =  log P ( t |  t , g ) , L  g2g =  log P ( g | t ,  g ) , L  tg2t =  log P ( t |  t , g ) , L t  g2g =  log P ( g | t ,  g ) , L  t  g2t =  log P ( t |  t ,  g ) , L  t  g2g =  log P ( g |  t ,  g ) , L total = L  t2t + L  g2g + L  tg2t + L t  g2g + L  t  g2t + L  t  g2g , (2) where L  t2t and L  g2g are standard pre-training loss on text (Section 3.1) and graph (Section 3.2), respectively. L  tg2t , L t  g2g , L  t  g2t and L  t  g2g denote joint pre-training losses (Section 3.3), respectively. For fine-tuning, the training objectives are: L amr2text =  log P ( t | t , g ) , L text2amr =  log P ( g | t , g ) , (3) where L amr2text and L text2amr are training loss of AMR-to-text generation and AMR parsing, respectively. We evaluate the effectiveness of our model on five benchmarks and compare the results with state-of-the-art models on AMR parsing and AMR-to-text generation, respectively. In addition to standard supervised training settings, we evaluate the robustness of our model in a zero-shot domain adaptation setting. Table 2 shows the statistics of datasets. Following Bevilacqua et al. (2021), we use the AMR2.0 (LDC2017T10) and AMR3.0 (LDC2020T02). We also evaluate the model performance on New3 , The Little Prince ( TLP ) and Bio AMR ( Bio ) corpora. For pre-training, we additionally use 200k silver data parsed by SPRING (Bevilacqua et al., 2021). These data are randomly selected from Gigaword (LDC2011T07) corpus, which shares the same textual source with AMR data. 2 4.2 Settings We follow Bevilacqua et al. (2021) in preprocessing and post-processing AMR graphs, except for omitting the recategorization step which does not consistently improve model performance in our preliminary experiments. Our model is built based on a vanilla BART 3 . The best model and hyper-parameters are selected by performance on the validation set. The detailed hyper-parameters are given in Appendix A. Metrics. Following Bevilacqua et al. (2021), we evaluate on the AMR parsing benchmarks by using Smatch (Cai and Knight, 2013) and other fine-grained metrics. 4 Regarding AMR-to-text, we use three common Natural Language Generation measures, including BLEU (Papineni et al., 2002), CHRF++ (Popovic, 2017) and METEOR (Baner-jee and Lavie, 2005), tokenizing with the script provided with JAMR (Flanigan et al., 2014). For AMR parsing , we consider following systems for comparison: 1) Lyu and Titov (2018; LyuT), a neural parser trained by jointly modeling alignments, concepts and relations; 2) Zhang et al. (2019b; Zhang+), a seq2seq approach that incrementally builds up an AMR via predicting semantic relations; 3) Zhou et al. (2020; Zhou+), an aligner-free parser enhanced by explicit dependency and latent structures; 4) Cai and Lam (2020a; CaiL), a graph-based parser that enhances incremental sequence-to-graph model with a graph-sequence iterative inference mechanism; 5) Bevilacqua et al. (2021; Bevilacqua+), a fine-tuned BART model that predicts a linearized AMR graph. For AMR-to-text generation , the compared models are: 1) Zhu et al. (2019; Zhu+), a Transformer-based model that enhances self-attention with graph relations; 2) Zhang et al. (2020; Zhang+), a graph-to-sequence model which uses a dynamic graph convolutional networks for better graph modeling. 3) Bai et al. (2020; Bai+), a graph encoder (Zhu et al., 2019) with a structural decoder that jointly predicts the target text and the input structure; 4) Mager et al. (2020; Mager+), a fine-tuned GPT that predicts text based on a PENMAN linearized AMR graph; 5) Bevilacqua et al. (2021; Bevilacqua+), a fine-tuned BART that predicts text based on a DFS linearized AMR graph; 6) Ribeiro et al. (2021; Ribeiro+), a fine-tuned BART based on a PENMAN linearized AMR graph. For a fair comparison, we leave out models based on T5 (Ribeiro et al., 2021a,b), which has about two times more parameters than BART. 6005 Setting AMR parsing AMR-to-text Full Model 83.6 45.6 Node/edge masking 83.4 45.1 Sub-graph masking 83.1 44.7 Table 4: Impact of two masking strategies. Table 3 shows results on the validation set of AMR2.0 under different model settings, where we take a fine-tuned BART-based model (Bevilacqua et al., 2021) as our baseline. We first study the effectiveness of pre-training only on text and graphs. As shown in Table 3, both pre-training on the text (  tg2t ) and graph ( t  g2g ) leads to better results, and combining them can give better results on both tasks. Also, adding joint pre-training tasks improves the performance. In particular, t  g2g gives a Smatch improvement of 0 . 7 for AMR paring, and  tg2t reaches a BLEU of 45 . 3 for AMR-to-text generation, which is 2 . 8 points higher than baseline. Adding  t  g2g gives a Smatch of 83 . 2 for AMR parsing, and  t  g2t improves the baseline by 1 . 7 BLEU points for generation. By combining t  g2g and  tg2t , the performance increase by 0 . 6 and 2 . 5 points on AMR parsing and AMR-to-text generation, respectively. Similar trend can be observed by combining  t  g2g and  t  g2t . Finally, using all 6 pre-training tasks, our model reach a result of 83 . 6 Smatch and 45 . 6 BLEU, respectively. We also study the impact of two graph self-supervised training strategies. In particular, we evaluate the performance after removing the node/edge or the sub-graph masking task independently. As shown in Table 4, the performance decreases on both AMR parsing and AMR-to-text generation tasks without the node/edge level masking strategy. The performance drop is larger when removing the sub-graph masking task, with a margin of 0 . 5 Smatch and 0 . 9 BLEU, respectively. Figure 3(a) compares the performance of standard pre-training (  t2t ,  g2g ) and fine-tuning ( t2g , g2t ) with our unified framework. The unified framework gives better results than standard versions on both tasks. This confirms our assumption that our unified framework is helpful for reducing the gap between pre-training and fine-tuning. Besides, we find that by unifying pre-training and fine-tuning formats, our model converges faster than the baseline during fine-tuning ( cf. Appendix C.1). Figure 3(b) shows the model performance regarding different scales of silver data. Even without silver data, the performance of our model is better than the baseline, indicating that graph pretraining is beneficial for downstream tasks when using various auxiliary tasks. When silver data are available, the performance of both AMR parsing and AMR-to-text generation tasks increases as the scale of silver data increases, with a margin of 2 BLEU score. We also fine-tune a BART model on silver data under our unified framework (i.e., tg2t and tg2g ), and find that our dual graph and text denoising tasks are more useful ( cf. Appendix C.2 for more analysis and discussion). AMR parsing . Table 5 lists the result of different models on AMR2.0 and AMR3.0. Among previous works, Bevilacqua+ (2021, large) achieves the best results, consistently outperforming other systems. Compared with the system of Bevilacqua et al. (2021), our model obtains significantly ( p <0.01) better Smatch scores in both base and large settings on both datasets. In particular, our base model outperforms the Bevilacqua+ (2021, base) by 0 . 9 Smatch point on AMR2.0, and our large model obtains a Smatch of 85 . 4 and 84 . 2 on AMR2.0 and AMR3.0, respectively. To our knowledge, these are the best-reported results, showing the effectiveness of our method. Besides, Bevilacqua+ (2021, large) s uses silver data for fine-tuning, yet does not lead to consistent improvement over Bevilacqua+ (2021, large). In contrast, our large model gives 1 . 1 and 1 . 2 higher Smatch than Bevilacqua+ (2021, large) s on AMR2.0 and AMR3.0, respectively. This indicates that our pre-training framework is a better way than fine-tuning to make use of silver data. The main 6006 Model Smatch Unlab. reason is that our models are pre-trained using a denoising auto-encoding manner, which is less sensitive to silver (or noisy) data than fine-tuning. We also find that further fine-tuning our models on silver data (same with pre-training) cannot bring improvement ( cf. Appendix C.3). AMR-to-text generation . We report the results of different systems on AMR2.0 and AMR3.0 in Table 6, respectively. With the help of BART, Ribeiro+ (2021) and Bevilacqua+ (2021, large) obtain significantly better results than previous graph-to-sequence and GPT-based models. Compared with Bevilacqua+ (2021), our models ( base and large ) give significantly ( p <0.001) better results in terms of all evaluation metrics. In particular, our base model achieves comparable or better performance than Bevilacqua+ (2021, large). Compared with Bevilacqua+ (2021, large) s , our large model improves the performance by 3 . 9 and 2 . 7 points on AMR2.0 and AMR3.0, respectively. Similar with AMR parsing, we observe that when fine-tuning our models on silver data cannot bring improvement for AMR-to-text generation task (Table 6 and Appendix C.3). Zero-shot Domain Adaption. We use the model trained on AMR2.0 to get predictions on out-of-domain test sets. Table 7 shows the results on AMR parsing and AMR-to-text generation tasks. Similar to in-domain experiments, our models achieve better results than existing methods. In particular, our base model can give comparable performance than Bevilacqua+ (2021, large), and our large model obtains the best-reported results. This indicates that Model BLEU CH. our model is more robust to new domains, thanks to joint graph and text pre-training. Regarding different domains, our method achieves bigger improvements on New3 than the other two domains. This is intuitive, as pre-training strengthens the model representation power on the domain of graph pretraining data, and New3 is closer to it than other two datasets. New3 (both tasks) and TLP (only AMR-to-text generation). In contrast, our model gives consistent improvements on all 3 domains. This can be because fine-tuning leads to catastrophic forgetting of distributional knowledge (Kirkpatrick et al., 2017). Table 8 shows the effects of the graph size, graph diameter and reentrancies on the performance. We split the test set of AMR2.0 into different groups and report the performance improvement over the baseline model (Bevilacqua et al., 2021). All models are trained on AMR2.0. We first consider graph size, which records the number of nodes in an AMR graph. Our model consistently outperforms the baseline model on both tasks, with the performance gap growing on larger graphs. This indicates that our system is more powerful in dealing with larger graphs. The main reason is that our joint text and graph pre-training mechanism enhances the model with the ability to capture word or span level correlation between text and graph, which is helpful for dealing with long sequence and large graphs. The graph depth is defined as the longest distance between the AMR node and root node. A graph with deeper depth has more long-range dependencies. For AMR parsing, our model gives a better Smatch than the baseline model on the first two groups of graphs, and a comparable score on graphs with a depth bigger than 6 . For AMR-to-text generation, our model consistently improves over the baseline model on all graphs, and the improvements are bigger on deeper graphs. This shows that our model is better for learning more complex graphs. It can be that our graph masking strategies train the model to learn the relationships between a sub-graph and the remaining graph context, making it easier to understand deep graphs. Reentrancy is the number of nodes that has multiple parents. Reentrancies pose difficulties to both AMR parsing and AMR-to-text tasks (Damonte and Cohen, 2019; Szubert et al., 2020). The more reentrancies, the harder the graph is to be understood. Our method gives significantly ( p <0.01) better results on both tasks when the input graphs have less than 4 reentrancies. For graphs with more than 4 reentrancies, the proposed model is 0 . 4 better on AMR-to-text generation task and comparable than the baseline model on AMR parsing task. This means that our system has an overall better ability on learning reentrancies. Table 9 presents two cases of AMR parsing, with the model outputs generated by our model and the baseline model, and the gold output given the same input sentence. As shown in the first case, the baseline model omits the semantic unit  hard , thus generates an incomplete AMR graph of a different meaning compared with the input sentence. In contrast, our system preserves the concept  hard  and transfers the semantic relations correctly, thanks to the modeling of correspondence between text and graph during pre-training. In the second case, the baseline output includes a cyclic sub-graph ( i.e. , ( z1 harm-01 :ARG1 z1 ) ), which is contrary to the grammar that AMRs should be acyclic . Our system gives a valid AMR graph which is semantically similar with gold graph. Table 10 lists two AMR graphs and model outputs of our AMR-to-text model and the baseline model. In the first case, although the baseline generates a fluent sentence, it ignores the concept  have-purpose-91 , resulting in that the generated sentence is of a different meaning compared with the input graph. In the second AMR graph,  before  modifies the phrase  won many championships . However, in the baseline output,  before  is used to 6008 Text#1: It's getting hard to keep strong and keep carrying on with life. system recovers all concepts and maps the modification relationship from the AMR graph to text correctly. This indicates that our model generates more faithful sentences than the baseline. We investigated graph pre-training as a complement to text pre-training for AMR parsing and AMR-to-text generation tasks, using a novel unified framework with dual graph and text denoising. We find that graph pre-training is highly effective for both AMR parsing and AMR -to-text generation, and is a more effective way of making use of silver data compared with fine-tuning. Our methods give the best results on multiple benchmarks for both tasks. Yue Zhang is the corresponding author. We would like to thank anonymous reviewers for their insightful comments. This work is supported by the National Natural Science Foundation of China under grant No.61976180 and the Tencent AI Lab Rhino-Bird Focused Research Program.
abstain abstain abstain objective abstain method result objective abstain abstain abstain abstain abstain other other abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain method result result objective other other other other other other other other abstain method abstain other other method method other other other other other method method other objective method abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain method method method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain method abstain other abstain method method abstain abstain abstain abstain method abstain method other abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain other abstain other abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain result result result abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain result abstain result result result abstain abstain result abstain abstain abstain result result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective result result other other other