We propose neural models to generate high-quality text from structured representations based on Minimal Recursion Semantics (MRS). MRS is a rich semantic representation that encodes more precise semantic detail than other representations such as Abstract Meaning Representation (AMR). We show that a sequence-to-sequence model that maps a linearization of Dependency MRS, a graph-based representation of MRS, to English text can achieve a BLEU score of 66.11 when trained on gold data. The performance can be improved further using a high-precision, broad coverage grammar-based parser to generate a large silver training corpus, achieving a final BLEU score of 77.17 on the full test set, and 83.37 on the subset of test data most closely matching the silver data domain. Our results suggest that MRS-based representations are a good choice for applications that need both structured semantics and the ability to produce natural language text as output. Text generation systems often generate their output from an intermediate semantic representation (Yao et al., 2012; Takase et al., 2016). However many semantic representations are taskor domain-specific (He and Young, 2003; Wong and Mooney, 2007), while rule-based text generation systems often have incomplete coverage (Langkilde-Geary, 2002; Oepen et al., 2007). In this work we combine the advantages of Minimal Recursion Semantics (MRS; Copestake et al., 2005) with the robustness and fluency of neural sequence-to-sequence models trained on large datasets. We hypothesize that MRS is particularly well-suited for text generation, as it is explicitly compositional, capturing the contribution to sentence meaning of all parts of the surface form (Bender et al., 2015). In contrast, semantic representations such as Abstract Meaning Representation (AMR; Ba-narescu et al., 2013) seek to abstract away from the syntax of a sentence as much as possible. Therefore MRS captures meaning distinctions that AMR fails to represent (see Fig. 1). Our approach (2) uses neural sequence-to-sequence models (Sutskever et al., 2014; Bah-danau et al., 2014) to map linearizations of directed acyclic graphs (DAGs) to text, similar to the approach proposed by Konstas et al. (2017) to generate text from AMR. We use Dependency MRS (DMRS; Copestake, 2009), a graph-based representation in which nodes are MRS predicates (annotated with additional attributes) and edges represent relations between predicates. MRS and DMRS are interconvertible and the graph-based representation enables more convenient linearization and manipulation than MRS's variable-based representation (Copestake et al., 2016). Results (3) show that neural DMRS to English text generation can obtain up to 83.37 BLEU and 32% exact match, substantially higher than previous work. In particular, we obtain an 11.6 BLEU improvement through semi-supervised training using the output of a grammar-based parser, compared to training on gold data only. In comparison a grammar-based generator obtained 62.05 BLEU, and an approach based on DAG Transducers (Ye et al., 2018) 68.07 BLEU. Ablation experiments show that node attributes encoding fine-grained morpho-semantic information such as number and tense contribute more than 12 BLEU points. The highest reported result for AMR generation is 33.8 BLEU (Konstas et al., 2017); on the same dataset our best model obtains 75.8 BLEU. While a more detailed meaning representation is harder to produce, our results suggest that MRS could be suitable for text generation applications where precise semantic representations are required. Our gold training data are parallel MRS and English text corpora, derived from the 1214 release of the Redwoods Treebank (Oepen et al., 2002). 1 MRS is implemented as the semantic layer of the English Resource Grammar (ERG; Flickinger, 2000, 2011), a broad-coverage, hand-engineered computational grammar of English. The Redwoods annotation was produced in conjunction with the ERG by parsing each sentence into a for-est (discarding unparsable sentences), followed by manual disambiguation (Flickinger et al., 2017). About half of the training data comes from the Wall Street Journal (sections 00-21), while the rest spans a range of domains, including Wikipedia, e-commerce dialogues, tourism brochures, and the Brown corpus. The data is split into training, development and test sets with 72,190, 5,288, and 10,201 sentences, respectively. We use PyDelphin 2 to convert MRS annotations to DMRS. In order to apply sequence-to-sequence models to graph-to-text generation, we then linearize the DMRS into PENMAN format (which is also used to represent AMR). We follow Goodman (2018, pp. 8286) in finding normalized spanning 1 http://svn.delph-in.net/erg/tags/ 1214/tsdb/gold 2 https://github.com/delph-in/pydelphin PENMAN (10002 / _see_v_1 :tense PRES :sf PROP :perf -:mood INDICATIVE :ARG1-NEQ (10001 / named :carg "Kim" :pers 3 :num SG :ind +) :ARG2-NEQ (10004 / _boy_n_1 :pers 3 :num SG :ind + :RSTR-H-of (10003 / _a_q))) Linearization ( _see_v_1 mood=INDICATIVE|perf=-|sf (cid:44)  =PROP|tense=PRES ARG1-NEQ ( (cid:44)  named0 ind=+|num=SG|pers=3 ) (cid:44)  ARG2-NEQ ( _boy_n_1 ind=+|num= (cid:44)  SG|pers=3 RSTR-H-of ( _a_q ) ) (cid:44)  ) Figure 2: The DMRS for the sentence Kim sees a boy. trees through depth-first traversal over the directed acyclic DMRS graphs. 3 The PENMAN format de-fines each node once, supports node attributes and edge labels, marks edges whose direction is reversed in the traversal, and represents edges which are not covered by the spanning tree. The PENMAN format is processed further to obtain a linearization appropriate as input to sequence-to-sequence models, similar to the approach proposed by Konstas et al. (2017) for AMR linearization (see Fig. 2). Node variable identifiers are removed, node attributes are concatenated, and named entities are anonymized. Predicates that appear only once in the training data are treated as unknowns. Preprocessing and unknown word handling are described in greater detail in Appendices A and B. 2.3 Model Our neural generator follows the standard encoder-decoder paradigm (Bahdanau et al., 2014). The encoder is a two-layer bidirectional LSTM. Predicates and their attributes are embedded separately; their embeddings are then concatenated (Sennrich and Haddow, 2016). The 3 https://github.com/goodmami/ mrs-to-penman decoder uses global soft attention for alignment (Luong et al., 2015), and pointer attention to copy unknown tokens directly to the output (Gulcehre et al., 2016). The models are trained using Adam (Kingma and Ba, 2014). Dropout is applied to non-recurrent connections. Decoding uses beam search (width 5). The generator is implemented using OpenNMT-py (Klein et al., 2017). Hyper-parameter details are given in Appendix C. Our code is available online. 4 2.4 Semi-supervised training We augment the gold training data with a silver dataset generated using ACE, 5 a parser for the ERG, to parse sentences to MRS. We sample one million sentences from the Gigaword corpus (Parker et al., 2011), restricted to articles published before the year 2000, to match the domain of the Wall Street Journal data. The parser failed to parse about 10.3% of the Gigaword sentences, so these were discarded. While there are robust MRS parsers (Buys and Blunsom, 2017; Chen et al., 2018), the MRSs they produce are less accurate and not guaranteed to be well-formed. Our approach thus differs from Konstas et al. (2017), who used self-training to improve AMR to text generation by iteratively training on larger amounts of data parsed by their neural parser. 6 3 Results We compare the performance of our neural generator when trained on either gold, silver, or gold and silver data (Table 1). Generation quality is primarily evaluated with BLEU (Papineni et al., 2002), using SacreBLEU (Post, 2018). 7 We evaluate the neural models on both the full Redwoods test set (All') and the WSJ subset. The results show that our neural generator obtains very strong performance. Semi-supervised training leveraging the ERG parser leads to an 11 BLEU point improvement on Redwoods, comparing to supervised training only. We found that the best semi-supervised results are obtained by up-4 https://github.com/shlurbee/ dmrs-text-generation-naacl2019 5 ACE version 0.9.25, with the 1214 ERG release, available at http://sweaglesw.org/linguistics/ ace 6 The ACE parser obtained 93.5 Smatch score on parsable sentences (Buys and Blunsom, 2017), while the neural AMR parser (Konstas et al., 2017) obtained 62.1 Smatch (on a different domain). sampling the gold data so that the gold to silver ratio in training examples is 1:2. Interestingly, training on silver data performs only slightly worse than training on both gold and silver. Our baselines are the ERG's grammar-based generator (Carroll et al., 1999; Carroll and Oepen, 2005) and the DAG transducer generator of Ye et al. (2018). To compare our models against the grammar-based generator, implemented in ACE, we need to restrict the evaluation to examples from which ACE is able to generate (All overlap'). 8 In addition to BLEU, we also report exact match accuracy on the overlapping subset. Results show that our neural models outperform the grammar-based generator by a large margin. ACE ranks candidate generations with a discriminative ranker based on structural features over its derivations (Velldal and Oepen, 2006). However, it does not use a language model trained on large amounts of text, which would likely improve fluency substantially. The DAG transducer was trained to generate from Elementary Dependency Structures (EDS; Oepen and Lnning, 2006), an MRS-derived representation almost equivalent to DMRS (after edge properties are removed, which Table 3 shows has an effect of less than 1 BLEU point). It was evaluated against the same WSJ test set reference generations, but trained using both less gold data (only the WSJ subsection) and less silver data (300K vs 900K sentences). Our model trained on WSJ gold data performs only slightly worse (65.78 BLEU; see Table 2) and all our semi-supervised models obtain substantially higher results. We evaluate the inand out-of-domain performance of our approach by training models on either WSJ gold data only, or both WSJ gold data and Gigaword silver data, and evaluating on different domains. The results in Table 2 show that while the generator performs best on test data which matches the training domain (news), semi-supervised training leads to substantial out-of-domain improvements on the Wikipedia and the Brown corpus portions of the test set. 8 Despite all test sentences being parsable by the ERG, there are gaps in generation coverage, primarily because ACE is unable to generate words outside the grammar's vocabulary. To understand which elements of MRS contribute most to our generator's performance, we ablate node (predicate) and edge attributes from both the training and test DMRS graphs (Table 3). In the training data, number and tense show the most variation among node attributes, and subsequently have the largest effect on the reported BLEU score. The most common value for number is SG , but 62.36% of sentences contain a node with PL . Similarly, 42.41% of sentences contain a tense value other than PRES or UNTENSED . Many other attributes are less informative: Mood has a value other than INDICATIVE in only 0.38% of sentences, and perf is + in just 9.74% of sentences. Edge features (including H, EQ and NEQ) encode constraints on scopal relationships (see Copestake 2009). Removing them, which makes the DMRS representation close to equivalent to EDS, has only a small impact on performance. We compare our approach to AMR-to-text generation by evaluating our generator on a standard AMR test set (LDC2015E86). As we do not have manually verified MRSes available on this test set, we use ACE to parse the reference sentences to silver MRSes. We then evaluate the outputs that our generator produces from those MRSes. About 20% of the examples could not be parsed by ACE, and are discarded for the MRS evaluation. We compare our generator to the neural AMR generator of Konstas et al. (2017) for models trained on gold as well as gold plus silver data. 9 We evaluate DMRS models both with and without predicate and edge attributes, as these attributes contain information that is absent from AMR. 10 The results in Table 4 show that our MRS generator performs better than the AMR generator by a large margin, even when the additional MRS attributes are excluded. Our system results are reported on the subset for which we obtained MRS parses. AMR results are as given by Konstas et al. (2017) and cover the entire test set. 9 The AMR and DMRS systems have different gold training data, but the same source of silver data. 10 Recently, Donatelli et al. (2018) proposed adding tense and aspect to AMR, but this annotation is not yet available in a large AMR corpus. We sampled 99 items for error analysis from the dev set, 33 each from among sentences with sentence-level BLEU scores of 80-89, 60-69, and 40-49. 11 We identified all differences between these strings and the reference strings and then labeled each difference with a fine-grained error type. 12 We classified the differences into 238 errors, distributed across five levels of severity (Ta-ble 5). Almost half of the differences (47.1%) were unproblematic, including spelling variants, meaning-preserving punctuation variation and grammatical alternations (such as optional that or auxiliary contraction as in (1)). The slightly problematic category includes close synonyms (e.g. sometime v. someday ), spelled out number names where Arabic numerals are preferred, and differences in formatting. The next more serious category (mod-erately problematic) includes meaning-changing differences in punctuation, tense or aspect, and mi-nor grammatical errors such as swapping who and which in relative clauses or a v. an . Finally, among the most serious errors, we find cases where the generator provided ungrammatical output or grammatical output not conveying the correct semantics. The former include spurious additional tokens, ungrammatical word orders, and ungrammatical inflection. Serious errors that nonetheless resulted in grammatical strings include meaning-changing dropped or swapped 11 Items with BLEU scores lower than 40 tend to be very short and primarily involve formatting differences. 12 This was done by a single annotator only. The labels were generated bottom up, with new labels added as needed in the course of annotation. (2) a. For such cases, machine learning techniques emulate human linguistics and learn from training examples to predict future events. [sys.] b. For such cases, machine learning techniques emulate human cognition and learn from training examples to predict future events. [ref.] In summary, we find that the BLEU scores underestimate the quality of system outputs, due to unproblematic differences (N=112) and differences, like formatting markup (N=6), not reflected in the input semantic representations. Among the 108 moderate to serious differences, about a third (35) involve punctuation, suggesting that meaning signalled by punctuation could be better reflected in the semantic representations. About half (52) involve added, dropped, or swapped tokens, showing room for improvement in the generator's ability to learn appropriate connections between semantic predicates and surface forms. The remainder (21) involve inflection, grammatical alternations (such as who / which ) and word order constraints, showing room for improvement in mimicking grammatical processes. We have shown that neural sequence-to-sequence models can be used to generate high quality natural language text from Minimal Recursion Semantics representations, in contrast to both existing MRS-based generators and neural generators based on other broad-coverage semantic representations. Furthermore, we have demonstrated that a large hand-crafted grammar can be leveraged to produce large training sets, which improves performance of neural generators substantially. Therefore we argue that the ability to generate high quality text from MRS makes it a good choice of representation for text generation applications that require semantic structure. For future work, we are interested in applying graph-to-sequence neural networks (Beck et al., 2018; Song et al., 2018) to MRS-to-text generation. Thanks to Yannis Konstas for sharing preliminary results on DMRS generation, and Swabha Swayamdipta for discussions. This research was supported in part by NSF (IIS-1524371) and Sam-sung AI Research.
objective abstain result abstain result abstain abstain method abstain abstain abstain abstain method abstain abstain result abstain abstain result result method abstain abstain abstain abstain method method other abstain abstain abstain abstain abstain method abstain abstain other abstain abstain abstain abstain method method abstain abstain method method abstain method method abstain other abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain method method abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective method method other other