Systems that generate sentences from (abstract ) meaning representations (AMRs) are typically evaluated using automatic surface matching metrics that compare the generated texts to the texts that were originally given to human annotators to construct AMR meaning representations . However, besides well-known issues from which such metrics suffer (Callison-Burch et al., 2006; Novikova et al., 2017), we show that an additional problem arises when applied for AMR-to-text evaluation because mapping from the more abstract domain of AMR to the more concrete domain of sentences allows for manifold sentence realizations. In this work we aim to alleviate these issues and propose MF_\beta, an automatic metric that builds on two pillars. The first pillar is the principle of meaning preservation \mathcal{M}: it measures to what extent the original AMR graph can be reconstructed from the generated sentence . We implement this principle by i) automatically constructing an AMR from the generated sentence using state-of-the-art AMR parsers and ii) apply fine-grained principled AMR metrics to measure the distance between the original and the reconstructed AMR. The second pillar builds on a principle of (grammatical) form \mathcal{F} , which measures the linguistic quality of the generated sentences , which we implement using SOTA language models. We show - theoretically and experimentally - that fulfillment of both principles offers several benefits for evaluation of AMR-to-text systems, including the explainability of scores \mathcal{F}_\beta does not necessarily rely on gold AMRs, it may extend to other text generation tasks} .
Systems that generate natural language text from abstract meaning representations such as AMR are typically evaluated using automatic surface matching metrics that compare the generated texts to reference texts from which the input meaning representations were constructed. We show that besides well-known issues from which such metrics suffer , an additional problem arises when applying these metrics for AMR-to-text evaluation , since an abstract meaning representation allows for numerous surface realizations. In this work we aim to alleviate these issues by proposing MF_\beta, a decomposable metric that builds on two pillars. The first is the principle of meaning preservation \mathcal{M}: it measures to what extent a given AMR can be reconstructed from the generated sentence using SOTA AMR parsers and applying ( fine-grained ) AMR evaluation metrics to measure the distance between the original and the reconstructed AMR. The second pillar builds on a principle of (grammatical) form \mathcal{F} that measures the linguistic quality of the generated text , which we implement using SOTA language models. In two extensive pilot studies we show that fulfillment of both principles offers benefits for AMR-to-text evaluation, including explainability of scores . Since \mathcal{M\mathcal{F}_\beta does not necessarily rely on gold AMRs, it may extend to other text generation tasks} .
R
sentences from (abstract ) meaning representations (AMRs)
natural language text from abstract meaning representations such as AMR
R
the texts that were originally given to human annotators to construct AMR meaning representations . However,
reference texts from which the input meaning representations were constructed. We show that
R
(Callison-Burch et al., 2006; Novikova et al., 2017), we show that
,
R
applied
applying these metrics
R
because mapping from the more abstract domain of AMR to the more concrete domain of sentences allows for manifold sentence
, since an abstract meaning representation allows for numerous surface
R
and propose
by proposing
R
an automatic
a decomposable
D
pillar
R
the original AMR graph
a given AMR
R
. We implement this principle by i) automatically constructing an AMR from the generated sentence using state-of-the-art
using SOTA
R
ii) apply
applying (
R
principled AMR
) AMR evaluation
R
, which
that
R
sentences
text
R
We show - theoretically and experimentally -
In two extensive pilot studies we show
R
several benefits for evaluation of
benefits for
R
systems, including the
evaluation, including
A
. Since \mathcal{M