Abstractive summarisation is notoriously hard to evaluate since standard word-overlap-based metrics are biased towards specific words in the human reference. We introduce a new evaluation metric which abstracts away from the word-level and instead is based on fact-level content weighting, i.e. relating the facts of the document to the facts of the summary. We follow the assumption that a good summary will reflect all relevant facts, i.e. the ones present in the ground truth (human-generated reference summary). We confirm this hypothesis by showing that our weightings are highly correlated to human perception and compare favourably to the recent manual highlight-based metric of Hardy et al. (2019). Text summarisation compresses long textual documents into short summaries while retaining the most important information from the source. In contrast to extractive summarisation, which directly copies the most relevant fragments, abstractive summarization retains the most important facts and expresses them via paraphrasing, aggregating and even inferring new facts. Recent advances in neural decoders led to a number of single-document summarisation systems that exhibit some level of abstraction in their outputs, usually in the simplest form of paraphrasing (See et al. (2017); Narayan et al. (2018); Liu and Lapata (2019), inter alia ). Evaluating abstractive summarisation remains an open challenge (Schluter, 2017; Kryscinski et al., 2019): First, decoders are amenable to pathoge-niessuch as hallucination and/or omission of important information, which are hard to capture using existing evaluation metrics (Cao et al., 2018; Rohrbach et al., 2018; Dusek et al., 2020). Second, most datasets used for abstractive summarisation only contain a single reference summary, e.g. (Narayan et al., 2018; Volske et al., 2017), which most existing automatic metrics evaluate against, e.g. ROUGE using exact n-gram overlap (Lin, 2004), and thus tend to downvote paraphrases. We propose a new evaluation metric based on content weighting, where we abstract away from the particular surface form of the target summary, but represent it as facts using Semantic Role Labelling (SRL). In this way, we aim to better capture the semantic correctness of a summary, i.e. be more sensitive to hallucinations and omissions. 1 In particular, we weight the facts present in the source document according to the facts selected by a human-written summary. This alignment is conducted using contextual, rather than token-level, embeddings, e.g., BERT (Devlin et al., 2019). For evaluation, we measure whether an automatically generated summary is able to capture the same facts as the target. We also show that the computed weights correlate well with human perception. Our code is available at https://github. com/XinnuoXu/CorrFA_for_Summarizaion . The problem of reference bias has been addressed in several ways. First, metrics based on token-level or wider context embedding similarities which aim to better capture paraphrases but remain largely word-oriented, e.g. (Sun and Nenkova, 2019; Zhang et al., 2019; Zhao et al., 2019; Clark et al., 2019). Goodrich et al. (2019) come close to our approach by using entity and relation extraction, but their approach is limited to texts that lend themselves to be represented by RDF triples. An alternative is manual evaluation against the source document. This entails selecting content either using domain experts, e.g., the PYRAMID method (Nenkova and Passonneau, 2004), factoids 1 Note that we do not make any claims about fluency, which we assume is less of a problem for neural text generation. FACT1-tweet : [ARG0: the queen] has [V: tweeted] [ARG1: her thanks] [ARG2: to people who sent her 90th birthday messages on social media] FACT2-send : the queen has tweeted her thanks to [ARG0: people] [R-ARG0: who] [V: sent] [ARG1: her 90th birthday messages] [ARGM-LOC on social media] FACT1-tweet ARG0 V ARG1 ARG2 the queen had tweeted her thanks SRL Propositions Tree MR ARG0 V ARG1 ARGM-LOC people R-ARG0 who sent her 90th birthday messages on social media FACT2-send Figure 1: List of SRL propositions and corresponding tree MR with two facts for the sentence The queen has tweeted her thanks to people who sent her 90th birthday messages on social media . (Teufel and van Halteren, 2004), or via crowdsourcing (Shapira et al., 2019; Hardy et al., 2019). However, evaluation based on a small human-labelled test set is noisy, time consuming, and costly. Xe-nouleas et al. (2019) propose a referenceless metric, which only checks properties of the summary, not its relation to the original document. Sun and Nenkova (2019) compare average token and sentence ELMo embeddings against the document and claim good (system-level) correlations. Another option to avoid reference bias is question-based evaluation, either elicited manually (Clarke and Lapata, 2010; Narayan et al., 2018) or automatically (Scialom et al., 2019). However, it requires reference summaries as base for generating questions, thus only checking the summary contents indirectly. We represent facts in a sentence by adapting SRL (Palmer et al., 2005), which roughly captures who did what to whom in terms of predicates and their arguments. Given a list of parsed propositions for a sentence, 2 each predicate-argument structure is considered as one separate fact , where the predicate stands for the event and its arguments are mapped to actors, recipients, time, place, etc (see Fig. 1). Following a simple observation that arguments can function as separate predicates themselves, we construct a hierarchical tree structure for the whole sentence. We create the tree meaning representa-2 We use the SRL implementation of He et al. (2018) found in https://allennlp.org with 86.49 test F1 on the Ontonotes 5.0 dataset. tion (MR) from the list of facts by choosing the fact with the largest coverage as the root and recursively build sub-trees by replacing arguments with their corresponding sub-facts (ARG2 in FACT1 is replaced by FACT2 in Fig. 1). 3 3.2 Automatic Content Weighting We compute argument and fact weights by measuring the similarity of facts/arguments in the original document and the target summary based on their BERT word embeddings (for content words only) and their distance in the tree MR. We denote tokens of a document D and its summary S as t D = (cid:8) t D 1 , t D 2 ,    t Dn (cid:9) and t S = (cid:8) t S 1 , t S 2 ,    t Sm (cid:9) . To get their corresponding contextual embeddings e Dk and e Sk , we concatenate the two texts, 4 feed them into a pre-trained BERT model (Devlin et al., 2019) and take the contextualized embedding output from its last Transformer layer. Argument-based weighting: We first represent the summary and the document as two sequences of leaf arguments 5 (cid:8) AD 1 , AD 2 ,    ADN (cid:9) and (cid:8) AS 1 , AS 2 ,    ASM (cid:9) respectively, and weight the i -th leaf argument in the document as: w ai = avg j =1 ...M cosdist (cid:16) E Di , E Sj (cid:17) (1) i.e. the average embedding cosine distance to all arguments in the summary. Argument embeddings E Di and E Sj are average embeddings of content-word tokens belonging to the arguments: 6 E  i = avg k  A  i ,k (cid:54) stops e  k (2)   { D, S } , stops denotes a list of stopwords. Fact-based weighting: We can represent the summary and the document as two sequences of facts (cid:8) FD 1 , FD 2 ,    FDN (cid:48) (cid:9) and (cid:8) FS 1 , FS 2 ,    FSM (cid:48) (cid:9) , and weight the i -th fact in the document by its average distance to facts in the summary: w fi = avg j  1 ...M (cid:48) d fij (3) 3 We avoid using sentence-level MRs such as AMR (Ba-narescu et al., 2013), since current state-of-the-art performance of parsers is far behind compared to the simpler SRL task. 4 By concatenating, the information in each text can be embedded in each other through self-attention. This is useful since the summary sometimes contains additional and/or common-sense knowledge not captured in the document. 5 For example, in Fig. 1, ARG0, V, ARG1 in FACT1, and all the arguments in FACT2 are leaf arguments in the sentence, whereas ARG2 in FACT1 is not. 6 For example, in Fig. 1, her and thanks are two tokens directly attached to the argument ARG1 of FACT1. Thus, the embedding for ARG1 of FACT1 is the average embedding of these two tokens. The fact-level distance d fij is defined on top of argument weighting: d fij = avg A Dl  F Di ,A Sk  F Sj  il  jk (cid:98) cosdist (cid:16) E Dl , E Sk (cid:17) (cid:99) > (4) It is computed as the average cosine distance over embeddings of all leaf arguments in the subtrees of fact F Di in the document and fact F Sj in the summary, which is (1) filtered by a threshold  to discard argument pairs with weak semantic relation 7 and (2) weighted by MR tree distances of arguments to facts:  il = 1  treedist ( F i ,A l ) . 8 4 Content-weighting-based Metrics We now use these weights to introduce two metrics: Corr-F (fact-level) and Corr-A (argument-level). Let w fgold and w fcand denote the fact-level content weights calculated using the procedure from Section 3 based on human-reference and system-generated summaries, respectively. Similarly, w agold , and w acand denote the argument-level weights. Corr-F is then the Pearson Correlation Coefficient (PCC) between w fgold and w fcand . Corr-A is PCC between w agold and w acand . In other words, Corr-F and Corr-A indicate whether the generated summary focuses on the informative main points in the document (i.e. the same points as the reference summary), on two different levels of granularity. We validate our Corr-F and Corr-A metrics by collecting human judgements. In the following, we (1) collect content highlights from human judges using the Amazon Mechanical Turk platform 9 and calculate manual content weighting based on them, (2) calculate correlations of the manual content weights with our automatic content weights, (3) compare our metrics against existing reference-based ROUGE (Lin, 2004) and BERTScore (Zhang et al., 2019), as well as the referenceless manual HROUGE score (Hardy et al., 2019). 10 We use the extreme summarisation dataset (XSum; Narayan et al., 2018), which consists of 7 In this work, we set the threshold to 0.6. 8 E.g., in Fig. 1, treedist(FACT1, ARG1: her thanks) = 1, treedist(FACT1, ARG0: people) = 2, treedist(FACT2, ARG0: people) = 1. 9 Using the interface from https://github.com/ sheffieldnlp/highres . 10 Note that Corr-F/A are calculated with content weighting with respect to the reference. Therefore, strictly speaking, Corr-F/A are different to all existing metrics but still share some properties with them. We show the correlation between Corr-F/A and existing metrics in terms of relative system ranking, rather than a head-to-head metrics comparison. BBC articles and accompanying single-sentence summaries, i.e. sub-headlines of the original articles, professionally written by the authors of the articles. Due to the abstractive nature of the summaries, factoid content selection on phrase level is required beyond sentence-level extraction or token-level matching, making this dataset a popular test bed for abstractive summarisation. We use the outputs of three recent abstractive summarization systems as evaluation targets for our metrics: (i) the Pointer-Generator model (PTGEN ; See et al., 2017); (ii) the Topic-aware Convolutional Sequence-to-Sequence model (TCONV S2S; Narayan et al., 2018) and (iii) the abstractive summarization model using pretrained BERT encoders (BERTSUMABS ; Liu and Lapata, 2019). 11 5.1 Manual Annotation Collection Manual Content Highlighting: By extending the framework of Hardy et al. (2019), we collect manual content highlights on fact and argument levels, where we present human judges with the source document and the gold summary, with one fact/argument typeset in bold. The judges are required to select phrases or sentences in the document that support the bolded fact/argument (see Figure 4-9 in Appendix B). In both cases, judges are allowed to select parts of the text with any granularity. We limit the number of allowed continuous chunks and the maximum number of words to encourage highlights of fact/argument level. 12 We employ 3 judges per document in both cases. We use the same 50 articles and gold summaries sampled from the XSum test set as Hardy et al. (2019). Manual Content Weighting Calculation: Argument Level: Given a document D and a summary S , we define the weight of each token t Dk with respect to a summary argument A Sj as: w tkj = NumH (cid:0) t D k , AS j (cid:1) NumA (cid:0) A Sj (cid:1) (5) NumH ( t Dk , A Sj ) denotes the number of times token t k was selected and NumA ( A Sj ) is the total number of annotators who were shown A Sj bolded. We use token weights to compute manual argument-level weights w aman (parallel to Eq. 1): w aman,i = avg j =1 ...M avg t Dk  A Di w tkj (6) 11 For the first two, we use candidate summaries provided by the authors. For the third, we generated summaries by training a model with code and data offered by the authors. 12 We allow 4 chunks of max. 50 words total for fact-level and 5 chunks of max. 20 words for argument-level annotation. Fact Level: By adapting Eq. 5, we calculate a weight w tki for each token in document D w.r.t. bolded fact F Si in the summary S . The weight w fij between fact F Di in the document and F Sj in its summary is calculated using Eq. 6. We use Eq. 3 to get the manual fact content weighting w fman . Correlation: We evaluate how automatic content weighting w agold and w fgold correlates with manual content weighting w aman and w fman . Using the Pearson Correlation Coefficient directly over the content weights (PCC-W), we evaluate the correlation between content weights assigned by human judges and automatically calculated weights  PCC ( w  gold , w  man ) . As a more extreme form of weighting, we compute the correlation between content selected (i.e. ignoring computed weights) by human judges and the automatic mechanism (PCC-S); we set the value to 1 if the weight is over 0, meaning the fact/argument is selected. While content-weighting correlations are just moderate, content-selection correlations are strong, especially the fact-based (Table 1). In other words, the automatic method attends to facts human judges consider important, but weighs them differently. System-level Agreement: We check system-level agreement on Corr-F and Corr-A metrics when using automatic vs. manual content weighting (Table 2): We compute fact/argument-level content weights w  cand for each system (cf. Section 4). We then calculate Corr-F and Corr-A of w  cand against both w  man (manual weighting) and w  gold (automatic weighting) on the 50 articles with human annotation introduced in Section 5.1. The Corr-F metric shows the same system-level ordering for both manual and automatic content weighting. Furthermore, both manual and automatic content weighting agree that TCONV S2S and PTGEN achieve similar performance but are strongly outperformed by BERTSUMABS . Corr-F/A vs. referenceless metrics: HROUGE score (Hardy et al., 2019) is a content-weighting-based referenceless evaluation metric. Unlike our Model Corr-F Corr-A Manual content weighting  w  cand vs. w  man TCONV S2S 0.2274 0.2464 PTGEN 0.2180 0.2433 BERTSUMABS 0.2508 0.2662 Automatic content weighting  w  cand vs. w  gold TCONV S2S 0.6203 0.6280 PTGEN 0.5822 0.5727 BERTSUMABS 0.6714 0.6533 Table 2: System-level scores for manual and automatic content weighting on 50 human-annotated documents. approach, it operates on token level and is entirely based on manual annotation. The evaluation results in Table 3 show that Corr-F/A's ranking is identical to HROUGE's unigram and bigram precision, with Corr-F also assigning similar proportions. 13 Corr-F/A vs. reference-based metrics: ROUGE (Lin, 2004) and BERTScore (Zhang et al., 2019) are both reference-based metrics, which compute a similarity score for each token in the candidate sentence with each token in the reference sentence. However, instead of exact matches as used in ROUGE, BERTScore computes token similarity using contextual embeddings. Comparing to ROUGE and BERTScore on the full XSum test set (see Table 4) shows full agreement on system ordering for both metrics. We now provide examples demonstrating the strength and weaknesses of Corr-F/A by analysing system outputs where BERTScore and Corr-F/A demonstrate different ordering. Strengths: (1) Corr-F/A are more sensitive to content-level hallucination than BERTScore. Summaries with facts/arguments never mentioned in the original document get much lower Corr-F/A scores than summaries with content that appears in the document verbatim or as a paraphrase. Example 1 in Table 5 shows Corr-F/A penalizing the incorrect fact to become the next president generated by BERTSUMABS , while giving higher scores to TCONV S2S which paraphrased abdicate with 13 We computed HROUGE for BERTSUMABS using https://github.com/sheffieldnlp/highres . step down. (2) Corr-F/A better identify paraphrases , especially those containing extra content mentioned in the document but not in the ground-truth summary. Example 2 in Table 5 shows that Corr-F/A do not penalize BERTSUMABS for generating the argument after only eight games in charge, which is mentioned in the document. Weaknesses: (1) Corr-F is weaker in identifying token-level hallucination, 14 as in Example 3 in Table 5. Corr-F gives a higher score to TCONV S2S output with one hallucinated token robotic. However, Corr-A's more fine-grained approach works slightly better in this case. (2) Corr-F/A tend to under-score summaries containing content mentioned in the ground truth but only touched briefly in the document. In Example 4 in Table 5, Corr-F/A score the output of TCONV S2S lower, even though it correctly captures an academy for children with mental health, which is mentioned only once in the document. In sum, Corr-F/A is less dependent on the reference summary by also considering the source document, and thus has less of a reference bias than BERTScore. In addition, Corr-F/A helps to identify ungrounded facts, i.e. content-level hallucinations, which is important for identifying misinformation in automated news reporting. tem outputs with Corr-F/A calculated using a lower-performing SRL tool (He et al., 2017). 15 The results are shown as Corr-F/A(L) in Table 4 and show full agreement with Corr-F/A in terms of system ordering. However, the better performing original SRL system widens the margin between systems. We present an automatic evaluation framework for abstractive summarisation, which is low-cost and robust, as it does not rely on expert annotators nor is susceptible to crowdsourcing noise. Using fact representations, we are able to capture semantically similar, but at the same time distant in surface form, content in the summary that aligns with arbitrarily far-apart parts of the input document, casting our metric to be directly interpretable. Our metric is more sensitive to perturbations of the facts in the target summary, which resemble common hallucination phenomena of neural decoders (see Figure 2-3 in Appendix A for examples). In the future, we intend to investigate different meaning representation formalisms, such as AMR (Banarescu et al., 2013) and Dynamic Syntax (Kempson et al., 2001) and extend to other datasets (e.g. multiple-reference summarization) and tasks (e.g. response generation in dialogue). This research received funding from the EPSRC project MaDrIgAL (EP/N017536/1) and Charles University project PRIMUS/19/SCI/10. We would like to acknowledge the AWS Cloud Credits for Research programme. 15 F1 on the Ontonotes 5.0 dataset is 81.6% (  =  4 . 89 ).
abstain objective method result abstain abstain abstain abstain abstain objective objective abstain abstain method result other abstain other other method other method other other other other other other other abstain abstain method other abstain method method other abstain method abstain abstain abstain abstain abstain other method abstain abstain abstain abstain abstain method other method abstain other abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method method other abstain method abstain abstain abstain abstain abstain abstain abstain method abstain method method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method result abstain other other other