Despite recent progress in abstractive summarization, systems still suffer from faithfulness errors. While prior work has proposed models that improve faithfulness, it is unclear whether the improvement comes from an increased level of extractiveness of the model outputs as one naive way to improve faithfulness is to make summarization models more extractive. In this work, we present a framework for evaluating the effective faithfulness of summarization systems, by generating a faithfulness-abstractiveness trade-off curve that serves as a control at different operating points on the abstractiveness spectrum. We then show that the baseline system as well as recently proposed methods for improving faithfulness, fail to consistently improve over the control at the same level of abstractiveness. Finally, we learn a selector to identify the most faithful and abstractive summary for a given document, and show that this system can attain higher faithfulness scores in human evaluations while being more abstractive than the baseline system on two datasets. Moreover, we show that our system is able to achieve a better faithfulness-abstractiveness trade-off than the control at the same level of abstractiveness. Generating abstractive summaries of documents has been a long-standing goal of summarization. While there has been tremendous progress towards this goal (Kryscinski et al., 2018; Dong et al., 2019; Zhang et al., 2019; Lewis et al., 2020), abstractive summarization systems still suffer from faithfulness errors (Cao et al., 2018), generating information that is not present in the original text. This has led to an increased research in faithfulness evaluation of summarization systems (Falke et al., 2019; Kryscinski et al., 2020; Durmus et al., 2020) as  Equal contribution. Corresponding author for queries: faisal@cs.columbia.edu. well as methods to improve faithfulness of generated summaries (Kang and Hashimoto, 2020; Chen et al., 2021). Intuitively, one straightforward way of improving faithfulness of generated summaries is to copy a larger amount of content from the source article (i.e. more extraction). Thus, any methods that increase the level of extractiveness, whether intentionally or not, would improve faithfulness. Without reported extractiveness, it is unclear whether prior improvements mainly arise from increased extractiveness. We argue that in order to make progress in abstractive summarization, it is important to tease apart faithfulness improvements due to increased extractiveness versus improvements due to improved abstraction. In order to tease this apart, we develop a framework for evaluating progress in faithfulness, by considering the effective faithfulness , i.e. the improvement in faithfulness over a baseline system ( control ) operating at the same level of extractiveness. In particular, we split the training examples into different groups by the extractiveness of the summary, and train the control models on each group. Each of these models corresponds to a specific tradeoff between abstractiveness and faithfulness, forming a trade-off curve indicating how much faithfulness can be improved solely by increasing extractiveness. Systems that improve effective faithfulness should lie above this curve. Using this framework, we show that the improved faithfulness of recently proposed methods comes mainly from an increased extractiveness. We then conduct further analysis to explore whether it is possible to have a system that can be both more abstractive and more faithful than the baseline system. We train a selector on a small set of human-annotated data that, given a set of output summaries with varying levels of extractiveness, picks the most abstractive output that is faithful to the source. Our proposed system is both more abstractive and more faithful than the baseline. Moreover, we show that 1410 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 Extractiveness 0.4 0.5 0.6 0.7 0.8 0.9 M e t r i c S c o r e M1 M2 M3 M4 M5 M1 M2 M3 M4 M5 M1 M2 M3 M4 M5 Entailment FactCC DAE Figure 1: Extractiveness of generated outputs versus automated metric scores for Entailment, FactCC and DAE on the Gigaword dataset. our system is able to improve the effective faithfulness , achieving a better trade-off than the control at the same point on the abstractiveness spectrum. To summarize, our contributions are as follows: 1. We present a framework to evaluate the progress in improving effective faithfulness of models considering the control at the same level of extractiveness. 2. We illustrate the importance of considering effective faithfulness by showing that recently proposed methods for improving faithfulness are able to attain higher faithfulness scores than the baseline, but do not consistently improve over the control curve , indicating that most of their improvements come from generating more extractive outputs, on average. 3. We propose a selector that picks the most abstractive and faithful summary from a set of possible summaries, and show that this method gets higher effective faithfulness compared to the existing methods. We conduct our study on two English abstractive summarization datasets, one from the news domain, and one from a non-news domain. For the news do-main dataset, we decided against using the popular CNN/Dailymail dataset since its reference summaries tend to be very extractive (Kedzie et al., 2018; Bommasani and Cardie, 2020), making it a poor choice for studying faithfulness in abstractive summarization. Similarly, we also decided against using XSum, another popular news summarization dataset, since almost 77% of the gold reference summaries contain hallucinations (Maynez et al., 2020). Instead, we opted for Gigaword and Wikihow, which are datasets with substantial abstraction without as much hallucination problems as XSum. Gigaword reference summaries have substantially less hallucinations than XSum (Kang and Hashimoto, 2020), and WikiHow summaries tend to be of a higher quality since they are written and curated by humans (Koupaee and Wang, 2018; Ladhak et al., 2020). Wikihow (Koupaee and Wang, 2018) is a dataset of how-to articles covering a diverse set of topics, collected from the wikihow.com website. Each article contains several paragraphs detailing step by step instructions for a procedural task. There are about 12 M such paragraphs in the dataset, paired with a one sentence summary. extracted from news articles that were collected as part of the Gigaword corpus (Graff et al., 2003). The model is tasked with generating the headline of the article given the first sentence. We follow the process detailed by Grusky et al. (2018), and use extractive fragment coverage and extractive fragment density as the measures of extractiveness of a given summary. Henceforth we will refer to these as coverage and density respectively. Coverage is the percentage of words in a summary that are from the source article. Density is the average length of the text spans copied from the document that are contained in the summary. A summary that copies larger chunks of text from the source article will have a higher density. Recent studies of faithfulness evaluation have proposed model-based automated metrics to detect whether a given summary is faithful to the source article. For example, Falke et al. (2019) (Entail-ment) have studied using pretrained entailment based methods to assess the probability of the generated output being entailed by the source article. Kryscinski et al. (2020) (FactCC) augment hallucinated summaries by applying rule-based transformations to the document sentences and train a BERT-based model to classify whether the generated output is faithful. Goyal and Durrett (2021) (DAE) have collected fine-grained annotations to study word-, dependencyand sentence-level faithfulness and use these annotations to train a factuality detection model. Figure 1 shows the relationship between the average coverage of the generated outputs (extrac-tiveness) vs. average metric scores (faithfulness) assigned to various abstractive summarization models trained on Gigaword. 1 We observe that there is a positive correlation between extractiveness and faithfulness scores, as models whose generated summaries have a higher average coverage tend to also get higher scores for each of the faithfulness metrics. This correlation between exractiveness and faithfulness makes it unclear whether a model gets higher factuality scores simply because it is more extractive or it is capable of generating faithful summaries at the original level of extractiveness. This highlights the need for accounting for extractiveness in order to compare faithfulness across different abstractive summarization systems. Given that extractiveness is confounded with faithfulness, we propose a framework for evaluating effective faithfulness , which takes into account the extractiveness of a system. In order to do this, we first need to determine the faithfulness of a system operating at a given level of extractiveness. We call this the Faithfulness-Abstractiveness Tradeoff and we describe it further in 4.1. The effective faithfulness of a system is then simply the relative difference between the faithfulness score assigned to the system, and the score of a system operating with the same average extractiveness according to the trade-off curve. In order to understand the effectiveness of a proposed system for improving faithfulness, we need to be able to account for its extractiveness. We finetune pre-trained BART models (Lewis et al., 2020) for different levels of extractiveness, without any explicit recourse for improving faithfulness. We then use these systems to create a faithfulness-abstractiveness trade-off curve that can serve as a control to measure the effective faithfulness of summarization systems. Models that improve effective faithfulness should lie above the faithfulness-abstractiveness trade-off curve . 2 In particular, we sub-sample the training data into extractiveness quartiles by computing the coverage of the references with respect to the source articles. We then fine-tune BART on each of these quartiles to obtain quartile models with varying level of extractiveness. In addition, we also finetune BART on all of the data, which we call the baseline . We collect faithfulness annotations for summaries generated by each of these models for a random sample of 200 articles. We collect three annotations per example on Amazon Mechanical Turk asking whether an output is faithful or unfaithful with respect to the corresponding source article. We then compute the percentage of annotators that selects "faithful", and use this as the faithfulness 2 Human evaluation data and trade-off curves can be found at https://github.com/fladhak/effective-faithfulness. Article Once you decide what to outsource, look for the right contractors. Start by asking for referrals from your own professional network. Talk to other business owners and professionals about how and where they outsource. You can also check professional associations or trade groups field in which you are trying to outsource work. Use other social media platforms such as Facebook or Twitter to advertise what you are looking for. Alternately, you can connect with contractors and freelancers on sites such as eLance, Guru and oDesk. These websites allow business owners to place an ad that describes what kind of work they need to have done, and contractors respond with their qualifications and rates. [TRUNCATED] ... Baseline Search for contractors and freelancers to outsource the work. Q1 Conduct an initial search for qualified contractors and freelancers. Q2 Search for qualified contractors and freelancers to work on your project. Q3 Search for contractors and freelancers to do the work. Table 2 shows the coverage and faithfulness scores for the baseline and the quartile models, where Q1 is the most abstractive and Q4 is the most extractive quartile. 4 We observe that the models that are fine-tuned on more extractive quartiles produce outputs with significantly higher coverage and faithfulness scores. The baseline model generates relatively extractive outputs with coverage closest to Q3 on both Gigaword and Wikihow. Furthermore, we observe that the baseline model has a higher coverage than the model fine-tuned on Q3 but it has lower faithfulness score for Gigaword. Table 1 shows an article from the Wikihow dataset and corresponding output summaries generated by the baseline and each of the quartile models. We observe that the generated summaries are very similar in meaning; however, the output generated by the Q1 model includes a higher number of novel words (i.e. lower coverage) compared to the other models while staying faithful to the article. Conversely, Q4 model has a coverage of 1 in this example; all the words generated by this model are from the source article. On average, the Q1 model generates outputs that are more abstractive and less faithful while Q4 generates outputs that are more extractive and more faithful. We first aim to understand whether it is possible to mitigate the faithfulness-abstractiveness tradeoff by designing several oracle experiments where we have access to human judgments. baseline + faithfulness (bf). We use the output from the baseline model if it is faithful (i.e. at least two out of three annotators agree that the output is faithful). If the baseline output is not faithful, we select the output from the quartile model that is more extractive than the baseline to see whether we can have a similar coverage as the baseline but preserve faithfulness. baseline + faithfulness-extractiveness (bfe). This oracle system behaves similar to the one described above when the baseline output is unfaithful. However, rather than always selecting the base-1413 Dataset Cov. line output when it is faithful, we pick the output from the quartile model that is more abstractive than the baseline whenever it is also faithful according to human judgement. quartile + faithfulness-extractiveness (qfe). Amongst the outputs of all four quartile models, we pick the most faithful output with the highest level of abstractiveness to understand whether it is possible to generate abstractive output while remaining faithful. Analysis. Table 3 shows the coverage and faithfulness of the baseline and each of these oracles for Gigaword and Wikihow. We observe that it is possible to be more faithful than the baseline at a similar level of abstractiveness (bf). Furthermore, we can be more abstractive than the baseline while being more faithful (bfe). Selecting the most faithful and abstractive output from the quartile models achieves a really high faithfulness score (  98%) while having significantly less coverage than the baseline. This oracle analysis suggests that it should be possible to build models that can mitigate the faithfulness-abstractiveness trade-off by controlling the level of extractiveness. Given this, we further explore whether we can learn a selector that is capable of doing this selection automatically to mitigate the faithfulness-abstractiveness tradeoff. Kang and Hashimoto (2020) have proposed a method to adaptively remove high loss examples to optimize the distinguishability of samples from the model and the reference. They have shown that the samples generated by this Loss Truncation model achieves higher factuality ratings compared to the baseline methods. We study this method to understand where it lies in terms of faithfulness-abstractiveness trade-off and whether it can achieve a improved effective faithfulness over the control . Goyal and Durrett (2020) have proposed a factuality evaluation metric (DAE) that evaluates whether each dependency arc in the generated output is consistent with the input. They show that their proposed metric works better than existing factuality metrics, while also being able to localize the parts of the generated output that are non-factual. Goyal and Durrett (2021) take advantage of DAE's ability to localize factuality errors, and train a summarization model only on the subset of tokens that is deemed factual according to the DAE metric. We follow their methodology to train summarization models, and assess them using our evaluation framework. We aim to understand whether we can build a model that achieves a better effective faithfulness than Loss Truncation. We propose a selector that can identify the most abstractive but faithful output to improve this trade-off. We first generate four possible candidate summaries using the quartile models for each example in the validation set. This results in outputs with varying levels of extractiveness. For our selector, we fine-tune a FactCC model (Kryscinski et al., 2020) on the data we collected to generate the trade-off curve, using 10-fold cross validation, to assign faithfulness scores to the generated summaries (in the test folds). 5 In addition, we learn a threshold for the faithfulness score that maximizes the area under the ROC curve (Selector-ROC) (also using 10-fold cross validation). For each example in the test fold, we select the most abstractive candidate (amongst the four possible candidates from the quartile models) that is considered faithful according to the fintuned FactCC model (i.e. the faithfulness score is above the tuned threshold). Instead of maximizing for the area under the ROC curve, we can also tune the faithfulness threshold to maximize F  scores ( Selector-F  ). Using F  score with  < 1 allows us to assign a higher weight to the precision of our selector which would result in outputs with higher coverage and faithfulness. 5 We collected annotations for 200 articles for each of the quartile models. We find that the fine-tuning FactCC is important since the pre-trained FactCC model is trained on a different dataset and does not transfer well to our setttings. This is consistent with the findings of Goyal and Durrett (2021). Table 4 shows the coverage and faithfulness results for the baseline, Loss Truncation, DAE, and the selectors. We observe that as we use smaller values for  for Selector-F  , we get more extractive and more faithful outputs. This allows us to have a trade-off between faithfulness and abstractiveness. Moreover, with both Selector-ROC and Selector-F  , we produce output with less coverage but higher faithfulness scores than the baseline. For Wikihow, Selector-ROC produces outputs with lower coverage but similar faithfulness scores to Loss Truncation. We can further obtain a higher faithfulness score at a similar coverage level as DAE and Loss truncation with Selector-F  with  = 0 . 1 . For Gigaword, Select-ROC produces output with significantly lower coverage than Loss Truncation and DAE. Selector-F  produces output with similar coverage to Loss Truncation with a higher faithfulness score (  = 0 . 1 ). It is important to understand whether models improve faithfulness by simply being more extractive or if they are able to improve effective faithfulness . In order to understand this, we measure whether the models get improvement in faithfulness over the control operating at the same level of extractiveness. In Figure 2, we plot the faithfulness-abstractiveness curve with the faithfulness and abstractiveness of the quartile models. If a model lies above this curve, it improves the effective faithfulness . If the model is below this curve, it is not able to improve the effective faithfulness and it has a worse tradeoff than the control operating at the same level of extractiveness. For both Gigaword and Wikihow, Selector-ROC lies above the curve improving this trade-off. However, both the baseline and Loss Truncation models get worse trade-off than the control operating at the same level of extractiveness. Similarly, we can obtain several models that lie above the curve for both Gigaword and Wikihow using Selector-F  . The selector approach allows us to get better effective faithfulness at different points in the abstractiveness-extractiveness spectrum. The DAE based model is able to improve effective faithfulness on the Wikihow dataset, but not on the Gigaword dataset, indicating that the improvements are not consistent across datasets. Table 5 shows example summaries generated by the baseline, Loss Truncation, DAE and the Selector-ROC models. We observe that selector model is able to generate summaries that are faithful to the original article while having more novel words and phrases in the generated summaries. There has been a lot of recent work in abstractive summarization showing that state-of-the-art systems suffer from generating inconsistent information with respect to the source article, despite their improved success in producing fluent summaries (a) Selector-ROC and the baseline trade-off on Gigaword . (b) Selector-F  and the baseline trade-off on Gigaword . (Falke et al., 2019; Lux et al., 2020; Wilber et al., 2021). Since word-overlap based metrics such as ROUGE have low correlation with human scores of faithfulness (Kryscinski et al., 2019; Fabbri et al., 2020), there has been significant effort to develop automated metrics that can detect such errors (Zhou et al., 2021; Gabriel et al., 2021; Pagnoni et al., 2021a). For example, Falke et al. (2019), Maynez et al. (2020) and Goyal and Durrett (2020) have proposed to assess faithfulness using entailment models, where a faithful summary should be assigned a high entailment score with respect to the original article. Kryscinski et al. (2020) presented FactCC, a weakly-supervised BERT-based entailment model, by augmenting the dataset with artificial faithfulness errors. Durmus et al. (2020) and Wang et al. (2020) proposed question-answering based evaluation frameworks by automatically generating questions from the generated summary, and comparing the corresponding answers from both the source and the generated summary in order assess information consistency. Furthermore, several benchmarks have been proposed to evaluate the strengths and weaknesses of these evaluation metric (Gabriel et al., 2021; Pagnoni et al., 2021b). Previous studies in faithfulness evaluation, however, has not accounted for the effect of extractiveness of the output summaries. As we show in this study, the extractiveness of the output is correlated with the faithfulness scores assigned by these automated metrics. Therefore, it is not clear whether the models with higher scores are better at abstraction, or extract more from the source article. We suggest that we need to account for this confounding factor in order to assess the real progress in building models that are better at abstraction. We note that there is concurrent work that also argues for accounting for extractiveness in assessing the faithfulness of models (Dreyer et al., 2021), however, unlike our work, they do they do not propose any mitigation for the faithfulness-abstractiveness trade-off. world scenarios, as such recent work has studied methods to improve the faithfulness of abstractive summarization systems (Matsumaru et al., 2020; Zhao et al., 2020; Dong et al., 2020; Goyal and Durrett, 2021; Xu et al., 2020; Chen et al., 2021; Zhu et al., 2021). For example, Goyal and Durrett (2021) train summarization systems by modifying the training objective to maximize the likelihood of the subset of summary tokens that are considered faithful according to their factuality detection model. Zhao et al. (2020) specifically target hallucination of quantities in generated summaries, and train a verification model that they use to re-rank summaries such that summaries containing quantities consistent with the source article are up-ranked. Although these methods have shown improvements over the compared baselines, unlike our work, they do not measure the effective faithfulness taking extractiveness of the generated outputs into account. Recent studies that propose methods to improve faithfulness evaluate progress by conducting human evaluation on generated summaries and check whether the faithfulness scores are higher for their proposed system as compared to their baselines. We show that there is a strong relationship between the extractiveness and faithfulness of generated outputs (i.e., more extractive outputs tend to be more faithful), and therefore we cannot simply disregard extractiveness in faithfulness evaluation. We propose that we should instead be measuring effective faithfulness and introduce a framework that takes into account the faithfulness-abstractiveness trade-off curve that is generated by training control models at different points in the abstractiveness spectrum. We demonstrate the importance of measuring effective faithfulness by showing that recently proposed methods that improve faithfulness over the baseline fails to consistently improve over a simple control operating at the same level of abstractiveness. We argue that measuring effective faithfulness is important since our goal is to build abstractive, faithful summarization systems. If the objective was to optimize for faithfulness alone, we could do so by simply building more extractive systems (such as the Q4 model we trained above). Limitations. Note that this method relies on some diversity in the extractiveness of reference summaries, since we rely on sub-sampling to train models for the control . It is less likely to be effective for datasets with very little variation in the extractiveness of the generated summaries. However, in general, we see significantly more faithfulness problems for datasets with higher diversity of abstractiveness. Therefore, we suggest to account for the faithfulness-abstractiveness trade-off for such datasets in future work. This work was partly supported by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via contract #FA8650-17-C-9117, Sam-sung Advanced Institute of Technology (Next Generation Deep Learning: From Pattern Recognition to AI), and a collaborative grant from Amazon to the Columbia Center for Artificial Intelligence entitled Extremely Abstractive Summarization. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the funding agencies. We further thank the anonymous reviewers and the Stanford NLP group for their helpful feedback.
abstain abstain method objective result result abstain abstain abstain abstain abstain abstain abstain abstain abstain objective method abstain abstain objective objective method objective result result objective result objective objective method abstain method method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain method method method other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method method method abstain abstain method abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain other other other other other other other other other other abstain other abstain abstain other other other abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain other other other