Joshua Maynez Google Research joshuahm@google.com dipanjand@google.com Michael Collins Google Research mjcollins@google.com lapata@google.com Abstract We propose Composition Sampling, a simple but effective method to generate diverse outputs for conditional generation of higher quality compared to previous stochastic decoding strategies. It builds on recently proposed plan-based neural generation models (Narayan et al., 2021) that are trained to first create a composition of the output and then generate by conditioning on it and the input. Our approach avoids text degeneration by first sampling a composition in the form of an entity chain and then using beam search to generate the best possible text grounded to this entity chain. Experiments on summarization (CNN/DailyMail and XSum) and question generation (SQuAD), using existing and newly proposed automatic metrics together with human-based evaluation, demonstrate that Composition Sampling is currently the best available decoding strategy for generating diverse meaningful outputs. In many NLG tasks, it is important to be able to generate multiple diverse outputs from a model. Tasks like summarization (Mani, 2001; Nenkova and McKeown, 2011) and question generation (Zhou et al., 2017) exhibit one-to-many relationships; there can be multiple semantically diverse summaries or questions for the same source, and it may be useful for a model to be able to generate multiple outputs. Yet, the primary focus of recent research in NLG has been on improving the quality of single-best outputs (Raffel et al., 2019; Lewis et al., 2019; Dong et al., 2019; Zhang et al., 2020a; Narayan et al., 2021), while diversity remains an unsolved problem (Hashimoto et al., 2019; Zhang et al., 2021). This is particularly challenging in conditional generation, where diversity in the target sequence should not come at the cost of correctness or faithfulness; for example, alternate summaries are not valuable if they are unfaithful to the input document(s) (Maynez et al., 2020; Kryscinski et al., 2020). In this work, we investigate decoding methods for generating semantically diverse text which is also faithful to its input focusing on two tasks, namely summarization and question generation. Beam search (Li et al., 2016; Wiseman et al., 2017) has proven successful for single-best generation (Rush et al., 2015; Barrault et al., 2020; Meister et al., 2020), but struggles to generate diverse output (Vijayakumar et al., 2016). Stochastic sampling strategies, such as topk sampling (Fan et al., 2018) and nucleus sampling (Holtzman et al., 2020), are better at generating diverse sequences but are not suitable for conditional generation as they degenerate, 1 producing output that is not faithful to the source. Figure 1 exposes degeneration in summary output using nucleus sampling. To address these shortcomings, we propose Composition Sampling , a simple but effective hybrid decoding method for diverse and faithful conditional generation. It builds on recently proposed generation models (Narayan et al., 2021) that are trained to first plan a semantic composition of the target and then generate the text conditioned on the composition and the input. Composition sampling first samples a composition in the form of an entity chain and then uses beam search to generate the best possible sequence grounded to the sampled entity chain. Unlike topk or nucleus sampling, it avoids degeneration by instilling diversity in composition, rather than directly on the surface form. Our contributions can be summarized as follows: (a) we introduce Composition Sampling, a simple yet effective decoding method for diverse conditional generation, which combines planning with stochastic sampling; (b) we propose several metrics to compute semantic diversity in generated text; our metrics are complementary to lexical diversity 1 Holtzman et al. (2020) use the term degeneration' to describe automatically generated text that is generic, repetitive, and awkward for story continuation. These issues are less common in conditional generation. In our case, degenerate' refers to text unfaithful or inconsistent to the input. Haman Written Summary: Chelsea star Eden Hazard is set to make his 100th top-flight appearance. Santi Cazorla should hit the same milestone when Arsenal meet Burnley. Both players have impressed since moving to the Premier League in 2012. Hazard has more goals this season but Cazorla has one more assist. Sportsmail's reporters choose the player who has excited them the most. (e.g., Self-BLEU; Zhu et al. 2018; Alihosseini et al. 2019) and assess whether a set of diverse outputs are contextually dissimilar ( Self-BERTscore ; Zhang et al. 2020b) or non-entailing ( Self-Entailment ); and (c) finally, we introduce, EDNA , a novel metric aiming to  E valuate D iversity a N d f A ithfulness for summarization by quantifying whether summaries in a diverse set are faithful to their input without entailing each other. Evaluation on two popular summarization tasks, namely highlight generation (CNN/DailyMail; Hermann et al. 2015) and extreme summarization (XSum; Narayan et al. 2018), and question generation (SQuAD; Rajpurkar et al. 2016; Zhou et al. 2017), shows that composition sampling is most effective in generating diverse summaries or questions. When assessed by humans, composition sampled summaries are as faithful as the best summaries produced with beam search. In comparison, nucleus sampled summaries can be as diverse but far less faithful. Taken together our results demonstrate that Composition Sampling is currently the best available decoding strategy for generating diverse and meaningful output. 2 2 Background Conditional generation tasks such as summarization (See et al., 2017), data-to-text generation (Wiseman et al., 2017), and machine translation 2 Our checkpoints and spaCy annotation code are available at https://github.com/google-research/ language/tree/master/language/frost . (Bahdanau et al., 2015), are typically modeled using attention-based encoder-decoder architectures (Bahdanau et al., 2015; Gu et al., 2016; Vaswani et al., 2017). The encoder first encodes the input text d and then the decoder predicts the output s 1: n (e.g., the translation or summary of d ) one token at a time as p ( s i | s 1 , . . . , s i  1 ; d ) , where, n is the output length and s i is the i th token in the output. Often these models benefit from large scale task-agnostic pretraining (Song et al., 2019; Radford et al., 2018; Lewis et al., 2019; Rothe et al., 2020; Raffel et al., 2019; Zhang et al., 2020a). Plan-based Conditional Generation Narayan et al. (2021) develop a plan-based approach for neural summarization; their decoder generates a composition c 1: m of target summary s as p ( c j | c 1 , . . . , c j  1 ; d ) , and then the same decoder produces s as p ( s i | s 1 , . . . , s i  1 ; c ; d ) conditioned on input d and composition c 1: m , with m being the composition length. Specifically, they adopt entity chains as the composition c of summary s , under the assumption that entities in the chain ought to be observed in the output summary. During inference, the model takes document d as input and generates c ; s , the concatenation of composition and summary sequences, instead of generating s directly; c and s are prefixed with special markers  [ CONTENT ]  and  [ SUMMARY ] , respectively, as shown in Figure 2. If s consists of multiple sentences, markers |||  denote sentence boundaries in composition c . The approach allows to directly manipulate the content of summaries and their quality. For example, we might inspect the predicted chain during inference and drop entities which are not present in the input document, thereby controlling for hallucinations (Narayan et al., 2021). Outwith summarization, similar constraints can be easily adapted to other conditional generation tasks. Maximization-Based Decoding In order to obtain the most likely output  s from encoder-decoder models, we typically solve a maximization-based objective:  x = arg max x p ( x | d ) , where x is either the predicted output text s (for models without planning) or the concatenation of the predicted composition and the output text c ; s (for models with planning). It is standard practice to use beam search (Tillmann and Ney, 2003; Li et al., 2016; Wiseman et al., 2017) as solving the objective for the optimal sequence with neural sequence models is not tractable (Chen et al., 2018). Stochastic Sampling for Diverse Decoding Sampling-based strategies have been widely used to induce diversity in language models. Temperature sampling uses a temperature to skew the distribution towards high probability tokens at each decoding step (Ackley et al., 1985; Ficler and Goldberg, 2017; Fan et al., 2018), while topk sampling truncates the distribution to k high probability tokens (Fan et al., 2018; Holtzman et al., 2018; Radford et al., 2019). Similarly to topk sampling, nucleus sampling (Holtzman et al., 2020) also truncates the tail of the distribution but chooses k dynamically. At each decoding step, it samples high-probable tokens from a nucleus N defined as the smallest subset of tokens from the vocabulary V with cumulative probability p (cid:48)  p , where p is the pre-specified mass of the nucleus. Aralikatte et al. (2021) introduce focus sampling to promote diversity in summarization models. It constructs a subset V k  V by sampling k source-relevant and topical tokens from the vocabulary distribution. Standard beam search decoding is then used to generate a summary limited to V k . However, the authors show that focus sampling is very sensitive to k ; increasing it improves generation quality but at the cost of diversity. Composition Sampling is a novel hybrid method which combines stochastic sampling with maximization-based decoding, whilst leveraging plan-based generation (Narayan et al., 2021). Specifically, we employ nucleus sampling to obtain diverse compositions c sample from p ( c | d ) where d is the input text and c are entity chains (prefixed with  [ CONTENT ]  in Figure 2). We first employ nucleus sampling to obtain diverse compositions from p ( c | d ) , where d is the input text. And then employ beam search to generate the most-likely diverse output s (prefixed with  [ SUMMARY ]  in Figure 2), given input d and composition c sample as 1321 p ( s | c sample ; d ) . We will experimentally show that composition sampling enables the generation of fluent, faithful and diverse texts for conditional generation. Why Entity Chains? Unlike topk or nucleus sampling, composition sampling avoids degeneration by introducing diversity in composition, rather than directly on the surface form. For this to effectively work, the choice of c needs to be well correlated with an underlying notion of semantic composition, which we want to diversify; if c 1 and c 2 are two semantic compositions for input d such that c 1 (cid:54) = c 2 , then two summaries s 1 = arg max s p ( s | c 1 ; d ) and s 2 = arg max s p ( s | c 2 ; d ) are bound to be diverse. In our work, we have cho-sen entity chains to model semantic compositions; entity chains have been widely studied to model entity-level lexical cohesion (Barzilay and Elhadad, 1997) and coherence (Halliday and Hasan, 1976; Azzam et al., 1999) in text. Also, entity chains are unique to d , and thus can be easily distinguished from compositions for other inputs. Moreover, entity chains provide a very effective knob for content control in abstractive generation, e.g., compositions can be constrained to entities only present in the input document, thereby avoiding hallucinations and entity degeneration. Hypothesis 1: If the semantic composition c of the output text s corresponds to entity chains, then learning p ( c | d ) is much easier than learning p ( s | d ) ; d is the input. Hence, we can sample from p ( c | d ) with higher confidence than sampling directly from p ( s | d ) , and then compute arg max s p ( s | c ; d ) . We demonstrate the effectiveness of entity chains as a choice for c using the summarization example in Figure 3. The negative log likelihood of generating the summary s from scratch without planning (  log p ( s | d ) ) is 121 . 18 , while the negative log likelihood of generating composition c with planning (  log p ( c | d ) ) is 46 . 95 ; hence, the model is much more confident when sampling from p ( c | d ) than directly from p ( s | d ) . Why Grounded Generation? The generation of s is inherently grounded to its entity composition c ; following Narayan et al. (2021), the entity chains are extracted from their targets during training. Hence, once the hard part of planning the composition is done, the model is less perplexed during generation of the output. In Figure 3, the plan-based model is more confident in predicting entities than its counterpart without planning; perplexities of predicting entities in the summary with and without planning are 0 . 24 and 1 . 36 , respectively, and perplexities of generating the whole summary with and without planning are 1 . 15 and 1 . 48 , respectively. In fact, despite the increased length of the target in the plan-based model (i.e., c 1: m ; s 1: n instead of s 1: n ), we find that the perplexity of predicting the longer sequence ( c 1: m ; s 1: n ) is lower than predicting just the summary without any planning, due to grounding ( 1 . 16 vs 1 . 48 ). Overall, p ( c ; s | d ) , the plan-based approach, learns a more confident distribution at each decoding step compared to no planing, i.e., p ( s | d ) . For the example in Figure 3, the average cumulative probabilities for the top 15 tokens in the vocabulary distribution at each decoding step are 0 . 283 for p ( s | d ) and 0 . 433 for p ( c ; s | d ) . In the following we assess composition sampling for its ability to generate semantically diverse output for two tasks, namely summarization (Sec-1322 tion 4) and question generation (Section 5). We evaluate our decoding strategy on two popular single document summarization datasets: CNN/DailyMail highlight generation (Hermann et al., 2015) and XSum extreme summarization (Narayan et al., 2018), using the original train/validation/test splits. Inputs and outputs were truncated to 512 and 128 for XSum, and, 1,024 and 256 for CNN/DailyMail. 3 We conduct experiments with state-of-the-art pretrained models for summarization, namely PEGASUS (Zhang et al., 2020a) and FROST (Narayan et al., 2021). Our PEGASUS finetuned model generates summaries directly, whereas FROST generates the entity chain followed by the summary. In both cases we use large transformer architectures (Vaswani et al., 2017) with L = 16 , H = 1 , 024 , F = 4 , 096 , A = 16 (568M parameters), where L denotes the number of layers for encoder and decoder Transformer blocks, H is the hidden size, F the feed-forward layer size, and A the number of self-attention heads. Since this paper is proposing a decoding strategy, there is no need to train new summarization models. We use the publicly available PEGASUS and FROST checkpoints. Training details and model hyperparameters can be found in Zhang et al. (2020a) and Narayan et al. (2021). All models are decoded with a beam size of 8 and a length-penalty of 0 . 8 . For nucleus sampling and composition sampling, we use nucleus probability p set to 0 . 95 . 4 For focus sampling (Aralikatte et al., 2021), we use k = 10 , 000 . We assess our decoding strategy for likelihood, flu-ency, relevance, faithfulness, and diversity, using both automatic and human evaluation. FROST models predict a plan in the form of an entity chain, followed by a summary. All evaluations, except likelihood, are done on the summary, while the predicted entity chains are stripped out. For each diverse decoding strategy, we sample 5 times for each test document and report the average. 3 We also experimented with MultiNews (Fabbri et al., 2019), a multi-document summarization dataset. Results can be found in the Appendix (Table 7). Sequence Likelihood We report the perplexity of the generated sequence (i.e., entity chains concatenated with their summaries for planning models and summaries only for the others) using various decoding strategies. Lexical Fluency and Relevance We report ROUGE-L F1 scores (Lin and Hovy, 2003) against reference summaries. 5 Semantic Relevance We report BERTScore (Zhang et al., 2020b) which computes the contextual similarity between a candidate and its reference. Faithfulness We follow Maynez et al. (2020) and report on textual entailment (Pasunuru and Bansal, 2018; Falke et al., 2019; Kryscinski et al., 2020). In particular, we report the probability of a summary entailing ( Entailment ) its input document using a classifier trained by fine-tuning an uncased BERT-Large pretrained model (Devlin et al., 2019) on the Multi-NLI dataset (Williams et al., 2018). We further assess faithfulness by humans. Our annotators, proficient in English, were tasked to read a document and then grade its summary on a scale of 14 ( entirely unfaithful , somewhat unfaithful , somewhat faithful , and entirely faithful ); a summary is entirely faithful if its content is fully supported or can be inferred from the document. We collected 3 ratings for each (document, summary) pair; we report average system ratings (across doc-uments). With summaries deemed somewhat un-faithful and somewhat faithful, annotators were asked to also specify what was faithful or unfaithful, to improve agreement. Diversity We report the number of times (out of five samples), a decoding technique is able to generate a completely new summary ( Unique ). We also use Self-BLEU (Zhu et al., 2018; Alihosseini et al., 2019) to measure lexical diversity in the generated summaries. We consider all pairs of summaries out of 5 samples, and for each pair we compute the BLEU score (Papineni et al., 2002) considering one summary as a hypothesis and the other as a reference. We report the average BLEU score as the Self-BLEU of the document. The lower the Self-BLEU for a decoding strategy is, the better it is in generating a more diverse set of summaries. We propose two additional measures to capture semantic diversity in summaries: Self-Entailment and Self-BERTScore . Similar to Self-BLEU, we compute the Entailment score and BERTScore for each possible pair of summaries, respectively and report the average. A lower Self-Entailment value suggests that the generated summaries do not entail each other. Analogously, a low Self-BERTScore value indicates that the decoding technique is able to generate more contextually dissimilar summaries. We further assess diversity by humans. Our annotators, proficient in English, again read two summaries (out of five samples) and then graded the pair on a scale of 14 ( identical , somewhat identical , somewhat diverse , and diverse ); the document was not shown in this assessment. Two summaries are identical if they are semantically equivalent, while the same information may be presented differently in the case of somewhat identical. A somewhat diverse pair may introduce one or two new concepts or topics in one summary. A diverse pair should introduce new concepts or topics in each summary. We collected 3 ratings for each pair and report their average. This assessment was only done with single-sentence XSum summaries, in future work we will explore how to do this effectively for multi-sentence summaries. Diversity and Faithfulness For summarization, diverse summaries are not meaningful if they are not faithful to the input. We propose EDNA , a novel measure for  E valuating D iversity a N d f A ithfulness in summaries. EDNA is the harmonic mean of Entailment and ( 1  Self-Entailment); higher values of EDNA imply more faithful and diverse summaries. The reason EDNA relies on Self-Entailment to measure diversity is because the faithfulness metric is also based on Entailment. This means that both components will be mapped to a score in a similar output space (i.e., they both yield values between 0 and 1 obtained through the same trained model), making it more likely to be properly balanced when mixed. Table 1 presents ROUGE results on the XSum and CNN/DailyMail test sets. The top block includes results for models which employ maximization-based decoding. GSum (Dou et al., 2020) is a state-of-the art system which decodes summaries guided by an an extractive model at test time. CTRLsum (He et al., 2020) controls the summarization output trough keywords and automatically extracted sentences. FAME (Aralikatte et al., 2021) uses a focus attention mechanism to encourage the decoder to proactively generate tokens that are similar or topical to the input document. As mentioned earlier FROST (Narayan et al., 2021) first generates an entity chain and then a summary while FROST ++ is a constrained variant which restricts the predicted entities to those present in the document. We also show results for a vanilla PEGASUS model (Zhang et al., 2020a) finetuned on our datasets. The bottom block focuses on diverse decoding (we report averages across five samples). We show results with Focus sampling (Aralikatte et al., 2021) built on top of FAME , Nucleus sampling (Holtzman et al., 2020) with PEGASUS and FROST , and our Composition sampling. Table 2 presents more detailed faithfulness and diversity results, on challenge sets consisting of 50 documents for each XSum and CNN/DailyMail 1324 Models ppl With Ref. summaries. We construct these challenge sets by randomly selecting documents whose reference summaries have non-extractive entity chains in them; an entity chain is extractive if all entities in it can be found in the input document. Narayan et al. (2021) have found that models struggle to generate faithful summaries for documents with data-divergence issues (Dhingra et al., 2019). The same challenge sets were used for our human evaluations of faithfulness and diversity. Diminishing as Nucleus Sampling Single-best decoding for FROST achieves 39.76 ROUGE (RL) on XSum,; nucleus and composition sampling fare worse showing an average drop of 7.27 and 2.78, respectively. Similarly, for CNN/DailyMail, ROUGE drops for nucleus sampling by an average of 6.51 points, compared to an average drop of 3.28 points for composition sampling (with FROST ). Nucleus sampling is even more damaging for non-plan based models, such as PEGASUS ; we see an average drop of 8.59 and 7.30 ROUGE points on XSum and CNN/DailyMail. These gaps are slightly larger for the challenging subsets in Table 2 which is expected due to the highly abstractive nature of the reference summaries therein. FROST ++ performs slightly worse than with vanilla FROST in terms of ROUGE . This is due to the extreme abstractive nature of the XSum reference summaries (Maynez et al., 2020); as a result, a model is required to hallucinate factual content, that is not necessarily faithful to the input (see examples of XSum summaries in the Appendix, Figure 5). But Composition(F ROST ++ ) only keeps supported entities in the sampled plans giving rise to summaries which diverge from their reference. This is not the case with CNN/DailyMail which is mostly extractive and we see that ROUGE performance improves with Composition(F ROST ++ ) in Table 1. Predictions than Nucleus Sampling Perplexity for FROST predictions increases from 0 . 31 to 0 . 83 for nucleus sampling, but only to 0 . 51 for composition sampling, on XSum. PEGASUS shows an even larger increment in perplexity (from 0 . 51 to 1 . 47 ) for nucleus sampling. Similar patterns are observed for CNN/DailyMail summaries. Composition(F ROST ++ ) is more perplexed when generating XSum summaries due to the reference summary divergence issue discussed earlier; perplexity increases from 0 . 51 to 0 . 74 compared to Composition(F ROST ). Interestingly, Composi-1325 tion(F ROST ++ ) is almost as confident in generating diverse summaries as single-best beam decoding (FROST ; perplexities of 0 . 71 vs 0 . 74 for XSum). Unsurprisingly, Composition(F ROST ++ ) is more confident in generating CNN/DailyMail summaries than FROST ( 0 . 46 vs 0 . 52 ) due to their extractive nature. Generating Meaningful Diverse Summaries It is no surprise that nucleus sampling is able to generate the most diverse summaries on both XSum and CNN/DailyMail (achieving best scores for Self-BLEU, Self-Entailment, Self-BERTScore, and diversity assessed by humans); however these summaries perform poorly on faithfulness measures. Composition(F ROST ++ ) is most effective in generating faithful summaries, as demonstrated automatically (with best entailment scores on XSum and CNN/DailyMail) and by humans (with highest ratings on XSum and CNN/DailyMail); these summaries are also diverse achieving highest EDNA scores on both summarization datasets. We also examined whether models differ in terms of faithfulness and diversity as rated by our participants. We carried out pairwise comparisons using one-way ANOVA with post-hoc Tukey HSD tests ( p < 0 . 01 ). The difference between Nucleus(P EGASUS ) and Nucleus(F ROST ) is not significant. Nucleus(P EGASUS ) was also not significantly more faithful than Focus(F AME ) for XSum summaries. All other pairwise differences were significant for both faithfulness and diversity. In sum, our results demonstrate that composition sampling is a better alternative to nucleus or focus sampling for generating meaningful diverse summaries. Figure 1 presents summaries from different decoding strategies for a CNN/DailyMail article. Other example predictions for XSum and CNN/DailyMail articles can be found in the Appendix (Figures 59). Faithfulness and Diversity Metrics Correlate with Human Judgements We estimate the extent to which automatic metrics of faithfulness and diversity correlate with human ratings (using Spearman's rank correlation coefficient) in Table 3. In line with previous work (Maynez et al., 2020; Kryscinski et al., 2019), we find that entailment scores are best correlated with faithfulness (mod-erate, 0 . 40  r  0 . 59 ). Like Self-BLUE, Self-Entailment and Self-BERTScore are also strongly Metric Faithfulness Diversity ROUGE-L 0.197  0.164 BERTScore 0.209  0.195 Entailment 0.588  0.067 1 Self-BLEU  0.208 0.880 1 Self-Entailment  0.187 0.771 1 Self-BERTScore  0.198 0.873 EDNA 0.482 0.174 Table 3: Different automatic metrics and their correlation against human assessments using Spearman's rank coefficient. correlated with diversity ratings. Compared to other metrics which capture a single dimension, EDNA is positively correlated with both dimensions of diversity and faithfulness. Finally, in Figure 4, we plot faithfulness and diversity scores for different decoding strategies with varying temperatures and nucleus probabilities. We find that summaries sampled with Composi-tion(F ROST ++ ) are consistently more faithful than single-best Beam(F ROST ) summaries, but worse than summaries decoded with Beam(F ROST ++ ). Summaries sampled with Composition(F ROST ++ ) achieve the best EDNA score (with p = 0 . 95 ) amongst all diverse decoding strategies. Question generation is often conceptualized as the task of generating a question from a passage-answer pair (Zhou et al., 2017). We experiment on SQuAD (Rajpurkar et al., 2016) and use the split of Zhou et al. (2017) consisting of 86,635, 8,965, and 8,964 source-target pairs for training, validation, and testing, respectively. 6 We follow Cho et al. (2019) and report BLEU-4 (Top-1, the single-best accuracy), Oracle (Top-5, the best accuracy among the 5-sampled hypotheses), and Self-BLEU (as defined in Section 4). For our question generation experiments we also compare models which employ single-best decoding against models which adopt diverse decoding techniques. The top block in Table 4 presents results for NQG++ (Zhou et al., 2017), a pointer generator-based model, CP+GSA (Zhao et al., 6 We also experimented with the split of Du et al. (2017). 2018), a model which combines a pointer mechanism with a gated self-attention encoder, and finetuned PEGASUS and FROST models. The second block in the table contains several diverse decoding approaches including topk sampling (Fan et al., 2018), diverse beam search (Vijayakumar et al., 2018), mixture decoding (Shen et al., 2019) and mixture content selection (Cho et al., 2019; Wang et al., 2020). We compare these models against nucleus sampling with PEGASUS and FROST , and composition sampling with FROST . As in our summarization experiments, we observe that composition sampling is not as performance diminishing as nucleus sampling, in terms BLEU. FROST achieves a BLEU of 21 . 04 (top-1) in the single-best decoding setting; in comparison, BLEU drops for nucleus sampling by 10 . 40 points (on average), and 2 . 27 points only for composition sampling (FROST ++ ). Nucleus sampled questions achieve the best pairwise diversity scores (Self-BLEU of 25 . 50 ), but very low BLEU Top-1 score of 10 . 64 . Composition sampled questions are less diverse then other methods, but outperform all baselines on Top-1 and Oracle metrics. Poor diversity (in terms of Self-BLEU) in composition sampled questions can be attributed to two limitations: (a) SQuAD questions are mostly extractive, and (b) questions are generated conditioned on the passage and the answer spans; leaving limited scope for models to generate diverse questions. An example in the Appendix (Figure 11) demonstrates the effectiveness of composition sampling in generating accurate and diverse questions compared to other sampling methods. 7 6 Conclusion We proposed Composition Sampling, a simple yet effective decoding method for faithful and diverse conditional generation. Our method is straightforward to implement and does not require any external system to augment the input during inference. Our experiments demonstrate that it is currently the best available decoding strategy for generating diverse and meaningful output. We also introduced Self-Entailment and Self-BERTScore, to automatically compute semantic diversity in summaries, and, EDNA , for jointly measuring faithfulness and diversity. We thank the reviewers, the ARR action editor, and the senior area chair for their valuable feedback. We would like to thank Ryan McDonald, Ankur Parikh, and Slav Petrov for their insightful comments. Many thanks also to Ashwin Kakarla and his team for their help with the human evaluation. The nature of text generation leads to multiple ethical considerations when considering applications. The main failure mode is that the model can learn to mimic target properties in the training data that are not desirable. Faithfulness and Factuality Since models create new text, there is the danger that they may neither be faithful to the source material nor factual. This can be exacerbated when the data itself has highly abstractive targets, which require the model to generate words not seen in the source material during training. This often leads the model to generate content inconsistent with the source material (Maynez et al., 2020; Kryscinski et al., 2020; Gabriel et al., 2021). Trustworthy Data If the data itself is not trustworthy (comes from suspect or malicious sources) the model will naturally become untrustworthy as it will ultimately learn the language and topics of the training data. For instance, if the training data is about Obama birther conspiracies, and the model is asked to generate information about the early life of Obama, there is a risk that false claims will be predicted by the model. Bias in Data Similarly, biases in the data around gender, race, etc., risk being propagated in the model predictions, which is common for most NLP tasks. This is especially true when the models are trained from non-contemporary data that do not represent current norms and practices (Blodgett et al., 2020). The above considerations are non-malicious, in that the model is merely learning to behave as its underlying source material. If users of such models are not aware of these issues and do not account for them, e.g., with better data selection and evaluation, then the generated text can be damaging. Generation models can also be misused in malicious ways. These include generating fake news, spam, and other text meant to mislead large sections of the general population.
abstain abstain abstain abstain abstain objective abstain objective abstain abstain abstain other abstain objective abstain abstain abstain objective abstain abstain abstain objective method abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain objective other abstain abstain abstain other abstain abstain abstain method abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain method abstain abstain method method method abstain method abstain abstain abstain method abstain method method abstain abstain method method abstain method abstain method method method abstain abstain method abstain abstain method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain method method abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain