Abstractive summarization models are commonly trained using maximum likelihood estimation, which assumes a deterministic (one-point) target distribution in which an ideal model will assign all the probability mass to the reference summary. This assumption may lead to performance degradation during inference, where the model needs to compare several system-generated (candidate) summaries that have deviated from the reference summary. To address this problem, we propose a novel training paradigm which assumes a non-deterministic distribution so that different candidate summaries are assigned probability mass according to their quality. Our method achieves a new state-of-the-art result on the CNN/DailyMail (47.78 ROUGE-1) and XSum (49.07 ROUGE-1) datasets. Further analysis also shows that our model can estimate probabilities of candidate summaries that are more correlated with their level of quality. 1 1 Introduction Neural methods for abstractive summarization (Rush et al., 2015; Nallapati et al., 2016; Chopra et al., 2016; Lewis et al., 2020; Zhang et al., 2020) formulate summarization as a sequence-to-sequence (Seq2Seq) problem (Sutskever et al., 2014), learning to generate the summary in an autoregressive manner. Such models are commonly trained with maximum likelihood estimation (MLE), maximizing predictive probability of the reference output given the gold sub-sequence before it. However, during inference the model must also generate the output based on possibly erroneous previous steps. This can hurt model performance, a phenomenon often called exposure bias (Bengio et al., 2015; Ranzato et al., 2016). To maintain reasonable performance even in the case of a sub-sequence with errors, we argue that the 1 We have made our code, results, and trained models publicly available at https://github.com/yixinL7/BRIO. model must accurately estimate relative quality of different generated outputs, since effective inference requires comparison among these candidates. To understand whether existing models can accurately perform such relative comparisons, we conducted a preliminary study on pre-trained BART (Lewis et al., 2020), first generating two candidate summaries from the model and observing whether a higher probability is assigned to the candidate with a higher ROUGE (Lin, 2004) score. As Tab. 1 shows, the accuracy is far from ideal. This is likely due to the fact that MLE training only encourages the model to assign high probability to the reference summary, and is agnostic about any relative comparison between non-reference summaries. However, we argue that it is also important for the order of model scores to be coordinated with the actual quality metrics by which the summaries will be evaluated  higher model scores should indicate better quality summaries. In the following we will refer to models that have such scores as coordinated for conciseness. We introduce a training paradigm which requires the abstractive model to be able to be accurate with respect to predicting the tokens in the reference summaries and coordinated with respect to 2890 Encoder Decoder         Source Input Reference Output   Encoder Decoder  ()  ()  ()  () Source Input Candidate Output  Decoder  ()  ()  ()  () Candidate Output    () ()  > Seq2Seq Generation Model Reference-free Evaluation Model   Figure 1: Comparison of MLE loss ( LMLE ) and the contrastive loss ( L Ctr ) in our method. the candidate summaries. In other words, we give the abstractive model a dual role: as a generation model, it generates the output summaries in an autoregressive way; as an evaluation model, it can be used to score the quality of candidate summaries by estimating a probability distribution over candidate outputs. The generation model is trained using the standard MLE loss, but to train the evaluation model we introduce a contrastive loss (Hadsell et al., 2006) defined over different candidate summaries generated by pre-trained abstractive models (Fig. 1), following previous work on ranking-based or contrastive learning (Hopkins and May, 2011; Zhong et al., 2020; Liu et al., 2021b). Our main contribution is to change the target distribution of abstractive models from a one-point deterministic distribution assumed by MLE training to a non-deterministic distribution in which candidate summaries are also assigned probability mass according to their quality. The new SOTA performance on CNN/DailyMail (Hermann et al., 2015) and XSum (Narayan et al., 2018) datasets demonstrated the effectiveness of our method. Our in-depth analysis also found that the abstractive models trained using our method can estimate the candidate summary quality more accurately, in concert with the the objective of our training paradigm. The goal of abstractive summarization is to create a function g that takes a source document D and generates an appropriate summary S Training Objective Neural abstractive summarization models aim to learn a neural model g that results in good summaries. Maximum likelihood estimation (MLE) is the standard training algorithm. It aims to maximize the likelihood of the reference summary S  , i.e.,   = argmax  (cid:88) i log p g  ( S  ( i ) | D ( i ) ;  ) (2) where  denotes the parameters of g and p g  denotes the probability distribution entailed by these parameters. The summation is over the training set and { D ( i ) , S  ( i ) } is the i -th training sample. For a specific sample { D ( i ) , S  ( i ) } , Eq. 2 is equivalent to minimizing the sum of negative log-likelihoods of the tokens { s  1 ,    , s  j ,    , s  l } in the reference summary S  whose length is l , which is the cross-entropy loss: L xent =  l (cid:88) j =1 (cid:88) s p true ( s | D, S  <j ) log p g  ( s | D, S  <j ;  ) (3) where S  <j denotes the partial reference sequence { s  0 ,    , s  j  1 } and s  0 is a pre-defined start token. p true is a one-hot distribution under the standard MLE framework: p true ( s | D, S  <j ) = (cid:40) 1 s = s  j 0 s (cid:54) = s  j (4) In practice, label smoothing (Szegedy et al., 2016) is a widely used and effective technique that modi-fies the target distribution in Eq. 4 to a "soft" label by assigning probability mass  to other tokens: p true ( s | D, S  <j ) = (cid:40) 1   s = s  j N  1 s (cid:54) = s  j (5) where N is the size of the dictionary. Inference and Exposure Bias During inference, the abstractive model g is used to generate the candidate summary in an autoregressive manner. It is intractable to enumerate all the possible candidate outputs, so in practice methods such as beam search are used to reduce the search space. One important step in search is estimating the probability of the next word s t given the previous predicted sequence S <t : Comparing Eq. 6 with Eq. 3, the major difference is that during inference the model makes new predictions based on its own previous predictions S <t instead of the reference S  <t . As a result, even if the generation model g achieves very high accuracy w.r.t. Eq. 3, once S <t starts to deviate from S  , there is the risk that the performance of g will significantly degrade. This problem has been identified as the exposure bias (Bengio et al., 2015). Eq. 6 implies that the abstractive model g should be able to assign higher estimated probability to the better candidate summary during inference. However, this intuition is not directly captured in the standard MLE objective used in training  a model obtaining zero MLE loss would assign zero probability to any candidate summary different from the reference. This is obviously improper for any task where multiple reasonable generations may exist (Khayrallah et al., 2020), and also does not say anything about the ordering of two imperfect references. We therefore advocate for making the alternative assumption that the probability of one candidate should be well-correlated with its quality as evaluated by an automatic metric M . Since it is intractable to enumerate all the possible candidate outputs, we only require our model to be able to accurately predict the ranking order of a set of the most probable candidate summaries  S , which are its own beam search results. In order to achieve this objective, we slightly modify the conditions of Eq. 5, maintaining the general functional form, but instead specifying the marginal probability of the non-reference candidates S to be  , and encouraging coordination of probabilities and qualities among non-reference candidates as follows:   p true  ( S | D ) = 1   S = S  (cid:80) S S p true  ( S | D ) =  S (cid:54) = S  p true  ( S i | D ) > p true  ( S j | D )  S i , S j   S , M ( S i ) > M ( S j ) (7) We next describe precisely how we encourage coordination through contrastive learning . Contrastive Learning for Coordination The candidate quality measure M can be defined in many ways. In this work we define it as the ROUGE (Lin, 2004) score of a candidate summary S i given the reference summary S  . To coordinate a pre-trained abstractive model, we 1) use it to generate different candidate summaries with various levels of quality, 2 then 2) encourage the model to assign higher estimated probabilities to better candidates by fine-tuning the model with a contrastive loss, following the previous work (Hopkins and May, 2011; Zhong et al., 2020): L ctr = (cid:88) i (cid:88) j>i max(0 , f ( S j )  f ( S i ) +  ij ) (8) where S i and S j are two different candidate summaries and ROUGE ( S i , S  ) > ROUGE ( S j , S  ) ,  i, j, i < j . ij is the margin multiplied by the difference in rank between the candidates, i.e.,  ij = ( j  i )   . f ( S i ) is the length-normalized estimated log-probability 3 f ( S ) = (cid:80) lt =1 log p g  ( s t | D, S <t ;  ) | S |  (9) where  is the length penalty hyperparameter. This loss gives the abstractive model a dual purpose, first as a reference-free evaluation model, which can be used in a two-stage summarization pipeline, where it is used to score the candidates generated by a pre-trained generation model and select the final output from them. However, since the autoregressive generation depends on both the token-level prediction accuracy and sequence-level coordination , the model fine-tuned with the contrastive loss alone can no longer be used as a generation model. Multi-task Fine-tuning Following Edunov et al. (2018), we combine the contrastive (Eq. 8) and cross-entropy (Eq. 3) losses to preserve the generation ability of the pre-trained abstractive model: L mul = L xent +  L ctr (10) where  is the weight of the contrastive loss. We note that the contrastive and the cross-entropy loss can effectively complement each other  since the contrastive loss is defined on the sequence level, the token-level cross-entropy loss serves as a normalization to ensure that the model could assign balanced probability mass across the whole sequence. Training Methods of Seq2Seq Models In order to align the training objective and evaluation metric, structured losses have been used for the Seq2Seq model training. Among them, margin-based losses (Herbrich et al., 1999; Taskar et al., 2004; Gimpel and Smith, 2010), which require the model to assign higher probability to the better output, are a major category. Many margin-based losses used in modern seq2seq models (Wiseman and Rush, 2016; Edunov et al., 2018) assume a deterministic (one-point) distribution: a model can achieve zero loss if it can assign a much higher probability to the (pseudo)-reference, regardless of relative comparisons of other candidate summaries. By contrast, our method has a non-deterministic assumption (Eq. 7), which focuses on the pair-wise ranking of a set of candidate summaries. One main challenge of directly optimizing a Seq2Seq model with quality scores of the output is that the discrete sampling process makes the loss non-differentiable. To circumvent this problem, reinforcement learning has been used to reformulate the conditional text generation tasks (Ranzato et al., 2016; Bahdanau et al., 2016; Li et al., 2016; Paulus et al., 2018; Li et al., 2019). Compared to this school of methods, our method is based on supervised learning, and it is more stable and less sensitive to the design choices (e.g. reward shaping), which are well-known challenges of reinforcement learning methods. Minimum risk training (Shen et al., 2016; Wieting et al., 2019) and other online sampling based methods (Bengio et al., 2015; Norouzi et al., 2016; Zhang et al., 2019) belong to another school of methods used to circumvent the problem of non-differentiability. However, they also exhibit similar problems of stability as reinforcement learning. Contrastive Learning Recently, contrastive learning (Hadsell et al., 2006) has been introduced into several conditional text generation tasks, such as machine translation (Yang et al., 2019; Pan et al., 2021), text summarization (Cao and Wang, 2021; Xu et al., 2021; Sun and Li, 2021), and other tasks (Uehara et al., 2020; Cho et al., 2021; Lee et al., 2021b). Among these application scenarios, most work deployed contrastive learning in the latent representation space , following the framework proposed in Chen et al. (2020). However, in this work we adopt contrastive learning over the discrete space of the generated texts. Besides, instead of constructing the contrastive learning examples by rule-based methods (e.g. perturbing the reference output), we use the generation models to construct the examples, which makes the contrastive learning task closer to the generation task. Sun and Li (2021) also adopted contrastive learning on the generated texts. However, their formulation belongs to the margin-based losses. We have discussed the difference between our method and the margin-based losses in the previous paragraphs. Discriminative Reranking Discriminative reranking has been widely studied for conditional generation tasks (Shen et al., 2004; Och et al., 2004; Wan et al., 2015; Mizumoto and Matsumoto, 2016). Some recent works (Liu and Liu, 2021; Lee et al., 2021a) have also explored discriminative reranking of candidates from neural natural language generation models, which adopt large pre-trained language models (e.g. BERT (Devlin et al., 2019)) as the reranker. In this work, we factorize the Seq2Seq model (e.g., BART) trained on the same dataset as the reranking model, which maximizes the parameter sharing across two stages. Besides, our approach contributes an instance of leveraging large pre-trained Seq2Seq models as a quality estimation model (Yuan et al., 2021). experiments (statistics in Appendix A). CNNDM 4 (Hermann et al., 2015) is a large scale news dataset. Following Nallapati et al. (2016), we treat the news articles as the source documents and the associated highlights as the summaries. XSum 5 (Narayan et al., 2018) is a highly abstractive dataset of articles from the British Broadcasting Corporation (BBC). NYT 6 (Sandhaus, 2008) contains articles from the New York Times and the associated summaries. We follow Kedzie et al. (2018) for data preprocessing and splitting, and use the associated archival abstracts as the summaries. Baselines We choose a variety of related models with strong performance as baselines. BART (Lewis et al., 2020) and PEGASUS (Zhang et al., 2020) are both large pre-trained Seq2Seq LMs standard in the literature. GSum (Dou et al., 4 https://cs.nyu.edu/~kcho/DMQA/ 5 https://github.com/EdinburghNLP/XSum 6 https://catalog.ldc.upenn.edu/LDC2008T19 2893 2021) is built on BART, and improves performance by using additional guidance from an extractive summarizer. SimCLS (Liu and Liu, 2021) introduces a two-stage framework where the pre-trained BART model is used to generate candidates and a pre-trained RoBERTa (Liu et al., 2019) model is fine-tuned as an evaluation model to score the candidate summaries and select from them. It achieves state-of-the-art performance on both CNNDM and XSum . GOLD (Pang and He, 2021) uses offline reinforcement learning to train the BART model by treating the reference summaries as the demonstrations, a different formulation that can also improve the performance of the original BART. SeqCo (Xu et al., 2021) and ConSum (Sun and Li, 2021) are two recent methods that aim to leverage contrastive learning to improve the performance of the abstractive summarization model (BART). Implementation Details In the following experiments, we use either BART or PEGASUS as a backbone. We label our proposed methods BRIO , with two variants: (1) BRIO-Ctr is fine-tuned with the contrastive loss (Eq. 8) only; (2) BRIO-Mul is fine-tuned with the multi-task loss (Eq. 10). We use BRIO-Ctr as an evaluation model that scores different candidate summaries generated by a Seq2Seq abstractive model and selects the final output from them, and BRIO-Mul as a standard Seq2Seq model that takes the source documents as input and generates the output in an autoregressive manner. Further details are in Appendix B. 5.2 Results The results are shown in Tab 2. For CNNDM and NYT we use BART as the backbone model while for XSum we use the pre-trained PEGASUS model as our base model since it achieves better performance than BART. We have the following observations: (1) BRIO-Ctr outperforms SimCLS , its counterpart as an evaluation model in a two-stage summarization framework. Specifically, both BRIO-Ctr and SimCLS are used to score the candidate summaries generated by a Seq2Seq abstractive model (BART). The final outputs are selected based on those scores. We attribute BRIO-Ctr's superior performance to its use of the same model architecture (BART) for both candidate generation and scoring, while SimCLS uses RoBERTa as the evaluation model. As a result, BRIO-Ctr maximizes the parameter sharing between the two stages, and preserves the power of the Seq2Seq model pre-trained on the same dataset. (2) BRIO-Mul is able to establish the new stare-of-the-art performance on CNNDM . Notably, the previous state-of-the-art model, GSum , takes additional guidance as input and needs a separate encoder to encode the guidance information, while BRIO-Mul uses the same parameterization of BART. Compared to other methods (ConSum, SeqCo, GOLD) that aim to improve upon BART, BRIO-Mul performs much better, showing the effectiveness of our training method. (3) Since on XSum we use PEGASUS instead of BART as the base model, the result shows that our method is not restricted to the specific choice of the base model. We further perform some in-depth analyses from diverse perspectives on the CNNDM dataset to gain more insights into our proposed method. Coefficients of the Multi-Task Loss The multitask loss (Eq. 10) used to train our model contains two parts: the cross-entropy loss and the contastive loss. As shown in Tab. 3, as the weight of the contrastive loss (  ) increases, the model's performance improves. However, the cross-entropy loss is still necessary to preserve the model's ability as a generation model. We argue that this is because the token level accuracy is still important during the autoregressive generation process, where the individual tokens are predicted sequentially. In addition, we also found that the model tends to achieve the best performance (w.r.t the ROUGE scores on the development set) faster with a higher  . Specifically, it requires less than one entire epoch to achieve the best performance on CNNDM , making our approach an efficient fine-tuning method. erate, we can use it to generate a new set of candidates in the same way as we used the pre-trained BART model, and continue fine-tuning it on this newly created set of candidates (Och, 2003). Fig. 2 illustrates this iterative process. The results shown in Tab. 4 illustrate that this new model (BRIO-Loop) outperforms BRIO-Mul. Besides, the model reached the best performance very quickly, showing the potential of adopting our method in an online framework where the new candidates are dynamically generated from the current model. We leave this direction for future work. Increasing the Beam Width While theoretically a larger beam width (i.e. the number of candidates maintained during beam search) would allow more candidates to be considered and therefore increase the upper bound of the performance, in practice model performance may be lower if the beam width is too large. The reason for this phenomenon is closely related to the low sequence-level coordination of the generator. Specifically, increasing the beam width may introduce candidates with lower quality (Stahlberg and Byrne, 2019), and the generator may not be able to differentiate them from high-quality candidates. In Tab. 5, we compare the performance of the pre-trained BART and our model (BRIO-Mul) with different beam widths used during inference. We observe that the performance of BART goes down as the beam width increases. On the other hand, our model is able to achieve better performance with a larger number of beams, demonstrating that our training method can improve the coordination of the model by encouraging the model to assign estimated probabilities to candidate summaries well-correlated with their quality. Training with Different Evaluation Metrics In the previous experiments, we used ROUGE as the evaluation metric to define the target ordering of the candidate summaries (Eq.7). To evaluate our method's performance beyond ROUGE, 2895 System R-1 R-2 R-L BS BART 44.29 21.17 41.09 27.38 BRIO-Mul (R) 47.78 23.55 44.57 32.11 BRIO-Mul (B) 47.53 23.22 44.37 32.59 Table 6: Results on CNNDM using different evaluation metrics as M in Eq.7. we use a model-based semantic similarity metric, BERTScore (Zhang* et al., 2020), 7 as the evaluation metric M in Eq.7 to compare the performance of different candidate summaries. Then, we trained another version of BRIO-Mul based on the order of candidate summaries calculated by BERTScore. The results in Tab. 6 show that (1) Our model can significantly improve the model performance when either ROUGE or BERTScore is used as the target evaluation metric for ordering candidate summaries. This suggests that it is possible to use our method to optimize any specific target metric, making our method an alternative to reinforcement learning or minimum risk training. (2) Our model that is trained on one evaluation metric (e.g. BERTScore) also achieves improvement on another metric (e.g. ROUGE) compared with the baseline model, which indicates that the improvement made by our model is not from exploiting the potential weaknesses of individual metrics. Besides, this result also demonstrates a non-trivial degree of agreement between ROUGE and BERTScore. Novel n -grams We compare the ratio of novel n -grams in reference, BRIO-Mul's, and BART's summaries. As Tab. 7 shows, our model is more abstractive compared to BART, although reference summaries still contain more novel n -grams. This is likely due to the fact that our model is optimized at the sequence-level, allowing more free-dom for paraphrasing and compression. We further investigate the relation of the ab-stractiveness" and model performance by com-7 https://github.com/Tiiiger/bert_score. paring our model (BRIO-Mul) with the baseline model (BART) on different buckets of test examples grouped by the novelty" of the reference summaries, 8 i.e., Novelty( D, S  ) = (cid:80) g  GS  1 ( g /  GD ) | GS  | (11) where D and S  are the source document and reference summary respectively, GD and GS  are the sets of bigrams in D and S  , 1 is the indicator function. The results in Fig. 3 show that when novelty is higher, (1) all models' performance decreases; (2) our model achieves larger improvement over the baseline model. Rank Correlation We computed the rank correlation between the estimated probabilities of the candidate summaries calculated by the generators and the quality scores of the candidate summaries. We use Eq. 9 to calculate the estimated probabilities 9 and we use ROUGE-1 as the quality score metric of the candidate summaries. We calculate 8 The calculation is performed using ExplainaBoard (Liu et al., 2021a). https://github.com/neulab/ExplainaBoard. 9 We found the value of the length penalty factor  in Eq. 9 by maximizing the rank correlation on the validation set. 2896 Dataset System ECE Acc Conf CNNDM BART .4097 .3711 .7365 BRIO-Mul .2719 .4271 .6652 XSum PEGASUS .2369 .4688 .6990 BRIO-Mul .1423 .4744 .5881 Table 9: Expected Calibration Error (ECE), accuracy (Acc) and confidence (Conf) on the test set of CNNDM and XSum . 0.0 0.2 0.4 0.6 0.8 1.0 confidence 0.2 0.4 0.6 0.8 1.0 a cc u r a c y CNNDMBART CoordSum-Mul 0.0 0.2 0.4 0.6 0.8 1.0 confidence 0.2 0.4 0.6 0.8 1.0 a cc u r a c y XSum PEGASUS CoordSum-Mul Figure 4: Reliability graphs on the CNNDM and XSum datasets. The accuracy of model's predictions is plotted against the model's confidence on these predictions. Spearman's rank correlation for each sample, and use the average score as the overall correlation, We investigated two specific settings: 1) ranking candidate summaries generated by a different model (PEGASUS); 2) ranking candidate summaries generated by themselves (BART & BRIO-Mul). We use 16 candidates in total for calculation. As Tab. 8 shows, our model achieves better rank correlation on the candidate summaries generated by both itself and the independent model. This suggests that our model can better estimate the quality of candidate summaries. 5.4 Token-level Calibration Calibration requires that a model's confidence on its predictions is equal to the accuracy of these predictions (Guo et al., 2017). Previous work (Mller et al., 2019; Kumar and Sarawagi, 2019; Wang et al., 2020) has found that a more calibrated text generation model tends to have better performance, and techniques like label smoothing can improve both the token-level calibration and sequence-level accuracy (i.e. the ability of generating better results). One intuitive explanation of this phenomenon is to interpret the model's estimated probability of a generated summary as the product of the model's confidences on a series of token-level predictions. Then, since a more calibrated model's confidence estimates better the accuracy of its predictions, the model's estimated probability of one sequence should be more indicative of the quality of this sequence, which is essential for the beam search during inference. However, the relation of token-level calibration and sequence-level performance remains inconclusive (Mller et al., 2019). 10 For example, a generator that always predicts a uniform distribution over all tokens would be perfectly calibrated, however, such a model would not generate high-quality outputs. We investigate this relation from the opposite direction by evaluating whether our model (BRIO-Mul), which is trained to have better sequence-level performance, would also be more calibrated at the token-level compared with the baseline models that are trained using MLE and label smoothing. We follow previous work by using the Expected Calibration Error (Naeini et al., 2015) (ECE) as the evaluation metric of calibration: ECE = M (cid:88) m =1 | B m | n | acc ( B m )  conf ( B m ) | (12) where the samples are grouped into M equal-width buckets by confidence (conf), B m denotes the m -th bucket, and n is the total number of samples. Following Wang et al. (2020), we evaluate model calibration on the system-generated summaries during inference and use the tercom toolkit 11 to assign labels (correct/incorrect) to the system-generated summaries based on the reference summaries. The results in Tab. 9 show that BRIO-Mul is better calibrated compared to BART, suggesting that our method helps to improve the token-level calibration by explicitly encouraging the model to have more accurate sequence-level probability estimations. The reliability graph is shown in Fig. 4. We found that (1) abstractive models are generally over-confident on their own predictions, (2) models are generally more calibrated on XSum than CNNDM . This is likely due to the fact that XSum has shorter summaries therefore it is less likely to be affected by the exposure bias. 5.5 Few-shot Fine-tuning The training paradigm proposed in this paper may be extended to any Seq2Seq model. However, it can be a non-trivial overhead to generate the candidate summaries using large neural models on the entire training set. On the other hand, recent work (Raffel et al., 2020; Zhang et al., 2020; Schick and Schtze, 10 In general, better token-level calibration doesn't guarantee better sequence-level performance. 11 http://cs.umd.edu/~snover/tercom/ 2897 System Summary Reference chelsea forward tammy abraham nets first-half double for chelsea. dominic solanke adds a third late on as chelsea look set to win trophy. manchester city struggle without injured star thierry ambrose. read: mourinho warns his young chelsea players he can not play them all. click here to read our match report from man city 's academy stadium. BART tammy abraham scored twice in the first half to give chelsea the lead. isaac buckley-ricketts levelled the game for manchester city. dominic solanke scored late on to put a gloss on the scoreline. click here to read sportsmail's player ratings from the youth cup final. BRIO-Mul chelsea beat manchester city 3-1 in the youth cup final at the etihad stadium. tammy abraham scored twice in the first half to give chelsea the lead. dominic solanke scored late on to seal the win for the home side. Reference alejandro valverde won ahead of julian alaphilippe and michael albasini. chris froome finished 123rd after a crash during the final 12 kilometres. team sky's sports director gabriel rasch praised froome for finishing. rasch said froome was banged up' but expects to ride tour de romandie. BART movistar rider alejandro valverde won fleche wallonne on wednesday. team sky's chris froome fell in the final 12km but finished the race. philippe gilbert pulled out of the race after a bad crash 50km from the end. click here for more cycling news. BRIO-Mul alejandro valverde defended his fleche wallonne title in belgium on wednesday. movistar rider finished ahead of julian alaphilippe and michael albasini. team sky's chris froome fell in the final 12km of the race but finished in 123rd. froome was involved in a crash but finished the race despite being banged up' Reference manuel pellegrini won the premier league and capital one cup last season. city currently sit fourth in the league table 12 points behind chelsea. pellegrini's contract expires at the end of the 2015-16 season. city players have been impressed with vieira's work with the youth team. pep guardiola is city's first-choice to succeed pellegrini at the etihad. BART manuel pellegrini's future at manchester city is under scrutiny. patrick vieira is highly-respected among the city players. city's first-choice managerial option is bayern munich boss pep guardiola. click here for all the latest manchester city news. click here for more premier league news. BRIO-Mul manchester city players have backed patrick vieira to replace manuel pellegrini as manager of the club. the frenchman is highly-respected among the players at the etihad stadium. pellegrini's future at the club is under scrutiny after a disappointing season. city's first-choice manager is current bayern munich boss pep guardiola. Table 10: Case Study on CNNDM . BRIO-Mul learns to ignore the noise pattern (click here") while BART cannot. Dataset System R-1 R-2 R-L CNNDM BART 44.29 21.17 41.09 BRIO-Few 45.81 21.91 42.61 XSum PEGASUS 47.46 24.69 39.53 BRIO-Few 47.95 24.89 39.71 Table 11: Few-shot Fine-tuning. BRIO-Few is trained on only 100/1000 training examples on CNNDM and XSum respectively. R-1/2/L are ROUGE-1/2/L F 1 scores. 2021; Fabbri et al., 2021) has shown that few-shot learning can be an effective fine-tuning method of pre-trained models for text generation tasks. Therefore, we investigate our model's performance in a few-shot setting. Specifically, we randomly sample 100/1000 examples from the training set of CNNDM / XSum , and fine-tune the models that are pre-trained using MLE loss on those examples. More training details can be found in Appendix C. The results are shown in Tab. 11. All experiments are repeated three times, and the reported results are the average performance. The results indicate that our model can achieve improvement over the baseline model under the few-shot learning setting with a small computational overhead. 5.6 Case Study on CNNDM Tab. 10 presents an interesting pattern we observed when comparing the results of BRIO-Mul and BART, which demonstrates that our method helps the abstractive model to filter out noise patterns in the original data. Specifically, some of the reference summaries (331/11490) in CNNDM contains the phrase click here, pointing to a hyperlink, and 103 source documents also contain this phrase. BART picked up this pattern, and generates this phrase in 96 output summaries. On the contrary, our model learns to ignore this noise pattern and never generated it across the whole test set, likely because it identified that generated candidates with this pattern rarely achieve a high ROUGE score, and downweighted the probability accordingly. In this work, we presented a new training paradigm that assigns candidate outputs probability mass according to their quality using contrastive learning. While our method has achieved significant improvement on abstractive summarization, we note several directions for the future work to explore. First, since our method makes no assumptions specifi-cally about the summarization task, it can be extended to other conditional text generation tasks such as machine translation. Second, it is possible to apply our method in a reinforcement learning setting, where the candidate summaries are dynamically generated. Finally, in experiments we only used diverse beam search to generate the candidate summaries, but it is likely that other candidate generation methods could yield further improvements. We thank the anonymous reviewers for valuable feedback and helpful suggestions.
abstain abstain objective objective result abstain abstain abstain abstain other abstain abstain abstain abstain abstain method abstain method abstain method method objective abstain objective abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method result objective method abstain method method abstain abstain abstain abstain method method other other other objective other other method other other other other method method other other abstain other other method abstain abstain abstain abstain abstain abstain method method abstain other abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain result abstain result abstain abstain abstain abstain abstain other other abstain result objective objective objective method method other