Multi-document summarization (MDS) has made significant progress in recent years, in part facilitated by the availability of new, dedicated datasets and capacious language models. However, a standing limitation of these models is that they are trained against limited references and with plain maximum-likelihood objectives. As for many other generative tasks, reinforcement learning (RL) offers the potential to improve the training of MDS models; yet, it requires a carefully-designed reward that can ensure appropriate leverage of both the reference summaries and the input documents. For this reason, in this paper we propose fine-tuning an MDS baseline with a reward that balances a reference-based metric such as ROUGE with coverage of the input documents. To implement the approach, we utilize RELAX (Grathwohl et al., 2018), a contemporary gradient estimator which is both low-variance and unbiased, and we fine-tune the baseline in a few-shot style for both stability and computational efficiency. Experimental results over the Multi-News and WCEPMDS datasets show significant improvements of up to +0 . 95 pp average ROUGE score and +3 . 17 pp METEOR score over the baseline, and competitive results with the literature. In addition, they show that the coverage of the input documents is increased, and evenly across all documents. Multi-document summarization (MDS) aims to consolidate salient points of information across a set of documents into a concise summary. The main requirement for the summary is that it adequately represent the document set, with low redundancy and high coverage across all documents, while at the same time being readable and fluent. Combined with this, is the need to develop techniques that can handle the significant memory complexity required to tackle MDS. Recently, the release of dedicated datasets (Fabbri et al., 2019; Gholipour Ghalandari et al., 2020), and intelligently designed Transformer models (Liu et al., 2018; Liu and Lapata, 2019; Beltagy et al., 2020), have helped drive advancements in multi-document summarization, generally improving the accuracy and fluency of the predicted summaries. However, aspects such as the requirement to cover as much salient information from the input documents as possible, whilst still maintaining low repetition and low redundancy, have certainly been less explored to date (Nayeem et al., 2018; Mao et al., 2020). Within the sphere of contemporary neural MDS models, two main lines of investigation can be iden-tified: graph-based approaches (Li et al., 2020; Pasunuru et al., 2021), and concatenation approaches (Liu et al., 2018; Zhang et al., 2020a). The former are approaches that rely on the construction of graphs to capture the interand intra-document relations. While powerful, they need to elicit the relations explicitly. The latter instead assume that all the input documents within a document set can be simply concatenated, possibly with document separators and tags, such that the relations can be discovered by the model. Like ordinary summarization, also MDS comes in two remarkably different styles: extractive , where the generated summaries consist of verbatim sentences from the original input documents (Nallapati et al., 2017), and abstractive , where the model is instead encouraged to generate a paraphrased understanding of the input documents. The intrinsic appeal of abstractive summaries and the advent of sequence-to-sequence models have increasingly shifted the trend toward abstractive summarization (See et al., 2017; Paulus et al., 2018; Fabbri et al., 2019; Lewis et al., 2020; Zhang et al., 2020a). As for what models are concerned, abstractive MDS has made increasing use of transformers, both conventional (Lewis et al., 2020; Zhang et al., 2020a) and modified to accommodate the characteristic input length 5112 of multi-document sets (Beltagy et al., 2020; Za-heer et al., 2020). Similarly to general summarization, the majority of MDS models are trained using the negative log-likelihood (NLL) as training objective, which aims to maximize the conditional log-likelihood of the tokens of a given reference summary. Despite its speed and efficacy, the NLL exhibits both the wrong-objective problem (Ding and Soricut, 2017), where the model is trained on a convenient objective rather than a desirable one, and the wellknown exposure bias problem (Bengio et al., 2015; Ranzato et al., 2016). To alleviate these issues, reinforcement learning has been adopted in summarization, as in other language generation tasks, to train the model with a more appropriate objective (Li et al., 2019; Parnell et al., 2021). However, its effective use for MDS requires a reward function that can appropriately balance the reference summary and the multiple input documents in the document set. For this reason, in this paper we propose exploring a reward that combines a reference-based metric such as ROUGE with a coverage term over the input documents. To implement the reinforcement learning approach, we employ a contemporary gradient estimator of the policy gradient, RELAX (Grathwohl et al., 2018), which is both low-variance and unbiased. In addition, to limit the computation and the risk of parameter drift, we apply the objective to fine-tune an NLL-pretrained model in a few-shot manner. In light of the above, this paper makes the following contributions: 1. a reward for reinforcement learning that combines a ROUGE score and a multi-document coverage score, to simultaneously adhere to both the reference summaries and the input documents; 2. a reinforcement learning implementation that leverages a low-variance and unbiased gradient estimator of the policy gradient, RELAX; 3. experimental results and a comprehensive analysis over two MDS datasets (Multi-News and WCEP), showing the empirical effectiveness of the proposed approach. The rest of this paper is organized as follows: first the related work is reviewed in Section 2, and then the proposed approach is introduced in Section 3. Section 4 describes the experimental set-up and main results, while Section 5 presents a more detailed analysis of the main components of the proposed approach. Eventually, Section 6 summarizes our findings and concludes the paper. Early work in multi-document summarization (MDS) that pre-dates the neural era (Mani and Bloedorn, 1997; Erkan and Radev, 2004; Christensen et al., 2013) was shaped around the notion of MDS as a collection of graph structures. As approaches in language generation naturally evolved into neural-based (Rush et al., 2015; Ranzato et al., 2016), later improved with the emergence of large, pre-trained language models (Devlin et al., 2019; Lewis et al., 2020; Zhang et al., 2020a), the effort shifted to integrating these graph structures into the models, often building on top of strong single-document summarization (SDS) baselines (Lebanoff et al., 2018; Zhang et al., 2018). Concurrently, the growing interest in multi-document summarization has led to the development of dedicated, multi-document datasets such as WikiSum (Liu et al., 2018), Multi-News (Fab-bri et al., 2019), Wikipedia Current Events Portal (WCEP) (Gholipour Ghalandari et al., 2020) and others. The typical amount of input data that comes with these datasets has increased the pressure on the models to be able to handle larger inputs. For instance, WCEP has up to 100 documents in each document set, and 63.7 on average. As such, the standard transformers used to develop successful SDS models such as BART (Lewis et al., 2020) and PEGASUS (Zhang et al., 2020a) have proved inadequate for MDS due to their limited maximum input length (in the order of 10 3 tokens) and quadratic memory complexity (Beltagy et al., 2020). In turn, this has prompted the development of long transformer models such as Longformer (Beltagy et al., 2020) (built upon BART) and BigBird (Zaheer et al., 2020) (built upon PEGASUS) which, thanks to their smart attention layers that scale linearly with the input length, have opened up the possibility of presenting the input documents at once, allowing these re-designed attention mechanisms to discover both interand intra-document relations. Document summarization, as have other language generation tasks, has often been criticized for using maximum-likelihood training objectives that may prove limitative for the eventual performance of the models (Ding and Soricut, 2017). For this reason, reinforcement learning has been employed 5113 as an alternative, to directly optimize the models over evaluation metrics and explicitly reward the quality of the model's predictions. Reinforcement learning approaches have used metrics such as ROUGE-1, ROUGE-2 and ROUGE-L F1 (Paulus et al., 2018), and also more contemporary scoring functions such as BERTScore (Zhang et al., 2020b) as rewards, often mixed with maximum-likelihood objectives. When applying reinforcement learning to MDS, we contend that the reward should not simply be a ROUGE score against the reference summary, since this would dismiss key characteristics of the task such as inter-document information transfer. For instance, Mao et al. (2020) have leveraged maximal marginal relevance (Carbonell and Goldstein, 1998) to mollify higher-order information redundancy between the input documents. Several other performance measures could potentially be included in the reward, such as extractive fragment coverage and density (Grusky et al., 2018) and MINT (Dreyer et al., 2021), but to the best of our knowledge they have never been utilized as, or for, training objectives. To address this gap, in this paper we propose leveraging a modified coverage reward to improve information coverage across all the documents in the input set, jointly with a principled policy gradient estimator (RELAX) and a performing long transformer model (the BART Longformer Encoder-Decoder, or BART-LED), in the hope of benefiting from the synergy between these components. In this section, we present the details of the proposed approach, including the reinforcement learning framework (Section 3.1), the multi-document coverage reward (Section 3.2), and the overall training objective (Section 3.3). Given a set of documents in input, simply noted as x , and a summary with T tokens, y = { y 1 , . . . , y T } , the predictive distribution, also known as policy in reinforcement learning, can be noted as p ( y t | y 1 , . . . , y t  1 , x ) . The policy gradient theorem (Sutton et al., 1999) states that an estimator for the gradient of the reinforcement learning risk can be expressed as:  =  r T (cid:88) t =1   log p ( y st | y s 1 , . . . , y st  1 , x ) (1) where y s 1 , . . . , y sT is a sequence sampled from the policy, r is a function that rewards its quality, and  collectively denotes all the policy's parameters. This estimator is the well-known REINFORCE (Williams, 1992) and is a baseline of reinforcement learning. At its turn, the gradient can be easily turned into a loss function to be used with automatic differentiation: LREINFORCE =  r T (cid:88) t =1 log p ( y st | y s 1 , . . . , y st  1 , x ) =  r log p ( y s ) (2) The sampled sequence in (2), y s = { y s 1 , . . . , y sT } , can be obtained with any usual sampling approach such as teacher-forcing, student-forcing, or scheduled sampling (Bengio et al., 2015). While the samples can be drawn from a standard categorical distribution, in our experiments we utilize the Gumbel-Softmax re-parameterization (Jang et al., 2017) to obtain the categorical samples from transformed samples of a Gumbel distribution. The reason for the re-parameterization is that the Gumbel-Softmax samples are needed for the RELAX estimator that we introduce in the following. For a generic sample, y st , the re-parameterization can be concisely expressed as: y st = argmax ( z t ) z t  Gumbel-Softmax ( p t ,  ) (3) where z t is a Gumbel-Softmax sample of size equal to that of the vocabulary that acts as a soft prediction, p t is the probability vector over the vocabulary at slot t ,  is a temperature parameter controlling the sparsity of z t , and argmax ( z t ) returns the index of z t 's largest value. This reparameterization is provenly equivalent to directly sampling y st from Cat ( p t ) (the reader can refer to Jang et al. (2017) for details). REINFORCE is an unbiased estimator of the theoretical gradient, but it typically suffers from a high variance which can affect the convergence and effectiveness of training. To curb its high variance, techniques based on control variates and the 5114 subtraction of simple baselines have been proposed and even applied to summarization (Rennie et al., 2017; Paulus et al., 2018). However, our early experiments showed that these approaches were not promising for the given task. In addition, some of these estimators introduce a bias, i.e. a mean difference with respect to the theoretical gradient. More recently, the RELAX gradient estimator has been shown to empirically outperform REINFORCE, thanks to its ability to reduce the variance while remaining unbiased (Grathwohl et al., 2018). The corresponding RELAX loss can be expressed as: LRELAX =  [ r  c  ( z )] log p ( y s ) + c  ( z )  c  ( z ) (4) In (4), c  ( z ) is a control variate of parameters  which is expected to correlate tightly with the reward to reduce the variance, and term c  ( z )  c  ( z ) ensures that the overall gradient remains an unbiased estimator of the theoretical gradient. Variable z = { z 1 , . . . , z T } denotes the sequence of the Gumbel-Softmax samples, while variable  z denotes the sequence of samples from a Gumbel-Softmax distribution conditioned on the observed values of y s . Operationally, z t is sampled first, unconditionally, then y st is derived with the argmax, and finally  z t is sampled from a suitably conditioned Gumbel-Softmax distribution; details can be found in Grathwohl et al. (2018), Appendix B Categorical. Overall, the RELAX estimator is both unbiased and low-variance. The control variate in our experiments is a simple two-layer feed-forward network that is constructed to correlate with the ROUGE scoring function. We obtain this by feeding the concatenation of the soft predictions, z (or, in turn,  z ), and the reference summary, y , as input to the control variate. This allows the model to learn to score the soft predictions and their targets in a way that mimics the ROUGE prediction-reference score. In detail, the architecture consists of two fully-connected linear layers, each followed by a ReLU linear activation function, and a final sigmoid activation function that normalizes the output of the last layer. Eventually, the output of the sigmoid is averaged to produce the control variate. 1 1 We release our code to permit complete reproducibility of our experiments: https://github. The design of an effective reward is another key aspect of a reinforcement learning objective. In our work, we have aimed to design an overall reward that could simultaneously remain faithful to: a) the reference summary, to ensure adequate generation performance, and b) the input documents, to cover as many important details as possible, and hopefully, support generalization. Relying solely on the reference summaries, given the large input size, does not seem to promise sufficient guidance, and our experiments have confirmed that. To implement the reward, we have chosen to use ROUGE-L F1 for the references and a multi-document coverage score for the input documents that we describe hereafter. Several quantitative measures of coverage exist in the literature, and have found ample use in describing the properties of summarization datasets and the performance of models. For our work, we have adopted the extractive fragment coverage (EFC) of Grusky et al. (2018). The EFC measures the percentage of words in a summary that are part of extractive fragments within an input document, which are simply multi-word phrases shared between the input document and the summary. It is a simple precision-type measurement that looks at how much of the prediction is in the input document. Noting an individual document as D , a summary as y and an extractive fragment as f , the EFC can be expressed as: EF C ( y, D ) = 1 | y | (cid:88) f F ( y,D ) | f | (5) where the |  | operator is used to denote length. To promote an even improvement in coverage across the input documents, we propose a multi-document extension of the EFC that reaches its highest value when the coverage across the input documents is evenly distributed. Let us note the input document set here as D , and the EFC coverage vector over the document set as cov ( y, D ) . We also note the sample mean of a vector x as  ( x ) , the sample standard deviation as  ( x ) , and their ratio (the inverse co-efficient of variation) as c  1 v ( x ) . This allows us to compute a normalized coverage score for a summary, c  1 v ( cov ( y, D )) , which takes larger values the more the scores are uniform across the document set. In addition, inspired by Kryscinski et al. (2018), we define a reward that pits the normalized coverage score of the prediction, y s , against that of 5115 the reference, y : r cov = c  1 v ( cov ( y s , D ))  c  1 v ( cov ( y, D )) c  1 v ( cov ( y s , D )) (6) Eventually, to ensure that short summaries are not unfairly rewarded with high coverage scores, we normalize the reward by the length ratio of the prediction and the reference:  r cov = r cov | y s | | y | (7) Overall, the  r cov reward regards a prediction as good if it enjoys high average coverage of the input documents, the coverage is evenly distributed, and the prediction is of sufficient length. The reference summary acts as a baseline, making the reward additive if the prediction outperforms the reference, and subtractive if otherwise. Since ROUGE-L F1 and the coverage reward are not necessarily up to scale, to obtain the final reward, r , we perform a convex combination with a scaling coefficient,  : r = ROUGE-L F1 ( y s , y ) +   r cov (8) 3.3 Overall Training Objective As training strategy, we first train the model with the negative log-likelihood and choose the best model with a criterion based on the validation performance. After that, the model is fine-tuned with the reinforcement learning objective. In many past works, the reinforcement learning objective has been used mixed with the NLL for stability (Paulus et al., 2018; Li et al., 2019; Parnell et al., 2021). However, we assume that the model has already warmed up to the training data during its NLL pretraining stage, and only use either LREINFORCE (2) or LRELAX (4) for fine-tuning. To prevent excessive drifting from the NLL pre-trained model, we limit the fine-tuning to a few (  1 , 000 ) shots and a relatively low learning rate ( 3  10  6 ). We have carried out multiple experiments over two MDS datasets in the news domain: Multi-News (Fabbri et al., 2019) and Wikipedia Current Events Portal (WCEP) (Gholipour Ghalandari et al., 2020). For WCEP, we specifically use the WCEP-100 version, which exclusively limits the number of articles within a document set to 100. We have chosen these datasets as they cover an ample spread of summary lengths and numbers of input documents, with Multi-News having longer reference summaries on average. Appendix A.2 reports the datasets' main statistics as presented in the original papers (Fabbri et al., 2019; Gholipour Ghalandari et al., 2020) 2 . Like most previous works, we use the F1 variants of the ROUGE-N scores 3 (Lin, 2004) for performance evaluation. In our use of ROUGE, we choose not to stem the predictions and the references during scoring. Since we use the ROUGE-L F1 score in our reward, to avoid circularity we also include METEOR 4 (Lavie and Agarwal, 2007) in the performance evaluation. Differently from our ROUGE implementation, METEOR uses stemming, synonyms, and other paraphrastic matching in the n -gram matching stage. In a recent study, both ROUGE and METEOR have displayed high correlation with a number of desirable summarization properties such as coherence, consistency, fluency, and relevance (Fabbri et al., 2021). We have implemented our approach on top of BART-LED (Beltagy et al., 2020). We utilize the generous maximum encoding length (16384 tokens) of this long-input transformer, by concatenating all the documents in a document set to form a single input to the model. The individual documents are separated by an [END] token, and the input is truncated to the maximum length. For every experiment, we report the average of three independently-initialized training runs. For each result, we have also run a nonparametric bootstrap test for statistical significance, and highlighted the results that are significantly different from the baseline. In the reward, the  hyperparameter has been set to 1.0 with a validation described in Appendix A.3. All other hyperparameters are described in Appendix A.1. Multi-News . Table 1 compares the results over the Multi-News test set for the baseline, our proposed approaches and previous work from the literature. We first note that our BART-LED model has performed as a strong baseline, with its results being comparable to those of BART-Long (Pasunuru et al., 2021), which is based on the same BART Longformer architecture. In detail, BART-Long has reported a higher ROUGE-1 score, our baseline has reported a higher ROUGE-L score, and both have reported similar ROUGE-2 scores. Therefore, we regard our performance as comparable on the whole, with the differences most likely due to different hyperparameters. Amongst our results, the models fine-tuned with REINFORCE have achieved worse results than the baseline. This is evidence that a vanilla implementation of the policy gradient is not necessarily better than a standard NLL objective. Conversely, the models fine-tuned with RELAX have surpassed both the NLL baseline and virtually all the previous work. The best results have been achieved with the inclusion of the coverage term, with an improvement of +0 . 36 ROUGE-2 pp over the NLL baseline and a marked improvement of +0 . 92 METEOR pp. In addition, both results have reported a p -value < 0 . 01 . These results give evidence to both the improved performance provided by the RELAX gradient estimator and the usefulness of the coverage term. In Appendix B, we also provide a qualitative example which shows that the increase in METEOR score is most likely given by the positive impact of the coverage term, which has allowed the model to retrieve relevant phrases from the input documents. WCEP . Table 2 shows the results over the WCEP test set. The trend is similar to that over Multi-News, but the improvements with the proposed models have been even more pronounced. In the first place, the NLL baseline has set a very strong performance compared to the previous work, showing the full potential of a long-input model such as the Longformer for MDS. As for Multi-News, the best results have been achieved with the RELAX gradient estimator, with improvements of 5117 up to +1 . 32 ROUGE-1 pp and +3 . 17 METEOR pp over the NLL baseline. The inclusion of the coverage term with RELAX has not been able to increase the ROUGE scores, but has increased METEOR by +1 . 64 pp. Again, we attribute this to the model's improved coverage of the input documents, which leads to an increased number of matches under METEOR's more relaxed matching scheme. A qualitative example is discussed in Appendix B. 5 Analysis In this section, we present a more detailed analysis of the impact of the coverage term, the few-shot fine-tuning, and the RELAX gradient estimator using the Multi-News validation set as reference. For a further insight into the coverage reward, we also include an analysis of its trajectory during training. All the selected hyperparameters are listed in Appendix A.1. Our rationale for including a coverage term in the reward is to ensure coverage of the input documents beyond what can be driven by the reference summaries alone. We note that this may or may not translate into an improvement of the evaluation metrics, but it seems to add intrinsic value to the summaries nevertheless. For this reason, we further analyze the impact of the coverage term hereafter. Figure 1 shows the average EFC coverage (5) for the documents in the input sets, indexed by the document position in the set (first, second etc). The figure shows that the inclusion of the coverage term with RELAX has led to a marked increase of the coverage, and almost evenly distributed across all the documents in the input set. In particular, the document in the last position has achieved the largest coverage improvement. In turn, Figure 2 shows the average ROUGE score for the documents in the input sets, obtained by averaging the ROUGE-1, ROUGE-2, and ROUGE-L scores computed between the predicted summary and the document (NB: not the reference summary). The figure shows that the improvements in ROUGE score across the document set are similar to those in EFC coverage, rather evenly distributed, and with an improvement of over +4 pp for the document in the last position. This is further evidence that the normalized coverage reward (7) is able to drive the model towards predictions that cover the input set more uniformly. To explore the behavior of the few-shot fine-tuning, we compare the validation-set performance on Multi-News with varying number of training examples, from 10 to 2000 . The model's configuration is the best, with RELAX and the coverage term in the reward. Table 3 shows that the performance is the highest with 1000 examples, and starts to drop beyond this number. This is an important observation, as it shows that the reinforcement learning objective may lead to undesirable parameterizations beyond a point, and that the number of fine-tuning samples has to be treated as a hyperparameter. The RELAX gradient estimator introduces two new hyperparameters: the temperature parameter,  , and the control variate, c  . Hereafter, we discuss their impact and design. Temperature parameter . The RELAX gradient estimator uses a temperature parameter,  , in the Gumbel-Softmax sampling (3). This parameter is maintained in log scale for convenience and is learnable alongside all other parameters; yet, its initial value can have a significant impact on the final model. To explore its behavior, Figure 3 shows the trajectory of parameter log  over 1000 Multi-News training steps for different initializations (0.25, 0.5 and 1.0). The trajectories show that, irrespective of its initial value, log  converges to a stable value within approximately 400 training steps. For the initializations at 0.25 and 1.0, within the first 200-300 training steps log  drifts significantly (   0 . 25 units) from its initial value. Conversely, with the intermediate initialization at 0.5, the value remains substantially stable over the whole trajectory. Since limiting drift during fine-tuning is generally desirable, we have initialized log  to 0.5 in all experiments. Control variate size . Many different architectures could be used for the control variate, but given our choice described in Section 3.1, the main parameter is the feed-forward layers' hidden size. To explore its impact, Table 4 shows the average values of the ROUGE score and the coverage score over the Multi-News validation set with different hidden sizes (128, 256, and 512). The ROUGE score is computed between the prediction and the reference and is the average of ROUGE-1/2/L, while the coverage score is the average EFC of all the input documents. The values in Table 4 show that, the larger the control variate, the more Figure 3: Trajectory of the log(  ) temperature parameter over 1000 Multi-News training steps for different initializations. the model is able to increase the coverage score. However, the average ROUGE score drops beyond a size of 256. We speculate that this behavior is due to the larger scale of the coverage reward, as by providing more capacity to the network, we allow the control variate to increasingly correlate with the multi-document coverage reward rather than the ROUGE reward. To strike a satisfactory trade-off, we have therefore chosen 256 as the hidden size for all experiments with Multi-News, and carried out an equivalent selection for WCEP. In a reinforcement learning framework, it could be useful to monitor the value of the reward over the training steps. Typically, the reward should exhibit an upward trajectory, since the reward should tend to increase as the model learns to make better predictions. In our case, we look to explore the impact of our coverage reward on the coverage distribution over the input documents. In particular, we want to verify whether the coverage reward is able to promote predictions that cover the input documents more evenly, which should translate into a decreased standard deviation. To this aim, Figure 4 shows a plot of the standard deviation of the cov-5119 erage scores (EFC) across the input document set against the training step. The trajectories show that both REINFORCE and RELAX have been able to decrease the standard deviation of the predictions to approximately 0 . 05 units from initial values of 0 . 08  0 . 09 . The drop in standard deviation occurs quite quickly during training, coinciding with the improvement in the reward value of the predictions. Comparing REINFORCE with RELAX also shows that RELAX has been able to achieve lower standard deviation values throughout the training, with the exception of the very start. In this paper, we have proposed fine-tuning a multi-document summarization model with a reward that balances the use of the reference summaries with the coverage of the input documents within a reinforcement learning framework. The rationale for the proposed reward is that the reference summaries alone may not be sufficient for an effective fine-tuning of the model in the presence of very large inputs such as those typical of MDS datasets. Another key component of the proposed approach is the use of a modern gradient estimator of the policy gradient, RELAX. The experimental results over two news-based MDS datasets, Multi-News and WCEP, have shown that the proposed approach has been able to achieve a marked improvement of ROUGE and METEOR scores compared to its NLL-pretrained baseline, and prove competitive against most existing approaches. In addition, the proposed approach has been able to increase the coverage of the input documents, and evenly across the entire document set. As future work, we aim to explore ways to prevent or mollify the model's drift with larger number of training steps, and explore alternative architectures and configurations for the control variate of the RELAX estimator.
abstain abstain abstain objective method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective method objective other objective objective other other other other other other other other other other method other abstain objective method abstain other abstain other abstain abstain other abstain abstain abstain abstain abstain abstain other abstain abstain abstain method abstain abstain abstain abstain other abstain method abstain abstain method method abstain method abstain abstain abstain method method method method method abstain method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain objective