A release note is a technical document that describes the latest changes to a software product and is crucial in open source software development. However, it still remains challenging to generate release notes automatically. In this paper, we present a new dataset called RNSum, which contains approximately 82,000 English release notes and the associated commit messages derived from the online repositories in GitHub. Then, we propose classwise extractive-then-abstractive/abstractive summarization approaches to this task, which can employ a modern transformer-based seq2seq network like BART and can be applied to various repositories without specific constraints. The experimental results on the RNSum dataset show that the proposed methods can generate less noisy release notes at higher coverage than the baselines. We also observe that there is a significant gap in the coverage of essential information when compared to human references. Our dataset and the code are publicly available. Recently, there has been considerable interest in applying natural language processing (NLP) techniques to support software development (Iyer et al., 2016; Yin and Neubig, 2018; Panthaplackel et al., 2020). One such task involves the automatic generation of release notes . A release note is a technical document that describes the latest changes to a software product, which is necessary for software developers to adjust their codes accurately for using the updated software. Since release notes are time-consuming to write manually, several studies have been done to explore automatic release note generation. Moreno et al. (2014) proposed ARENA, Figure 1: An example data in RNSum; this example is derived from the release of tag v2.6.4 in https: //github.com/vuejs/vue . an automatic release notes generator, which first extracts and summarizes the changes in the source code and then integrates them with the information provided by version-control and issue-tracking systems. Pokorn (2020) developed Glyph, which classifies commit messages into predefined release-note categories (e.g., Features, Bug Fixes) using pre-trained word embeddings and produces the categorized commit messages as the final release note. Despite the progress reported in these previous studies, usable release note generators are far from realization. We attribute this difficulty mainly to 8718 two problems. First, the existing resources for automatic release note generation are scarce; for example, Glyph was trained on only 5,000 commit messages, which is too little for obtaining a sufficiently generalized model. Second, the existing methods have limited applicability; for example, ARENA requires an issue tracker hosted on Jira, thus preventing it from being used for most GitHub repositories. Also, Glyph's predefined release-note categories do not include deprecations and removals, which are often indispensable in release notes. To alleviate the above problems, we introduce RNSum, a new large-scale dataset for automatic release note generation via commit logs summarization. An example data in RNSum is shown in Figure 1. RNSum consists of approximately 82,000 release notes derived from online repositories on GitHub. The contents of each release note are further categorized into four release-note classes: (1) Features, (2) Improvements, (3) Bug Fixes, and (4) Deprecations+ (Deprecations, Removals, and Breaking Changes). The release notes are associated with the commit messages that were used by human maintainers to write the release notes. The difficulty of this task is that there is no explicit alignment between each commit message and the release note categories. For example, in Figure 1, the first commit message (chore: make documentation clearer (#9450)) is not reflected in the release notes. In contrast, the second commit message (fix: empty scoped slot should return undefined fix #9452) is reflected as the third release note in the Bug Fixes class. We propose two approaches to this task: Classwise Extractive-then-Abstractive Summarization (CEAS) and Classwise Abstractive Summarization (CAS) models, which learn to produce categorized release notes given unlabeled commit messages in extractive and abstractive manners, respectively. The two proposed models can leverage modern transformer-based sequence-to-sequence (seq2seq) architectures (e.g., BART (Lewis et al., 2020)) and can be used for various repositories without any special constraints. We evaluate the proposed models and the previous systems on the RNSum dataset and report that our approaches generate less noisy release notes at higher coverage than the baselines given only unlabeled commit messages. We also perform human evaluations carefully to assess how well the systems could generate release notes in terms of quality (precision) and coverage (recall), revealing CM RN Dataset Text Class Text Class Size Mauczka et al. (2015)   967 Levin and Yehudai (2017)   1,151 Safdari (2018)   3,377 RNSum (ours)    81,996 Table 1: Comparison of RNSum with the existing datasets on commit logs. that there still remains a significant gap in the coverage when compared to human references. Our dataset and the source code are publicly available. 1 2 Task Formulation Here, we define the automatic release note generation task. The input is a set of commit messages (sentences), x = { x 1 , . . . , x n } . Given the input commit messages x , our goal is to generate labeled release notes y c for each predefined release-note class c  C . Each labeled release note is a collection of sentences, i.e., y c = y c, 1 , y c, 2 , . . . . According to Moreno et al. (2014), the major contents of most release notes can be categorized into the following classes: Fixed Bugs, New Features, New Code Components (CC), Modified CC, etc. Based on their observations, we define the release-note classes C comprising the following four categories: Features (F), Improvements (I), Bug Fixes (B), and Deprecations+ (D; Deprecations, Removals, and Breaking Changes). 2 Our classes do not include Refactoring, Document Changes, and Library Updates because most of the maintainers on GitHub omitted these changes in their release notes. Our dataset can be interpreted as a collection of quintuples, namely { x , y F , y I , y B , y D } . For simplicity, we also use ( x , y ) instead of the quintuple representation. It is worth noting that the labeled release notes y c can be empty. For example, in Figure 1, the software update is related to improvements and bug fixes. Thus, the release note contains only the Improvements and Bug Fixes classes, and y F = y D =  . Release Note Generation Automatic release note generation has been studied by several research groups. Moreno et al. (2014) proposed ARENA, which transforms the extracted source-code changes into the corresponding natural language release notes. However, ARENA relies on a versioning system and an issue tracker hosted on Jira, which makes it difficult or even impossible to use in a variety of software projects, especially those hosted on GitHub. Klepper et al. (2016) proposed a semi-automatic algorithm to generate the release notes depending on expected types of readers, e.g., team members, customers, and users. However, no experiments were reported in their work. Recently, a publicly available release note generator, Glyph (Pokorn, 2020), was developed. Glyph is a simple learning-based model that classifies each input commit message into one of five labels: Features, Bug Fixes, Improvements, Nonfunctional, and Other. These categorized commit messages are then used as the final release notes. The Glyph model was trained on 5,000 commit logs using Facebook's fastText framework (Joulin et al., 2017). We summarize the comparison of our dataset with the three data sources used in Glyph in Table 1. These existing datasets annotate only the commit message with the release note classes, making it difficult to use for release note generation. Also, the data size is quite small. To address the limitations of these three studies, we built a new large-scale dataset called RNSum, which contains approximately 82,000 release notes with commit messages. We reviewed the release notes carefully and redefined the four classes. We also propose classwise summarization methods to automatic release note generation, which can be applied to all English repositories on GitHub. Classwise Summarization There are several reported studies that use class information for summarization. Cao et al. (2017) and Yang et al. (2018) proposed using text categories to improve text summarization. Liang et al. (2019) proposed a clinical note extractive summarization system that generates summaries based on specific disease names. In 1 https://github.com/nlab-mpg/RNSum-Da taset . For licensing reasons, RNSum does not contain textual content of the release notes and the commit messages but only their URLs. To enable users to obtain the contents easily, we provide scripts using the GitHub API. 2 The Improvements class includes improvements to the existing features instead of the addition of new features. contrast to these studies, we developed classwise summarization methods for release note generation, which we confirm through experiments to be more effective than the baselines. We collected the release notes and their associated commit messages from several repositories on GitHub using the GitHub API. 3 Repositories First, we selected all public repositories that did not fork any repositories. A repository that did not fork means a repository that was not copied from others. Then, we filtered out repositories with less than 50 stars, assuming that repositories with many stars tend to contain high-quality release notes, which are suitable for learning reliable release note generation. These filtering steps resulted in 337,048 repositories as of March 2021. Release Notes with Classes We listed the past releases for each repository. For each of the four predefined classes, we manually created a vocabulary with up to 30 entries. For example, the vocabulary for the Improvements class contained terms such as improvements, enhancements, and op-timizations. We show the vocabularies used in this work in the Appendix. Then, for each release, we searched for the presence of terms in the vocabularies over the entire body text (including the subti-tles). We retained the release notes in which at least one class-relevant term was detected in the body text. We removed the repositories where only a single release note class appeared throughout themselves. Commit Messages On GitHub, the release notes are NOT tied to their corresponding commit messages. Therefore, we synchronized the release notes and commit messages using version tags (strings) and heuristic matching rules. Specifically, we first listed the version tags (e.g., v3.7.1 , v2.6.0 ) of the release notes in a repository. Then, the tags with beta versions, such as rc, alpha, and beta, were removed, and the tags were sorted chronologically. Next, considering all adjacent tag pairs, we retained only those that satisfied the heuristic matching rules. The heuristic matching rules focused on only the numerical parts of the version tags (e.g., v3.7.1  371) and compared 3 https://api.github.com 8720 # repositories 7,216 # release notes (RN) 81,996 Avg. the number of digits and the magnitude of the number. 4 For example, given a chronologically sorted version tags, v3.6.3.1  v3.6.4  v2.6.1  v3.7.0  v3.8.0 , only one tag pair, namely v3.7.0  v3.8.0 , can be retained. Then, for each retained tag pair, ( v t  1 , v t ) , we compared the two versions using the GitHub API and collected the list of commit messages for the version tag (or release note) v t . Postprocessing We filtered out release notes that were too complex, with more than 50 sentences and more than 250 commit messages, because it was computationally difficult to handle such large data. Finally, we obtained a total of 7,216 repositories and 81,996 release notes. We investigate the statistics of the RNsum dataset. Table 2 summarizes the results. Our dataset consists of 81,996 release notes from 7,216 repositories in total. The average number of release note sentences and commit messages per release note is 3.3 and 14.9, respectively. The average number of tokens in release notes and commit messages are 63.3 4 The details of the heuristic rules are described in the Appendix. and 260.4, respectively. 5 The number of unique word types (i.e., vocabulary size) is 833,984, which is significantly large because many project-specific terms such as class and method names are detected. We plot each data point in RNSum in Figure 2, where each point is represented by a two-dimensional vector of the number of tokens in the release notes and the commit messages. There is a correlation between the release notes and the corresponding commit messages regarding the number of tokens. Also, RNSum contains a wide variety of data of diverse sizes. We also examine the word overlap rate of the commit messages against the release notes. We remove special symbols such as URL, hash values, and issue numbers by using the spaCy POS tagger. 6 The resulting overlap rate is 56.7%, indicating that extractive approaches (e.g., Glyph), which simply classify commit messages into a fixed set of predefined classes, has a limitation to achieving higher recall. 7 The result also indicates that information outside the commit messages (e.g., pull requests, issues associated with the commit messages) may improve the performance further, which we leave left for future work. In the end, we examine the distributions of release note classes. There is an obvious class imbalance: Bug Fixes accounts for 60.0%, while Deprecations+ accounts for only 4.2%. This class imbalance problem makes the task more challenging. Automatic release note generation can be viewed as a task of summarizing commit messages x into the labeled release notes y c . In this paper, we introduce Classwise Extractive-then-Abstractive Summarization (CEAS) and Classwise Abstractive Summarization (CAS) models, which we instantiate by modern transformer-based sequence-to-sequence (seq2seq) networks and can be universally used in various repositories without any special constraints. 5 We used spaCy to tokenize the release notes and the commit messages. The Classwise Extractive-then-Abstractive Summarization (CEAS) model consists of two neural modules: a classifier F and a generator G . First, CEAS uses F to classify each commit message into five release-note classes: Features, Improvements, Bug Fixes, Deprecations+, and Other. Then, commit messages classified as the same class are concatenated to form a single document. The commit messages classified as Other are discarded. Then, CEAS applies G to the four labeled documents independently and generates release notes for each class. In this task, the direct correspondences between commit messages and release notes are not known. Therefore, to train the classifier F , we assign pseudo labels to each input commit message using the first ten characters of each commit message. The detail of assigning pseudo labels is described in the Appendix. If pseudo labeling generates commit messages whose class does not appear in the gold release notes, we omit such examples in training. For example, in Figure 1, if the pseudo labeling generates commit messages of Features, the commit messages are discarded because the class Features does not appear in the gold release notes. We model the Classwise Abstractive Summarization (CAS) approach by two different methods. The first model, which we call CAS-Single , consists of a single seq2seq network and generates a single long release note text given a concatenation of input commit messages. The output text can be divided into classwise segments based on special class-specific endpoint symbols, like <Features> and </Features>. In training, we concatenate all the gold labeled release notes y into one long document by inserting the classwise endpoint symbols and train the network to generate the target text. The second method, which we call CAS-Multi , consists of four different seq2seq networks G c , each of which corresponds to one of the release-note classes (Features, Improvements, etc.). We train each network G c to generate the corresponding release notes y c independently given a concatenation of the input commit messages. We divided the RNSum dataset into training, validation, and test splits, each containing 74K, 4K, and 4K examples. To avoid data leaks, examples derived from the same repository did not belong to multiple splits. We also removed the training examples with release note text (after concatenation) of longer than 500 tokens to shorten the training time. Since a release note y c can consist of multiple sentences, we concatenate the sentences by inserting spaces and represent the release note as one long text in evaluation. Following the conventional summarization literature, we employ ROUGE (Lin, 2004) as the automatic evaluation metric. We also employ BLEU (Papineni et al., 2002) to evaluate the fluency of generated release notes. Specifically, we compute ROUGE-L (F1), BLEU-3, and BLEU-4 scores. 8 We skip a test example if the reference text is empty. It is also important for the system not to generate release notes when the reference release note is empty (i.e., y c =  ). To evaluate such ability, we also compute Specificity , i.e., TN TN+FP , where positive means that the generated release note is NOT empty. As baseline systems for comparison, we develop Glyph (Pokorn, 2020) and a clustering-based commit message classifier. These baselines are extractive summarization methods because these methods generate release notes by just classifying each input commit message into a fixed set of release-note classes. In contrast, CEAS and CAS employ seq2seq generators to transform input commit messages into novel texts. Glyph Glyph is a publicly available commit message classifier, which groups each input commit message into the following five classes: Features, Improvements, Bug Fixes, Non-functional, and Other. The text classification model relies simply on pretrained word embeddings in fastText (Joulin et al., 2017). Since the Non-functional class is not included in our task, we exclude the commit messages classified as Other or Non-functional from 8 We calculate the BLEU and ROUGE scores using torch-text https://github.com/pytorch/text and HuggingFace framework (Lhoest et al., 2021), respectively. Clustering We also develop a clustering-based classifier for this task. This method classifies each input commit message based on the closest cluster centroid using Euclidean distance. First, we train a Continuous Bag-of-Words (CBOW) model (Mikolov et al., 2013) with a window size of 5 on 10 million commit messages collected from GitHub and obtained 300-dimensional word embeddings for this domain. 9 Then, we embed each input commit message using the averaged embeddings of the first three tokens (without punctua-tions). 10 Then, we perform the K-means clustering algorithm on the commit message embeddings in the RNSum training set and obtain k ( > 4) clusters. We determine the correspondence between the cluster IDs and the release-note classes (Features, Bug Fixes, Improvements, Deprecations+) based on the best alignment m that maximizes the total BLEU scores on the RNSum training set D , i.e., m  = argmax m  k P 4 (cid:88) ( x , y ) D (cid:88) c C BLEU( f m ( c ) ( x ) , y c ) where m ( c ) denotes a cluster ID corresponding to the release-note class c , and f k ( x ) is the commit messages classified as the cluster k . In inference, input commit messages classified as the remaining k  4 clusters are removed in the output. To determine the optimal number of clusters k , we tested k  [5 , 20] and found that 11 provided the best validation score. CEAS We employ BERT (Devlin et al., 2019) and CodeBERT (Feng et al., 2020) as the commit message classifier F . CodeBERT is a bimodal pre-trained model for programming language and natural language. Specifically, we apply a multilayer perceptron to the CLS embedding of the input. When training the classifier, class-identifiable words, such as fix: and feat:, are removed because they can be too strong class indicators. We describe the detail of removing class-identifiable words in the Appendix. We employ BART (Lewis 9 We used the gensim library (Rehu  rek and Sojka, 2010). We removed words with frequencies lower than 300 occurrences. 10 We also tested the average embeddings of all tokens or all nouns. The averaging of the first three tokens consistently outperforms these two counterparts. ROUGE-L (F1) BLEU-3 BLEU-4 Specificity et al., 2020) as the generator G (or G c ). We used the HuggingFace (Wolf et al., 2020) BertTokenizer, AutoTokenizer, and BARTTokenizer for tokeniza-tion. The learning rate was set to 4e-5, and we used the AdamW (Loshchilov and Hutter, 2019) optimizer. Mini-batch size was set to 20 for the classifier and 2 for the generator. To mitigate the class imbalance problem, we also used upsampling for the infrequent classes (Features, Improvements, and Deprecations+). We used the validation set to perform early stopping with a patience of 3 epochs. CAS We employ BART as the seq2seq network. All the CAS-Single and CAS-Multi networks are initialized with the same pretrained parameters, but the parameters are untied across the models and trained independently. We used the HuggingFace (Wolf et al., 2020) BARTTokenizer for to-kenization. Mini-batch size was set to 2 for the CAS-Single network and 8 for each network in CAS-Multi. Other training settings are the same as CEAS. We show automatic evaluation results in Table 3. We also show the results on the cleaned version of the test set, where we removed URLs, hash values, and email addresses, which are significantly difficult to produce accurately. CEAS and CAS achieved ROUGE-L scores more than 10 points higher than the baselines. In particular, on the cleaned test set, the score gap between the proposed methods and the baselines 8723 CM fix createOptions validation issue (#294) * fix createOptions validation issue * Update logic 2.5.1 (#295) Class Bug Fixes PR + fix createOptions validation issue Issues https://github.com/Microsoft/vscode-azure-iot-toolkit/pull/294 Error if create options is larger then 512 bytes https://github.com/Microsoft/vscode-azure-iot-toolkit/issues/293 RN Fix deployment JSON validation issue when create options is larger than 512 bytes Table 4: An example provided for human evaluators. jumped to more than 20 points. These results indicate that CEAS and CAS are significantly effective. In addition, CEAS got a better ROUGE-L score than CAS, suggesting that combining a classifier and a generator is effective and training the classifier using pseudo labels. The high coverage of CEAS can be achieved probably because the classifier can focus on selecting relevant commit messages for each class. Moreover, CEAS(BERT) got higher scores than CEAS(CodeBERT), indicating that it is better to use BERT for tasks where the commit message is the input. CodeBERT is closer to the domain of commit messages than BERT, but we assume that this is because it was trained with relatively little natural language data. Furthermore, CAS-Multi tended to yield higher ROUGE-L than CAS-Single, suggesting that it is also effective to independently develop different abstractive summarization models for each release-note class. Although not as apparent as ROUGE-L, CAS models (CAS-Single and CAS-Multi) generated comparable or higher BLEU scores than CEAS and the baselines. CAS models were also able to achieve significantly higher Specificity scores (> 30 points) compared to the baselines. These results indicate that CAS models can generate less noisy release notes than the baselines. We hypothesize that CAS is trained a lot to remove noise from all commit messages, including Other class, which strengths the ability to deal with noise. We employed twelve human evaluators to manually assess the quality of the release notes generated by the systems and the reference release notes. The evaluators were graduate students or working professionals with at least one year of experience read-ing release notes and updating software libraries. We randomly chose 120 release notes from the test set. The allocations of the Features, Improvements, Bug Fixes, and Deprecations+ classes were 40, 25, 40, and 15, respectively. We divided the evaluation tasks into three groups of 40 questions, and each group was assigned to four different evaluators. We used a crowdsourcing platform, Yahoo! Crowdsourcing, 11 operated by Yahoo Japan Corporation for the evaluations. In the following, we explain the evaluation task and scoring measures. Evaluation Task For each evaluation task, an evaluator is given a list of input commit messages and the target release note class. The evaluator is also given supplemental information about pull requests and issues. These supplemental data were not used to train the models, but we included them because they are often helpful for accurately evaluating the release notes. We show an example of this case in Table 4. The release notes that were manually prepared by the original maintainers contained the words JSON and larger than 512 bytes, but this information cannot be found in the commit messages. To accurately evaluate the human-generated release note, pull requests and issues are required. Thus, we instructed the evaluators to check the titles, pull requests, and issues if necessary. We selected CAS-multi for its better performance than the CEAS and CAS-single in terms of the quality of generated texts. For the Deprecations+ class, we evaluated the outputs of the CAS-Multi, the Clustering model, and the human reference because the Glyph does not produce release notes for the Deprecations+ class. Scoring We employed a five-point scoring scheme for evaluating the release notes. The evaluation scores were determined based on two criteria: Percentage of necessary information (cov-erage) and percentage of unnecessary information (noise contamination). For the coverage-oriented scoring, the following guidance was used for the scoring; 5: 90% or more necessary information (NI), 4: 70% or more NI, 3: 50% or more NI, 2: 30% or more NI, and 1: less 30% of NI. For the noise-oriented scoring, the following guidance was used; 5: no unnecessary information (UI), 4: less UI, 3: slightly less UI, 2: a little UI, 1: much UI. We use Fleiss' Kappa to measure inter-annotator 11 https://crowdsourcing.yahoo.co.jp/ . Results We show the results of the human evaluations in Table 5. For all the metrics, CAS-Multi achieves the highest human evaluation scores among the automatic systems (See the column of Avg.). In particular, in the noise-oriented (or precision-oriented) metric, CAS-Multi significantly improves over the baselines and even outperforms the human references. This fact suggests that the abstractive summarization approach is effective in transforming the noisy textual representations in the commit messages. However, the CAS-Multi's performance is still lower than those of the human references, suggesting the remaining challenges in this task. We also tested the statistical significance of the results using a permutation test (Pitman, 1937). Since evaluating all possible permutations would require a considerable amount of time, we used the approximation method. Note that sampling all permutation is typically not feasible unless the dataset size is relatively small. 12 We set the number of rounds to 10,000. We applied Cov. + Noise scores to the test. Comparing CAS-Multi with the baselines, all the p-values of the permutation tests were less than 0.001, indicating that the improvements are statistically significant. We qualitatively analyzed the outputs of CAS-Multi to identify the bottlenecks of the current approach. Table 6 shows several examples where the outputs of CAS-Multi and the human references are largely different. The input commit messages are lengthy, so we only show the commit messages related to each class in this table; the entire text is shown in the Appendix. First, we found that CAS-Multi tends to produce significantly shorter release notes than human references. In the first two examples in Table 6, CAS-Multi generates only a single sentence, while the human references contain multiple sentences. This is probably due to the fact that the release note classes are significantly scarce in the training set, and the model is trained to be reluctant to generate release note text. Actually, the numbers of release notes containing the Features and Improvements classes in the training set are only 33.4% and 14.0%, respectively. We used the upsampling technique to reduce the class imbalance problem; however, upsampling cannot inherently increase the number of unique training examples. Therefore, it is necessary to explore ways to augment the training patterns inherently in the future. Second, we found that release notes are often difficult to precisely and accurately produce without supplemental information outside the commit messages. In the second example in Table 6, CAS-Multi generates just Fixed date time filter ranges, which is a simple paraphrase of the first commit message. In contrast, the human reference enriches the content by rephrasing it as Not properly working Datetime filters Today, On, Between. Moreover, the third commit message (fix excel) is enriched as Excel report: first column not for-matted, which is impossible to generate without external information. In the last example in Table 6, without background knowledge of the repository, it is impossible to detect relationships between the two commit messages and to combine them into a single sentence. 8 Conclusion In this paper, we presented a new large-scale dataset for automatic generation of release notes. The dataset comprised approximately 82k release notes from over 7k repositories on GitHub. We formulated a task to automatically generate release notes by summarizing the commit messages, which can be applied to all software development projects that use English. We confirmed the validity of the proposed classwise extractive-then-abstractive summarization (CEAS) model and the classwise abstractive summarization (CAS) model via experiments to generate less noisy release notes at higher coverage than the baselines. However, there are still gaps in the coverage performance compared to manually generated outputs; it is expected that this could be improved by including additional information such as issues and pull requests with the commit logs. References Ziqiang Cao, Wenjie Li, Sujian Li, and Furu Wei. 2017. Improving multi-document summarization via text classification. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence , AAAI'17, page 30533059.
abstain abstain objective objective abstain result abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain objective abstain objective result abstain abstain method abstain objective abstain abstain method method abstain method abstain abstain abstain other other other other other other other other other objective other other objective abstain objective other other other other other abstain other objective method abstain abstain method abstain abstain method abstain result abstain abstain method abstain method objective abstain method other abstain abstain method method result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain method abstain method abstain method objective abstain method method method method abstain method method method method method method abstain method method abstain abstain abstain abstain other method abstain method method method method abstain method method abstain method abstain method method method method abstain abstain abstain method method abstain method method method abstain method abstain abstain result result abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain other result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain other abstain other