We study the task of long-form opinion text generation, which faces at least two distinct challenges. First, existing neural generation models fall short of coherence, thus requiring efficient content planning. Second, diverse types of information are needed to guide the generator to cover both subjective and objective content. To this end, we propose DYPLOC, a generation framework that conducts dynamic planning of content while generating the output based on a novel design of mixed language models. To enrich the generation with diverse content, we further propose to use large pre-trained models to predict relevant concepts and to generate claims. We experiment with two challenging tasks on newly collected datasets: (1) argument generation with Reddit ChangeMyView, and (2) writing articles using New York Times' Opinion section. Automatic evaluation shows that our model significantly outperforms competitive comparisons. Human judges further confirm that our generations are more coherent with richer content. Opinion articles serve as an important media to convey the authors' values, beliefs, and stances on important societal issues. Automatically generating long-form opinion articles has the potential of facilitating various tasks, such as essay writing and speech drafting, and it is the focus of this work. Though opinion generation has been investigated for constructing arguments (Hua and Wang, 2018), writing reviews (Ni and McAuley, 2018), and producing emotional dialogue responses (Song et al., 2019), those outputs are relatively short. While impressive progress in generation has been achieved by using large pre-trained Transformers (Radford et al., 2019; Lewis et al., 2020a), directly adopting United_States, Intelligence knowledge, attack America was never prepared and had a bad intelligence system. First, large models still fall short of producing coherent text due to the lack of efficient content control and planning (Ko and Li, 2020; Wu et al., 2020; Tan et al., 2021). A common solution is to use concatenated phrases or semantic representations to guide the generation process (Yao et al., 2019; Harkous et al., 2020; Ribeiro et al., 2020; Goldfarb-Tarrant et al., 2020), where content planning, including both content selection and ordering, is expected to be learned by attention mechanisms. However, attentions have only achieved limited improvements. Recent work also explores training a separate planning module to produce sorted content, which is then fed into a generator (Fan et al., 2019; Hua and Wang, 2020; Goldfarb-Tarrant et al., 2020). Nonetheless, this strategy results in a disconnection between planning and realization, and the output is not guaranteed to respect the planning results (Castro Ferreira et al., 2019; Prabhumoye et al., 2020). The second challenge for opinion generation resides in the diversity of information that is needed to produce an output with consistent stances and supported by pertinent facts. Though large models memorize significant amounts of knowledge, they cannot retrieve and operate with them precisely (Lewis et al., 2020b). Due to the argumentative nature of opinion text, simply including knowledge bases (Guan et al., 2020; Zhou et al., 2020) is insufficient to uphold the desired quality, as it requires the combination of subjective claims and objective evidence as supports. To this end, we propose a novel generation framework, DYPLOC (dynamic planning of content), to conduct content selection and ordering as text is produced. 1 Concretely, given a set of unordered content items, as displayed in Figure 1, we design mixed language models, with each implemented as a sequence-to-sequence model to encode one item and the input statement. At each decoding step, our system selects which items to reflect, and predicts a word based on probabilities marginalized over all language models. Crucially, our end-to-end trained framework (1) enables the generator to access multiple content items at all times and select content based on what has been generated so far, (2) can be directly built on large pre-trained Transformers, e.g., BART (Lewis et al., 2020a), with planning and generation modules jointly trained, and (3) outputs learned content selection scores to provide an interface for system decision interpretation. Furthermore, to ensure that our framework can be applied to a broad range of generation tasks, we design content items to cover three critical elements: entities and concepts that are central to many generation applications, and claims that are building blocks for opinion text. We show an example for counter-argument generation in Figure 1. Importantly, we employ BART to predict additional relevant concepts, derived from Concept-1 Data and code are available at: xinyuhua.github. Net (Speer et al., 2017), and generate claims, as central propositions, to enrich the generated text with both objective and subjective content. For experiments, we collect two datasets: (1) posts from Reddit ChangeMyView for argument generation, and (2) articles from the New York Times Opinion section (Sandhaus, 2008) for opinion article writing. Our proposed framework outperforms competitive comparisons, such as fine-tuning BART with the same content items, based on automatic metrics of BLEU, ROUGE, and METEOR. Human assessment further confirms that our system outputs have richer content and are more coherent in both tasks. below:  We present a dynamic content planning generation framework, which is directly built on top of BART. Our design of mixed language models overcomes the lack of control by existing models that use implicit planning with attentions or hard copying. We propose content plan augmentation by automatically generating relevant concepts and claims. We construct two opinion text generation datasets with content plans that capture prominent entities and concepts. Neural Generation with Planning. Text planning is seen as a crucial step to guide the generation of high-quality, well-organized natural language text (McKeown, 1992; Reiter and Dale, 2000). Incorporating planning modules to neural text generator has attracted significant research interests (Shen et al., 2019; Moryossef et al., 2019; Puduppully et al., 2019), which proves to be especially beneficial for long-form output (Fan et al., 2019; Hua and Wang, 2019). More recently, large pre-trained Transformers have established new state-of-the-arts for a wide range of text generation tasks (Lewis et al., 2020a; Roller et al., 2020; Kale and Rastogi, 2020). But it is non-trivial to integrate planning modules into them. Existing approaches resort to decoupling planning and decoding stages (Hua and Wang, 2020; Kedzie and McKeown, 2020), which inevitably increases system complexities and potentially introduces cascading errors. We take inspiration from the retrieval-augmented generation framework (Lewis et al., 2020b), which is designed to incorporate relevant documents for VC on t en t It e m E n c od i ng ( i n pa r a ll e l ) Plan Scoring Mixed Conditional Language Model Content Item-Conditioned LM <s> Claim Generation Concept Expansion <s> <s> <s> <s> <s> <s> <s> <s> <s> <s> <s> <s> <s> <s> <s> CMV. I believe 9/11 would not have happened if Al Gore were elected President. 0.80 x 0.10 x 0.05 x 0.05 x Figure 2: Our proposed text generation framework, DYPLOC. question answering. Our adaptation uses a trainable plan scoring module to reflect content selection and ordering, which is more suitable for long text generation and offers better interpretability. Concurrent work by Zhang et al. (2021) presents a mixture-of-expert decoder to tackle knowledge-grounded generation. However, their score distribution for language models is fixed across all decoding steps, whereas ours is updated as generation progresses and can better reflect the dynamic nature of content planning. Controllable Text Generation. Another related line of research investigates the controllability of generation models (Wiseman et al., 2017), including conditioning over keywords (Keskar et al., 2019; Hua and Wang, 2020; Xu et al., 2020), syntactic structures (Casas et al., 2020; Goyal and Durrett, 2020), or semantic representations (Wen et al., 2015; Elder et al., 2018). Our work differs from all previous methods as we combine different types of content, covering both objective and subjective information, and attain fine-grained sentence-level control using a novel design of mixed conditional language models. Opinion Text Generation. Our model tackles opinion articles, which differs from traditional text generation systems that mostly concern fact-based generations (Gardent et al., 2017; Novikova et al., 2017; Puduppully et al., 2019). An extensive body of work has studied summarizing (Wang and Ling, 2016; Suhara et al., 2020; Brazinskas et al., 2020) or generating (Ni and McAuley, 2018; Li et al., 2019) reviews and building dialogue systems enhanced with emotions (Li et al., 2016; Song et al., 2019). More recently, developments are made in generating argumentative text (El Baff et al., 2019; Hidey and McKeown, 2019), which primarily focus on constructing single sentence claims on a limited number of topics. In comparison, our model can handle substantially longer output with improved quality. Task Formulation. Our opinion text generation framework takes as input a set of content items . Each content item consists of a title t , a set of entities E i 2 , such as { United States , 9/11 attacks } , and a set of core concepts C i , such as { attack , knowledge } , that are often abstract notions. Our model first expands C i by predicting additional relevant concepts C + i and optionally generates a pertinent claim m i , and then outputs the final text with multiple sentences as y = { y t } , to faithfully reflect the content items with a coherent structure. An overview of our system is illustrated in Figure 2. Below we first describe the content item augmentation methods ( 3.1), followed by our generator with mixed language models that condition on expanded content items ( 3.2). Concept Expansion. With limited number of entities and concepts as input, generation systems are often incapable of producing long text with rich content, resulting in hallucination (Wiseman et al., 2017; Tian et al., 2019). Therefore, from the often-abstract core concepts, we aim to predict more specific concepts that are also relevant to the given title. For instance, as displayed in Figure 1, for core concepts { make , happen } and 2 Note that i distinguishes the items. Their order is random. entities { Bill Clinton , 9/11 attacks } , we grow the input with more concrete concepts of { mistake , administration } . We thus consider a concept expansion module g (  ) , which predicts additional relevant concepts, denoted as C + i , by conditioning on the original content item: C + i = g ( t , E i , C i ) (1) While g (  ) can be any conditional predictor, our experiment shows that fine-tuned BART model performs best on our tasks, where it generates C + i word-by-word by consuming the content item. 3 Training data construction is described in  4.2. Claim Generation. As discussed in  1, opinion text generation should be controlled with consistent propositions, which cannot be effectively expressed by disconnected concepts. Therefore, we argue that natural languages are more suitable for delivering central claims, since they better encode stylistic languages, e.g., persuasion strategies. Concretely, we fine-tune another BART model by taking in the title t and the entities E i , which then produces a claim with nucleus sampling for decoding (Holtzman et al., 2020). In this work, we assume the subset of content items that can be used to generate claims is known. Possible future work includes predicting such subsets and filtering claims with quality measurement. After obtaining the augmented content items, we leverage the BART model to encode each of them as a sequence, as illustrated in Figure 2. Segmenter <s> is added to indicate the change of elements in a content item. Our encoders run over all items { x i } in parallel, from which we extract content item representations { h i } , based on the last layer's hidden states of the first token. The standard sequence-to-sequence (seq2seq) framework models output probabilities by taking a single sequence as input. It is challenging to extend seq2seq to consider multiple sequences simultaneously, and conduct content planning concurrently. Therefore, we introduce a plan scoring network , 3 We also exploited a model that uses the structure of knowledge bases, e.g., ConceptNet, for learning to expand concepts, but it yields lower precision and recall than fine-tuning BART does. d ( x i | y <t ) , which learns to dynamically select and order content based on what has been produced previously while generating the outputs. As outlined in Figure 2, our generator is informed of all content items during generation. At each decoding step t , the probabilities of output words are estimated as a weighted sum of all content item-conditioned language models as follows: p ( y t | y <t ) = (cid:88) i d ( x i | y <t ) p ( y t | y <t , x i ) (2) d ( x i | y <t ) = softmax i ( e it ) (3) where p ( y t | y <t , x i ) corresponds to the i -th language model with x i as the input. Crucially, d ( x i | y <t ) determines the importance of x i when generating token y t and thus achieves the effect of content planning. We design a two-layer feed-forward network to estimate e it : e it = W o tanh( W d [ h i ; s t ]) (4) where h i denotes the representation of content item x i , s t is the decoder state, and W o and W d are learnable parameters. Although mixed language models have been used by Lewis et al. (2020b) to include retrieved documents for question answering, their relevance scores are given by external retrieval models, whereas our plan scorer d ( x i | y <t ) is learned together with the generator. Training and Decoding. Our model is end-to-end trained with both the standard cross-entropy loss L gen over the tokens in the target generations and a separate loss L plan for learning d ( x i | y <t ) : L (  ) = L gen (  ) + L plan (  ) (5) To create labels for L plan , we leverage the correspondence between content items and target tokens, i.e., d ( x i | y <t ) is optimized to approach 1 if y i is in the sentence that derives x i , otherwise 0 . 4 Details about training data construction is in  4.2. At each decoding step, the individual language models, p ( y t | y <t , x i ) , and the distribution scores, d ( x i | y <t ) , are first calculated in parallel. We then decode each token greedily based on the mixed language models in an autoregressive way. 4 We also experimented with a training objective consisting of the generation loss only, but the performance degraded significantly. Future directions include removing the training signals for planning. We experiment with the tasks of argument generation and opinion article writing ( 4.1). Both tasks require generating multi-sentence output, and contain a substantial amount of opinions and factual content. We describe the construction of initial content items and the training data for generating expanded concepts and claims in  4.2. We present models for comparison in  4.3. Finally, we provide implementation details in  4.4. Argument Generation. We collect arguments from Reddit ChangeMyView 5 (CMV) community, an online forum that features argumentative discussions. Each thread begins with an original post (OP) stating an opinion towards a controversial topic, e.g.,  The U.S. is too big for one government . High-quality replies that counter-argue with the OP and are labeled with community endorsement are collected in our prior work (Hua and Wang, 2020), covering content posted from 2013 to 2018. In this work, we extend the data collection to 2019. Our goal is to generate the entire reply (i.e., the target) given the OP title. Statistics about the CMV dataset are listed in Table 1. We reserve the most recent 1 , 000 samples for test and another 1 , 000 for validation. Opinion Article Writing. Our second task is to generate opinion articles, as collected from the New York Times (NYT) corpus (Sandhaus, 2008). We retain articles whose taxonomy labels include Top/Opinion . To ensure that articles can be processed by our computing resource, we only keep the ones with at most 20 sentences, representing 60% of all opinion articles. As shown in Table 1, NYT outputs tend to be significantly longer and contain less claims than CMV. Similarly, we keep 1 , 000 examples each for test and validation sets. From target references, we describe how to automatically construct the input content items consisting of entities and core concepts, and how to collect training data to fine-tune BART to predict more specific concepts and additional claims. Prior work has demonstrated the benefits of incorporating knowledge bases for text generation (Clark et al., 2018; Puduppully et al., 2019; Guan et al., 2020). We 5 https://www.reddit.com/r/ changemyview/ CMV NYT # Samples 77 , 245 113 , 616 Avg. thus consider two sources of knowledge: (1) entities from Wikipedia, which are useful for modeling events and opinion targets, and (2) concept words from ConceptNet (Speer et al., 2017), that cover more related details. Note that our setup is generally applicable to other text generation tasks, as these input items can be obtained through standard NLP pipelines, as described below. Entity Linking. We first segment a reference into sentences. The ones with fewer than 5 tokens are discarded for content item construction. For the rest, we extract entity mentions using Stanford CoreNLP (Manning et al., 2014), and further include nominal noun phrases. For entity linking, we adopt CrossWiki (Spitkovsky and Chang, 2012), which can process our large-scale data within a reasonable amount of time. CrossWiki maps a mention to a list of frequently linked Wikipedia entries. We further manually verify and correct the linking results for the top 500 most frequent mentions. Concept Extraction. To identify concepts in a reference, we match the lemmatized unigrams and their part-of-speech (POS) tags against all ConceptNet entries. To create a reasonably challenging task, we only keep a subset of the matches for inclusion in the core concept set (i.e., C i ), with the rest used as C + i , to be generated by our concept expansion model. Furthermore, we conjecture that an opinion article author tends to start with high-level topics that cover more abstract topical words. We thus leverage a lexicon (Brysbaert et al., 2014) with concreteness scores, ranging from 0 (abstract) to 5 (concrete), for over 40 k English words. We keep concepts that are verbs or have a concreteness score lower than 3 . 0 . Word coverage of references by using core concepts and additionally with augmented concepts are 13 . 2% and 16 . 9% on CMV respectively, and similarly on NYT (Table 1). Finally, we train a concept generator with BART to produce C + i , conditional on C i , the title, and the entities. Claim Detection and Generation. Claims are indispensable for opinion articles. As described in  3.1, we aim to enrich content items with claims targeting the given entities within the title's context. To this end, we first train a claim detector by fine-tuning a BERT base (Devlin et al., 2019) sequence classifier with a dataset consisting of sentences of claims and facts . Concretely, we collect 54 , 802 claim sentences from Kialo 6 , a repository for debate arguments. We then sample 50 , 000 sentences from Wikipedia, which are treated as facts. This classifier is applied on a reference, and sentences that are labeled as claims become the target for our claim generator. We then learn a claim generator using BART, which takes in the title and the entities, and outputs the claim. We augment our training data with replies collected from 30 active subreddits related to political discussions, with details in Appendix A. In total, 80 , 566 sentences, which contain at least one entity and are labeled by our classifier as claim s, are kept to train the generator. We compare with three baselines: (1) RETRIEVAL first calculates the TF-IDF weighted bag-of-words vectors for each content item, which is then used to query the training set sentences. The one with the highest cosine similarity is picked for each query, which are then ordered by a trained Pointer-Network (Vinyals et al., 2015) as described in Gong et al. (2016). (2) SENTPLANNER (Hua and Wang, 2019) is an LSTM-based seq2seq model with a separate sentence planning decoder, where the planner selects keyphrases by using attentions and the generator reflects the selections. We treat our entities and concepts as keyphrases to feed to this model. (3) SEQ 2 SEQ is a fine-tuned BART model, whose input is the original content items without augmentation , thus does not have access to the predicted concepts and claims. Additionally, we consider a strong comparison SEQ 2 SEQFULL , by fine-tuning BART with the same augmented content items as inputs as in our model. The difference is that the content items are 6 https://www.kialo.com/ concatenated before being used as input. We implement all models using the Huggingface Transformers library (Wolf et al., 2020) with Py-Torch (Paszke et al., 2019). We use the base model for BART, which has 768 dimensional states and 6 layers for both encoder and decoder ( 140 M parameters in total). Our newly added plan scoring network only contains 1 . 2 M parameters, less than 1% of the pre-trained model. Our generation model is optimized using Adam (Kingma and Ba, 2014), with a batch size of 3 . To improve efficiency, we adopt the mixed-precision (FP16) to train each model, using one NVIDIA Titan RTX GPU card with 24GB memory. The number of content items is limited to 10 per sample, and the numbers of entities and concepts per content item are capped at 20 , respectively. We also truncate the target output to at most 200 tokens during training. Early stopping is applied over validation loss. Our model converges after being trained for 38 hours ( 19 epochs) on CMV, and 45 hours ( 15 epochs) on NYT. The best validation perplexity reaches about 6 . 1 after model convergence on both datasets. Here we report results on test sets with standard automatic metrics: BLEU (Papineni et al., 2002) measures the n-gram precision (here we consider up to bigrams); ROUGE (Lin, 2004), calculated based on n-gram recall; and METEOR (Denkowski and Lavie, 2014), which also accounts for synonyms. In Table 2, we first present the results when gold-standard concept expansion is used. Our proposed DYPLOC model achieves significantly higher performance across all metrics on both datasets. In particular, the substantial lead over SEQ 2 SEQFULL , which has access to the same content items as ours, indicates that dynamic content planning with mixed language models produces superior generations . Among comparison models, the gap between SEQ 2 SEQFULL and SEQ 2 SEQ shows the effectiveness of content item augmentation. We also observe a significant drop for baselines without using large models, highlighting the importance of pre-training. Ablation Study. To verify the effect of each element in content items, we further train ablated models by removing concepts, claims, or entities. The Argument Generation (CMV) Opinion Article Generation (NYT) BLEU-2 ROUGE-2 METEOR Len. Effect of Hard Selection of Content Items. To test the necessity of using weighted-sum marginal-ization (Eq. 2), we experiment with two comparisons with hard selections, i.e., either randomly choosing a content item, or using the one with the highest predicted plan score (greedy selection). For both cases, we set the selected content item's plan score as 1 . 0 , with the rest of the candidates having a score of 0 . 0 , to ensure the probabilities summed up to 1 . 0 . As can be seen from the bottom two rows of Table 2, not surprisingly, random selection performs much worse. We observe that its generations lack coherence and fluency, implying the effectiveness of our learnable content planner. On the other hand, using greedily selected content items obtains comparable results with DYPLOC, where a weighted sum of content items is considered. Indeed, we find that DYPLOC's plan scores are often sharp where one content item has much Data System Gram. Results with Generated Concepts. Table 3 lists generation results with our system generated concepts as expansion. While all systems yield worse results compared to using gold-standard concepts, our DYPLOC still outperforms other models by substantial margins, showing its robustness when input concepts are noisy . Yet it also suggests the importance of having more accurate and comprehensive concept expansion, which should be explored in the future work. We hire three proficient English speakers to evaluate four key aspects of the generated outputs: (1) grammaticality ; (2) coherence , measuring if the text is logical and cohesive; (3) relevance , gauging topic relatedness to the input title; and (4) content richness , assessing the specificity and whether there is enough details in the outputs. Each aspect is rated on a scale of 1 (worst) to 5 (best). In addi-Content Items We randomly select 50 samples from the test sets for both tasks, and present outputs by SEQ 2 SEQ , SEQ 2 SEQFULL , and DYPLOC in random orders. Table 4 shows that DYPLOC receives higher scores across all aspects and tasks. In particular, the considerable differences in coherence and content richness indicate that our framework yields better content organization as well as retains more useful information . Overall, our system outputs are ranked best for 44 . 7% and 45 . 9% of the time in two tasks, significantly more than the comparisons. Analysis on Argumentative Quality. In the ablation study, we find that our full model's performance is similar to the version without having claims as input. We suspect this is because claims are often paraphrased or even not directly used when delivering an argument, which cannot be captured by the automatic metrics. To better understand how claims are used for generation, we randomly select 50 examples by DYPLOC and its variant without claims, and ask the same human judges to decide whether there is a clear central argument conveyed by each generated argument on CMV. We observe that 66 . 7% of the outputs by our full model are recognized as successfully delivering arguments with consistent stances , whereas only 61 . 3% are true for the model variant without claims. This gap confirms that claim drafts can indeed promote the argumentative quality as perceived by human readers. Evaluation results on generation quality have shown the effectiveness of our mixed language models. In this section, we aim to further understand the behavior of the plan scoring network, d ( x | y <t ) , such as how it affects the usage of content items for generation. Specifically, we adopt the following procedure to construct alignment between each sentence in the output and content items: for each token y t , we establish a mapping y t (cid:55) x i if x i is the most important item for producing y t , i.e., x i = argmax x d ( x | y <t ) , and d ( x i | y <t ) > 0 . 5 . If all tokens in an entire sentence are mapped to the same x i , we consider this sentence is aligned to that content item. Based on this rule, we show sample output and corresponding alignments in Table 5. For the rest of this section, we conduct analyses based on this alignment result. We first examine whether the model learns to utilize enough content items, i.e., high coverage. Then we provide insights on whether the generation faithfully reflects the argumentative claims using entailment relation labeling by human inspection. How many content items are used by the output? Human judges have rated our model output to contain more relevant information (Table 4). We believe this can be attributed to the enhanced capacity to access and reflect the input data with dynamic content planning, as a result of mixed language models. To verify this hypothesis, we calculate the percentage of content items that are aligned to at least one output sentence. Figure 3 shows that, using our system, the coverage reaches 87 . 25% on CMV and 83 . 89% for NYT. If we replace the generated concepts with gold-standard concepts (as extracted from references) instead, the coverage exceeds 90% on both tasks. These observations indicate that our model can indeed adequately utilize the input data, with more accurate concepts further encouraging higher coverage . How are claim content items realized? Claims are the central elements for opinion text construction. As mentioned in  4.2, a subset of the content items are supplied with claim sentences. In order to examine whether they are realized as claim sentences in the outputs, we leverage the fine-tuned BERT classifier ( 4.2) to label all output sentences. 90 . 96% of the sentences that are aligned to a claim element in the input are also labeled as claim on CMV. The percentage is only 69 . 41% for NYT, though, likely because the NYT opinion articles still contain more objective information. claim input and its aligned generated sentence. We randomly sample 50 outputs from test sets, and ask four human judges to read each. For each sample, we highlight one output sentence that is aligned to a content item with claim element. The judges determine a three-way ( ENTAIL , NEUTRAL , CONTRADICTORY ) entailment relation between the input claim (premise) and the output (hypothesis). Results show that ENTAIL accounts for 49 . 06% of all instances, while only 3 . 77% are deemed CONTRADICTORY . Upon inspection, the contradictory pairs are usually disagreements with regard to implicit sentiments, e.g.,  Journalist is the most responsible for the problem  vs.  Media coverage is a good thing. . This suggests that while our conditional language model achieves reasonable semantic control in most cases, it is still not guaranteed to capture more nuanced semantics encoded in opinions and arguments. Future work includes designing representations that can better model stances in opinions as well as argumentative structures. We present a novel text generation framework that enables dynamic content planning based on mixed conditional language models. We further employ large models to augment system inputs with diverse content that covers both objective and subjective information. The experiments on two distinct opinion text generation tasks show that our proposed model compares favorably against strong comparisons based on fine-tuned BART models with the same input. Human evaluation further confirms that our model generations have richer information and better content organization. This research is supported in part by National Science Foundation through Grant IIS-1813341. We thank three anonymous reviewers for their valuable suggestions on various aspects of this work. Large models that are pre-trained on heterogeneous web data are shown to encode biases and can be potentially harmful for marginalized populations. Along with the improved controllability, we also recognize that our system might be misused to create fabricated or offensive content. We therefore advocate cautious and responsible practices in real-world deployment.
method abstain abstain objective objective method result method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective method method method method result abstain method objective method method method objective method other other other other other other other other method other other other other objective other abstain other other abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain other abstain method method abstain method abstain abstain abstain abstain abstain method abstain method method method abstain method abstain method method method method abstain method method method abstain method method abstain other abstain method abstain objective abstain method method abstain method abstain method method method method method abstain abstain abstain abstain method abstain abstain objective method method method method method objective method abstain abstain method abstain method other method method method abstain method method abstain method abstain method abstain abstain result abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain result abstain method abstain method result result abstain method abstain abstain result result result abstain abstain result method abstain abstain abstain abstain abstain abstain abstain abstain abstain method result abstain abstain abstain abstain abstain result abstain objective objective objective method other other abstain method abstain