We propose MULTIOPED 1 , an open-domain news editorial corpus that supports various tasks pertaining to the argumentation structure in news editorials, focusing on automatic perspective discovery . News editorial is a genre of persuasive text, where the argumentation structure is usually implicit . However, the arguments presented in an editorial typically cen-ter around a concise, focused thesis, which we refer to as their perspective . MULTIOPED aims at supporting the study of multiple tasks relevant to automatic perspective discovery , where a system is expected to produce a single-sentence thesis statement summarizing the arguments presented. We argue that identifying and abstracting such natural language perspectives from editorials is a crucial step toward studying the implicit argumentation structure in news editorials. We first discuss the challenges and define a few conceptual tasks towards our goal. To demonstrate the utility of MULTIOPED and the induced tasks, we study the problem of perspective summarization in a multi-task learning setting, as a case study. We show that, with the induced tasks as auxiliary tasks, we can improve the quality of the perspective summary generated. We hope that MULTIOPED will be a useful resource for future studies on argumentation in the news editorial domain. News editorial is a form of persuasive text that conveys consensus opinion on a controversial topic from the editors of a newspaper. Much like an argumentative essay, a news editorial centers around a thesis, which represents the authors' perspective on the topic. Usually, a news editorial argues in favor of the authors' stance on the topic, and is substantiated by extensive factual evidence . As news editorials function as professionally produced written discourse for conveying media attitude and guidance, they have traditionally been studied by the community as a rich resource for many argumantation-related tasks. (Wilson and Wiebe, 2003; Yu and Hatzivassiloglou, 2003; Bal and Saint-Dizier, 2009). This work targets the problem of developing computational methods to identify and comparatively analyze the authors' perspectives and supporting arguments behind news editorials. One challenge to studying the argumentation structure in news editorials is that its elements are rarely expressed explicitly (El Baff et al., 2018). For example, Figure 1 shows two news editorials holding opposite views on whether a lockdown should continue. However, neither of them present their key perspectives explicitly. Instead, the perspective is conveyed through subtle rhetoric strategies to either affirm or challenge the readers' stance from prior belief on the topic, as a study by El Baff et al. (2018) discovers. As Figure 1 shows, the statement  The lock down should stop  concisely summarizes the perspective expressed in the article on the left. We refer to such statements as  perspectives  throughout the paper. The ability to abstractively summarize the perspectives from the editorial would allow us to understand multiple topic-aligned editorials in context and reason about their inter-editorial argumentation structure. To facilitate research along the line, we collect data from THEPERSPECTIVE 2 website, and construct MULTIOPED , an open-domain English news editorial corpus that supports various tasks pertaining to the argumentation structure in news editorials, focusing on automatic perspective discovery (Chen et al., 2019). The structure of the data is shown in Figure 1. For each of the 1,397 natural language query on a different topic in our dataset, it features two (rather long) news editorials. Each editorial features a single-sentence perspective, which is abstractively summarized from the editorial by human experts. A short abstract then highlights the details in the editorial that support the perspective . The perspectives of the two editorials represents responses of opposite stances towards the query. Naturally, the structure of the dataset induces a range of important argumentation-related natural language understanding tasks. For instance, the presence of the summary perspective allows for stance classification (Hasan and Ng, 2013) with respect to the query, which arguably is more tangible than inferring the stance from the entire editorial. Another example task is the conditional generation of the perspective from the abstract/editorial, which relates to the widely studied task of argument generation (Hua and Wang, 2018; Alshomary et al., 2020). We defer the more detailed description of the induced tasks to Section 3. One key advantage of MULTIOPED that is absent from earlier datasets is that a large number of argumentation-related tasks can be studied jointly using a single high quality corpus. To demonstrate this benefit and the utility of the MULTIOPED dataset 3 along with its induced tasks, we study the problem of perspective summarization in a multitask learning setting. We employ perspective relevance and stance classifications as two auxilliary tasks to the summarization objective. Our empiri-2 https://www.theperspective.com/ perspectives/ 3 Our code and data is available at http://cogcomp. cal and human analysis on the generated summaries show that the multi-task learning setting improves the generated perspectives in terms of the argument quality and stance consistency. In summary, our contributions in this work are three-fold. First, we propose a conceptual framework for identifying and abstracting the perspectives and the corresponding argumentation structure in news editorials, and define a set of tasks necessary for achieving this goal. Second, we propose the MULTIOPED dataset, a news editorial dataset that induces multiple argumentation-related tasks. Third, we demonstrate the utility of our multi-purpose dataset and induced tasks, by using the perspective summarization task as a case study. We include the induced tasks as auxiliary objectives in multi-task learning setting, and demonstrate their effectiveness to perspective summarization. Our goal of perspective discovery follows similar definition proposed by Chen et al. (2019), and is closely related to a widely studied area of argumentation mining, i.e. identifying the argumentation structure within persuasive text (Stab and Gurevych, 2014b; Kiesel et al., 2015). However, most studies in this domain focus on extractive methods, which becomes less applicable to our study. As the arguments are usually presented in an subtle and implicit way in news editorials, we instead focus on the generation methods for the perspectives. This closely resembles the argument conclusion generation task (Alshomary et al., 2020). One key distinction here is the presense of query to provide topic guidance during the perspective generation. Compared to other conditional text generation tasks, perspective generation subjects to a few more constraints with respect to the argumentation structure. For example, the perspective must constitute the same stance (Hasan and Ng, 2013) as the editorial towards the query. On the other hand, while the editorial may cover content not directly related to the query, the generated perspective must present a relevant argument in the query's context. Such structural constraints can be studied in the format of classification problems. And being able to study such problems along side the perspective summarization task on one high-quality corpus is important in our case, as it opens up the probability of modeling the tasks jointly. We show the benefit of Dataset Source Open Domain Cross Article Abstractive ARAUCARIADB (Reed et al., 2008) News Ed. doing so by presenting a case study in section 5. As the query provides topic guidance, it allows for the study of the topic-aligned pairs of editorials which presents counter-arguments to each other. Such property is absent from notable datasets of similar purposes to ours, as shown in Table 1. ARAUCARIADB (Reed et al., 2008) is the first effort to provide large-scale annotations of dense argumentation structure within individual news editorials. Stab and Gurevych (2014a); Eckle-Kohler et al. (2015) provide resources for extractive argumentation structure in persuasive essays and news articles, respectively. Later works (Hua and Wang, 2018; Chen et al., 2019) focus on the abstractive generation or identification of arguments from web corpora. All of these datasets focus on studies of argumentation structure within individual document. Instead, our proposed dataset presents the opportunity to study the cross-document argumentation structure. Following the design principles outlined in the previous section, we propose a topic-aligned English news editorial corpus, MULTIOPED . The structure of an example instance in MULTIOPED is shown in Figure 1. To clarify our description of the dataset, we use the following notations. Let q be a query about a controversial topic. Each q in the dataset is paired with two editorials e pro and e con , that constitute supporting and opposing stances to the query q respectively. Each editorial is abstracted into and a single-sentence perspective p , which provides a high-level summarization of the key argument presented in the editorial. The premises, or relevant details to support the perspective, forms the abstract a . Naturally, the relation between these elements induces several tasks, most of which encompass similar definitions to existing argumentation-related tasks. We define and describe the tasks and their connection to our end goal of perspective discovery below. 1. Generating an Abstract : Given an editorial e , a system is expected to identify and summarize the relevant arguments into an abstract paragraph a to the context provided by the query q . This is closely related to the task of argument synthesis (El Baff et al., 2019; Hua et al., 2019). We set aside this problem in our case study in section 5, and use the abstract provided by the dataset. 2. Perspective Summarization : Given the generated abstract a and the query q , a system is expected to generate the perspective p , a concise summary of the arguments presented in a . Conceptually, this problem resembles the task of argument conclusion generation (Alshomary et al., 2020). We adopt a slightly different setting where the target topic is expressed in the form of a natural language query. editorials e 's stance towards a query q . The generated perspective p from editorial e allows us to focus on a simpler task definition of classifying the stance of the perspective to the query q (Hasan and Ng, 2013; Bar-Haim et al., 2017). 4. Assessing the Relevance of Perspective : We want to measure the validity of the perspective by assessing whether the perspective presents a relevant argument towards the query (Chen et al., 2019; Ein-Dor et al., 2020). This can be formulated as a classification problem with the query q and a perspective p as inputs, as we show in section 5. Figure 4: Relevance classification: Decide if the perspective is relevant to the query. We extract the query, editorial article pairs, abstract paragraph pairs, along with their perspective summaries from THEPERSPECTIVE 4 website. The website presents controversial topics in the form of queries. For each query, two related editorial articles with opposing views from different sources 4 https://www.theperspective.com/ perspectives/ are selected by the writers from the website. The writers create a concise one-sentence summary of each article as the response to the query, and an abstract paragraph to summarize the relevant arguments from the article. An example structure of the data is shown in Figure.1. To verify the structure from the website, and collect additional annotations, such as stance of the perspectives, we conduct a few annotation experiments with Amazon Mechanical Turk 7 . For all of our annotation experiments, we require the workers to be located in the United States, as the controversial topics covered by the website are most applicable in the U.S. context. We also require the workers to have masters qualifications (i.e. Top performers recognized by MTurk among all workers). We compensate the workers $0 . 75 , $1 . 00 and $1 . 25 per 10 queries for the implicit reference resolution, topic annotation, and stance annotation tasks respectively. The compensation rates are determined by estimating the average completion time for each annotation experiments. Example screenshots of our annotation interface and more detailed annotation guidelines can be found in Appendix B. 4.2.1 Stance Annotation In our dataset, each query is presented with two perspectives with opposite stance to the query. However, the raw data that we collected does not specify the stance of each perspective individually. We ask two expert annotators label whether each perspective is offering a supporting or opposing view with respect to its query. The two experts discuss and adjudicate their decisions. We then ask on average three crowdsource workers per instance to verify the annotations. From the annotations collected by experts, we find that 30 out of 1 , 397 queries do not constitute a clear stance. Such queries are typically "open-ended" questions which cannot be responded with a yes or no answer, i.e. why or what questions. We leave these instances unlabeled and exclude them from the next verification step. To assess the quality of stance labels created, we randomly sample 500 perspectives, and ask three MTurk workers per instance to verify stance labels. We computed the inter-rater agreement fleiss'  = 0 . 81 among workers, and the agreement between majority decision from works and the ex-pert's adjudicated annotations is cohen's  = 0 . 92 . We describe how we measure the two types of agreements respectively in Appendix A.3. Some of the perspectives in our dataset have implicit references to certain subjects in the query. For instance, for a query Is Trump Right To Criticize Mail-In Voting? , and a perspective It's far too risky for an election , the word "It" in the perspective refers to Mail-in Voting in the query. As we assume that a perspective should presents a complete, valid argument on itself, we decide to replace such implicit reference in a perspective with the correct referent in the query. For example, the corrected perspective in the previous example would become Mail-in voting is far too risky for an election . We ask one expert annotator to identify implicit references and make modifications for every perspective in the dataset. In total, 1 , 301 out of 2 , 794 perspectives are identified and corrected by the expert annotator. We ask three Turkers to verify that the modifications do not introduce any grammatical error or change the original meaning. We randomly sample 500 modified perspectives and present Turkers with the question of "Will this modification change the original meaning or introduce grammar error?". The percentage of majority answers being No is 84%. We include both changed and original versions of the perspectives in our datasets. We create 9 topic labels according to the categorization from THEPERSPECTIVE website and major news outlets. We then as ask three MTurk workers to assign one of the 9 topic labels to each query. We regard the majority answer by the Turkers as the annotation for its topic category. In cases where all three annotators choose different categories ( 43 cases out of all 1397 queries), we label it as other topics . We show the distribution of topic categories in Figure 5. The inter-agreement among three annotators for this 9-class classification task is  = 0 . 65 Figure 5: Topic distribution of the 1397 queries in MULTIOPED . MULTIOPED consists of 1,397 queries about different news topics. Each query is presented with two perspectives, two abstracts and two linked news editorials. Despite a few stale urls and invalid redirections, we manage to extract the text for 2,584 news editorials. More detailed statistics are reported in Table 2. 5 Case Study: Multi-task Learning for Perspective Summarization 5.1 Multi-Task Framework To demonstrate the benefits of modeling the induced tasks on the argumentation structure, we present a case study on the task of perspective summarization. Given a query and an abstract from the related editorial, a system is expected to produce a concise and fluent summary perspective for the editorial. In addition, the generated perspective ideally should satisfy a few structural constraints with respect to the query. For instance, the generated perspective must constitute the same stance as the editorial towards the query. Also the perspective should be relevant in the context of the query. The two requirements resemble the perspective stance and relevance classification tasks defined in section 3 respectively. Motivated by this, we study the two tasks together with perspective summarization in a multitask learning framework. We choose BART (Lewis et al., 2020) as our base summarization model. BART is a pretrained auto-regressive transformer (Vaswani et al., 2017) encoder-decoder model, that have been proven effective in conditional text generation and other NLP tasks. Model ROUGE 1 ROUGE 2 ROUGELBERTSCOREREL . % STANCE % BART 28.24 11.34 26.96 88.67 91.91 72.32 + Rel 28.35 11.51 27.12 88.69 92.98 72.68 + Stance 28.19 11.53 26.93 88.75 91.25 73.39 + Rel & Stance 29.18 11.92 27.94 88.74 94.64 74.29 Table 3: Results of our multitask perspective summarization models. We compare to BART as a baseline, and experiment with different combinations of the auxiliary tasks. We report the F 1 score under ROUGE { 1 , 2 ,L } and BERTSCORE metrics, as well as the percentage of summaries with the correct relevance and stance label, as predicted by our pretrained classification models respectively. See Appendix A for training details and hyperparameters settings. Model RANK %1 STREL . STANCEBART 2.09 49.50 77.00 70.50 + Rel 1.74 60.50 87.00 70.50 + Stance 1.78 58.50 83.00 79.50 + R & S 1.76 59.00 82.00 69.00 Table 4: Human Evaluations results. R ANK  shows a model's averaged rank judged by the raters ( 1 = best , 4 = worst ) %1 ST  represents the percentage of generated summaries from one model that are ranked the best. We start with a pretrained BART base model with 139 M parameters, and finetune the model to output the target perspective given the query and abstract concatenated as input. In addition, we put two separate linear layers over the pooled embed-dings of the last decoder layer, and predict the relevance and stance labels of the generated summary respectively. The two tasks and the perspective summarization are learned jointly, and share the underlying model parameters from BART. One obvious challenge in the setup is that we do not have access to the ground truth stance and relevance labels for the generated summaries during training. To address this, we adopt similar strategies as in knowledge distillation (Hinton et al., 2015). We first train two separate BERT (De-vlin et al., 2019) classifiers as the teacher models for stance and relevance classificaiton respectively. Due to the size limit of our dataset, we pretrain both models on the PERSPECTRUM dataset (Chen et al., 2019), which contain over 7,000 instances of training data, with similar formats and definition to our ( query, perspective ) pairs. We further fine-tune the models on our training set. When measured against our test set, the relevance and stance models achieve binary accuracy of 92% and 75% respectively. During the perspective summarization model training, we use the pre-trained BERT models for relevance and stance classification to predict labels for each generated summary. We expect the BART plus linear layers to mimic the predictions made by the two pretrained BERT models respectively. Specifically: HQ = EOS ( DBART ( EBART ( Q ))) HA = EOS ( DBART ( EBART ( A ))) We feed the query and the abstract separately through the BART encoder ( EBART ) and decoder ( DBART ). We get their hidden representations HQ and HA as the embedding of the end-of-sentence (</s>') token from the decoder. We then concatenate HQ and HA , and feed the concatenation to the two linear layers. Finally, a softmax layer is applied to get stance/relevance predictions  y rel and  y stance . Next, We feed the query and the generated summary to the two pretrained BERT classification models to get the soft stance and relevance labels y rel and y stance . We use two mean square error (MSE) loss terms to measure the discrepancy between the BART predictions and the soft labels. We combine LREL and LSTANCE with the summarization objective, LSUM , which is the negative log-likelihood loss between generated and target perspective. The auxiliary losses LREL and LSTANCE are weighted by tunable hyperparameters  1 and  2 respectively. L = LSUM +  1  LREL +  2  LSTANCE 5.2 Results 5.2.1 Automatic Evaluations Table 3 shows our evaluation results of our multitask model with different combinations of auxiliary tasks. The reported results are averaged over three trained models with different random initialization. We first evaluate the generated perspective summaries against the target perspective with ROUGE (Lin, 2004) and BERTSCORE (Zhang et al., 2020) metrics. We observe that relevance and stance auxiliary tasks both increase the ROUGE and BERTSCORE , and combining the two objectives yields the best performance under the ROUGE metrics. To empirically verify whether the perspectives generated by our multi-task model are improved in terms of the relevance and stance correctness, we again use the two pretrained BERT classifiers to measure the percentage of generated summary with correct relevance and stance label. The results potentially suggest that by mimicing the predictions made by the two pretrained classifiers, our multitask framework is able to generate summaries with higher quality along the two dimensions. We randomly sampled 100 instances of abstracts with query from the test set, and ask two human raters to judge the quality of perspectives generated by the four systems. For the four summaries generated from an abstract by the different systems, we shuffle their order and ask the raters to rank each summary by the overall quality, with four criteria considered (1) Fluency (2) Grammatical Correctness (3) Faithfulness to the arguments offered in the original abstract (4) Salience . We allow ties among different summaries. We report their averaged ranks and the number of times a system is ranked first place in Table 4. The results are the averaged scores between the two annotators, and the level of agreement between them for this 4-class ranking task is  = 0 . 35 . For each summary, we ask the raters to annotate whether it (1) represents a relevant argument to the query (2) constitutes the correct stance as the target stance label. The kappa agreement between the two Query Should trump accept democrats' gov't spending bill? raters for these two tasks are 0.54 and 0.70, respectively. We show the human evaluation results in Table 4. We observe that while both the relevance and stance auxiliary tasks improve the quality of the generated perspective, combining the two auxiliary tasks does not guarantee a better summary quality. The results on ROUGE , BERTSCORE and human evaluation suggest that the perspective summarization model learning benefits from both the relevance and stance tasks. However, we also observe that the vanilla BART present a strong baseline in both automatic and human evaluations. We list two typical cases where we observe the relevance and stance objectives improve the quality of the generated summary. For the query shown in Table 5, the BART model generates an out-of-context word shutdown, which exists in the abstract, but is not applicable in the context provided by the query. The model with relevance objective, on the other hand, generates a perspective that is coherent to the context provided. For the query shown in Table 6, the baseline BART model incorrectly produces a supporting perspective to the query, while the editorial or abstract presents the opposite stance. The model with the stance objective generates a perspective with a matching stance. While we choose relevance and stance classification as the two auxiliary tasks in this case study, there exist many other candidate tasks that might be helpful in the setting. For instance, measuring the quality (Toledo et al., 2019), or more specifically persuasiveness (Carlile et al., 2018) of the perspective might be two, amongst other, viable options. As our study assumes that the abstract is provided for each editorial, the overall performance of perspective summarization will likely drop, if we use model-generated abstract instead of ground truth as input. News editorials have been studied as a resource for studying many argumentation-related tasks. Wilson and Wiebe (2003); Yu and Hatzivassiloglou (2003) use editorials for the study on sentiments and opinions. Later works (Reed et al., 2008; Bal and Saint-Dizier, 2009; Chow, 2016) shift focus on the argumentation structure within editorials, and their persuasiveness effect (Al Khatib et al., 2016; El Baff et al., 2020). A few other recent studies have explored argument quality (El Baff et al., 2018) and generation (El Baff et al., 2019) when using editorials as a resource. Our proposed dataset and study focus on the interplay between elements of the argumentation structure presented in editorial articles. Unlike previous work, we study these elements as the abstractive instead of extractive summary from the news editorials. Most early efforts in argument generation, i.e. generating components in an argumentation structure, study rule-based synthesis methods based on argumentation theories (Reed et al., 1996; Zuker-man et al., 2000). With the recent progress in neural, sequence to sequence text generation methods (Sutskever et al., 2014), a few studies have adapted such techniques for end-to-end argument generation. (Wang and Ling, 2016; Hua and Wang, 2018; Hua et al., 2019). The task of perspective generation in this work closely relates to argument conclusion generation (Alshomary et al., 2020). Our study focuses on the setting where the target topic, or the query , is given as input to the generation model. Due to the implicit nature of the perspectives (Habernal et al., 2018), one key challenge to the task is keep the semantics of the perspective generated truthful to the abstract and editorial article. We approach this by measuring the compatibility of the perspective to the context along the dimensions of content salience (Bar-Haim et al., 2020) and stance correctness (Bar-Haim et al., 2017). Our multi-task generation approach conceptually resembles the work by Guo et al. (2018), where multiple auxiliary tasks is employed to improve the quality of the generated summary. We present MULTIOPED an open-domain news editorial corpus that induces a number of argumentation-related tasks. The proposed dataset presents a few properties that are absent from existing datasets. First, the elements in the annotation structure are presented as abstraction over the text in editorial, as such elements usually exist implicitly in editorials. Second, as the pairs of editorials are aligned by topic, and exhibit opposing stance to each other, such structure allows for studies on cross-document argumentation structure. Third, the dataset allows for the study of multiple argumentation-related tasks together. To demonstrate the power of having multiple related tasks in a single high-quality dataset, we study the problem of perspective summarization in a multi-task learning setting. Our analysis shows that modeling stance and relevance classification jointly with the summarization task improves the overall quality of the perspective generated. In future work, we hope to utilize the corpus to improve the multi-task framework for perspective summarization. As we set aside the problem of abstract generation in our case study, we would also like to identify the challenges and potential solution to the problem. We hope that MULTIOPED presents opportunities and challenges to future research in argumentation. We collected data for MULTIOPED by automatically extracting data from www.theperspective. com/perspectives . The CEO of the website, Daniel Ravner, granted us permission to extract and use their data for academic research. We further annotated the data using crowd-workers. All crowd-workers were compensated by a fair wage determined by estimating the average completing time of each annotation task. Please refer to section 4.2 for more details. The queries, abstracts, and perspectives in MULTIOPED are written by the professional writers of the website. The website aims at presenting the perspectives in each article without unnecessary subjective interpretation, but there is no guarantee that no subjectivity is involved in their content creation process. The authors would like to thank Daniel Ravner, the CEO of www.theperspective.com , for kindly granting access to data from the site for academic research. This work was supported in part by a Focused Award from Google, and a gift from Tencent.
objective abstain method abstain method objective objective result abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain objective method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective other abstain objective objective objective objective objective abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain method method abstain abstain method abstain other abstain abstain result abstain abstain abstain abstain abstain abstain abstain result method abstain abstain result result abstain abstain result method abstain abstain method abstain abstain method abstain method abstain result method abstain abstain method abstain abstain method result abstain abstain abstain abstain method abstain objective abstain abstain abstain abstain other method method abstain method method abstain method abstain result method abstain abstain abstain abstain abstain method method objective abstain result abstain abstain objective result result method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain method abstain method other other other other objective method other other other abstain method other method abstain method abstain abstain abstain abstain objective result result method method method abstain method method abstain abstain abstain abstain other other