News editorials argue about political issues in order to challenge or reinforce the stance of readers with different ideologies. Previous research has investigated such persuasive effects for argumentative content . In contrast, this paper studies how important the style of news editorials is to achieve persuasion. To this end, we first compare contentand style-oriented classifiers on editorials from the liberal NYTimes with ideology-specific effect annotations. We find that conservative readers are resistant to NYTimes style, but on liberals, style even has more impact than content. Focusing on liberals, we then cluster the leads, bodies, and endings of editorials, in order to learn about writing style patterns of effective argumentation. The interaction between the author and the intended reader of an argumentative text is encoded in the linguistic choices of the author and their persuasive effect on the reader (Halmari and Virtanen, 2005). News editorials, in particular, aim to challenge or to reinforce the stance of readers towards controversial political issues, depending on the readers' ideology (El Baff et al., 2018). To affect readers, they often start with an enticing lead paragraph and end their argument with a punch (Rich, 2015). Existing research has studied the persuasive effect of argumentative content and structure (Zhang et al., 2016; Wachsmuth et al., 2016) or combinations of content and style (Wang et al., 2017; Persing and Ng, 2017). In addition, some works indicate that different types of content affect readers with different personalities (Lukin et al., 2017) and beliefs (Durmus and Cardie, 2018). However, it remains unexplored so far what stylistic choices in argumentation actually affect which readers. We expect such choices to be key to generating effective argumentation (Wachsmuth et al., 2018). This paper analyzes the persuasive effect of style in news editorial argumentation on readers with different political ideologies (conservative vs. liberal). We model style with widely-used features capturing argumentativeness (Somasundaran et al., 2007), psychological meaning (Tausczik and Pennebaker, 2010), and similar (Section 3). Based on the NYTimes editorial corpus of El Baff et al. (2018) with ideology-specific effect annotations (Section 4), we compare style-oriented with content-oriented classifiers for persuasive effect (Section 5). 1 While the general performance of effect prediction seems somewhat limited on the corpus, our experiments yield important results: Conservative readers seem largely unaffected by the style of the (liberal) NYTimes, matching the intuition that content is what dominates opposing ideologies. On the other hand, the style features predict the persuasive effect on liberal readers even better than the content features  while being complementary. That is, style matters as soon as ideology matches. Knowing about the specific structure of news editorials, we finally obtain common stylistic choices in their leads, bodies, and endings through clustering. From these, we derive writing style patterns that challenge or reinforce the stance of (liberal) readers of (liberal) news editorials, giving insights into what makes argumentation effective. Compared to other argumentative genres (Stede and Schneider, 2018), news editorials use many rhetorical means to achieve a persuasive effect on readers (van Dijk, 1995). Computational research has dealt with news editorials for retrieving opinions (Yu and Hatzivassiloglou, 2003; Bal, 2009), mining arguments (Al-Khatib et al., 2017), and 1 For reproducibility, the code of our experiments can be found here: https://github.com/webis-de/ acl20-editorials-style-persuasive-effect Feature Base Overview Reference Linguistic inquiry and word count Psychological meaningfulness in percentile Pennebaker et al. (2015) NRC emotional and sentiment lexicon Count of emotions (e,g. sad , etc.) and polarity words Mohammad and Turney (2013) Webis Argumentative Discourse Units Count of each evidence type (e.g., statistics ) Al-Khatib et al. (2017) MPQA Arguing Lexicon Count of 17 types of arguing (e.g., assessments ) Somasundaran et al. (2007) MPQA Subjectivity Classifier Count of subjective and objective sentences Riloff and Wiebe (2003) Table 1: Summary of the style feature types in our dataset. analyzing their properties (Bal and Dizier, 2010; Scheffler and Stede, 2016). While Al-Khatib et al. (2016) modeled the structure underlying editorial argumentation, we use the corpus of El Baff et al. (2018) meant to study the persuasive effects of editorials depending on the readers' political ideology. Halmari and Virtanen (2005) state that four aspects affect persuasion in editorials: linguistic choices, prior beliefs of readers, prior beliefs and behaviors of authors, and the effect of the text. Persuasive effectiveness reflects the rhetorical quality of argumentation (Wachsmuth et al., 2017). To assess effectiveness, Zhang et al. (2016) modeled the flow of content in debates, and Wachsmuth et al. (2016) the argumentative structure of student essays. Others combined different features for these genres (Persing and Ng, 2015). The impact of content selection relates to the notion of framing (Ajjour et al., 2019) and is well-studied in theory (van Eemeren, 2015). As Wang et al. (2017), however, we hypothesize that content and style achieve persuasion jointly. We target argumentative style here primarily, and we analyze its impact on liberal and conservative readers. In related work, Lukin et al. (2017) found that emotional and rational arguments affect people with different personalities, and Durmus and Cardie (2018) take into account the religious and political ideology of debate portal participants. In followup work, Longpre et al. (2019) observed that style is more important for decided listeners. Unlike them, we focus on the stylistic choices made in well-planned argumentative texts. The lead paragraphs and the ending of an editorial have special importance (Rich, 2015). Hynds (1990) analyzes how leads and endings changed over time, whereas Moznette and Rarick (1968) examined the readability of an editorial based on them. To our knowledge, however, no one investigated their importance computationally so far. In this paper, we close this gap by analyzing what style of leads and endings is particularly effective compared to the editorial's body. To model style, we need to abstract from the content of a news editorial. This section outlines the feature types that we employ for this purpose. Most of them have been widely used in the literature. Table 1 summarizes all features. LIWC Psychological word usage is reflected in the Linguistic Inquiry and Word Count (Tausczik and Pennebaker, 2010). LIWC is a lexicon-based text analysis that assigns words to psychologically meaningful categories (Tausczik and Pennebaker, 2010). We use the LIWC version of Pennebaker et al. (2015), which contains 15 dimensions listed in the following with examples. (1) Language metrics : words per sentence, long words. (2) Function words : pronouns, auxiliaries. (3) Other grammar : common verbs, comparisons. (4) Affect words : positive and negative emotion. (5) Social word : family, friends. (6) Cognitive processes : discrepancies, certainty. (7) Perceptual processes : feeling, seeing. (8) Biological processes : body, health. (9) Core drives and needs : power, reward focus. (10) Time orientation . (11) Relativity . (12) Personal concerns . (13) Informal speech . (14) Punctuation . (15) Summary variables . The last dimension (15) contains four variables, each of which is derived from various LIWC dimensions: (a) Analytical thinking (Pennebaker et al., 2014): The degree to which people use narrative language (low score), or more logical and formal language (high score). (b) Clout (Kacewicz et al., 2014): The relative social status, confidence, and leadership displaced in a text. (c) Authenticity (Newman et al., 2003): The degree to which people reveal themselves authentically. (d) Emotional tone (Cohn et al., 2004): Negative emotions, for scores lower than 50, and positive emotions otherwise. NRC Emotion&Sentiment To represent the mood of editorials, we use the NRC lexicon of Mohammad and Turney (2013). NRC contains a set of English words and their associations with (1) emotions such as anger , disgust , and fear as well as (2) negative and positive sentiment polarities. Webis ADUs To identify argumentative units in editorials that present evidence, we use the pre-trained evidence classifier of Al-Khatib et al. (2017). For each editorial, we identify the number of sentences that manifest anecdotal , statistical , and testimonial evidence respectively. MPQA Arguing Somasundaran et al. (2007) constructed a lexicon that includes various patterns of arguing such as assessments , doubt , authority , emphasis . For each lexicon, we have one feature that represents the count of the respective pattern in an editorial. MPQA Subjectivity We apply the subjectivity classifier provided in OpinionFinder 2.0 (Riloff and Wiebe, 2003; Wiebe and Riloff, 2005) on the editorials, in order to count the number of subjective and objective sentences there. As the basis of our analysis, we use the Webis-Editorial-Quality-18 corpus (El Baff et al., 2018). The corpus includes persuasive effect annotations of 1000 English news editorials from the liberal New York Times (NYTimes). 2 The annotations capture whether a given editorial challenges the prior stance of readers (i.e., making them rethink it, but not necessarily change it), reinforces their stance (i.e., helping them argue better about the discussed topic), or is ineffective for them. Each editorial has been annotated by six annotators: three with liberal and three with conservative ideology. To evaluate an editorial's persuasive effect on liberals, we computed the majority vote of their annotations for the editorial (and, similarly, for conservatives). We ended up with 979 editorials with effect labels for liberals and conservatives, because we found 21 duplicate editorials with the same content but different IDs (for these, we use the majority vote across all duplicates). The corpus does not have predefined evaluation datasets. To mimic real-life scenarios, we chronologically split it into a training set (oldest 80%) and a test set (newest 20%). Table 2 shows the distribution of ideology-specific effects in the datasets. 2 For copyright reasons, the corpus provides only annotations for IDs of editorials. The actual texts of these editorials come from the NYTimes Annotated Corpus (Sandhaus, 2008). To assess the impact of news editorial style on readers, we employ our style-based features on the task of predicting an editorial's persuasive effect: Given either of the two ideologies (liberal or conserva-tive), predict for each editorial whether it is challenging , reinforcing , or ineffective . We developed separate prediction models for the effect on liberals and conservatives, respectively. For each style feature type and for their combinations, we trained one SVM model with a linear kernel on the training set using scikit-learn (Pe-dregosa et al., 2011). Given the dataset split mentioned above (train-ing set 80%, test set 20%), we tuned the SVM's cost hyperparameter using grid search with 5-fold cross-validation on the training set. Since the distribution of effect labels is highly skewed, we set the hyperparameter class_weight to balanced. We then trained the best model on the whole training set and evaluated it on the test set. For comparison, we also built models for standard content features (lemma 1to 3-grams), and we consider the random baseline that picks an effect class by chance. For both ideologies, Table 3 reports the macro-and micro F 1 -scores for the style features, their best-performing combination, 3 the content features, and the best combination of content and style. 4 We computed significance using Wilcoxon's test to reveal differences between each two approaches among best style , content , best content+style , and baseline . 5 We obtained the means of F 1 -scores used in the significance tests by conducting five-fold cross-validation on the test set, using the same SVM hyperparameters as above. 3 Best style liberals: LIWC, MPQA Subjectivity . Best style conservatives: NRC Emotion&Sentiment, Webis ADUs 4 Content+style liberals: LIWC, MPQA Arguing, MPQA Subjectivity, Content . Conservatives: MPQA Arguing, Content 5 A non-parametric test was needed, because a normal distribution was not given. In general, the results indicate that the persuasive effect seems hard to predict on the given corpus. Still, we observe that the style features play a notable role in predicting the effect of editorials on liberals . They achieve a significantly better macro F 1 -score of 0.43 when combined with content compared to 0.36 when using content alone, at p < 0.05. On the other hand, the F 1 -scores of content (macro 0.37, micro 0.38) and style (both 0.36) in predicting the effect on conservatives , are insignificantly different even from the baseline (0.33, 0.34). These results suggest that style is important as soon as the ideology of a reader matches the one of the news portal (at least, this holds for liberal ideol-ogy), but not if it mismatches (here, conservative). Observing that the style of NYTimes editorials affects liberal readers, we seek to learn what patterns of writing style makes their argumentation effective. To this end, we (1) abstract each discourse part of an editorial (lead, body, ending) into a style label using cluster analysis and (2) identify sequential patterns of style labels that are specific to challenging, ineffective, and reinforcing editorials. Clustering Styles of Discourse Parts Given the importance of specific discourse parts of editorials (Rich, 2015), we split each editorial into lead, body, and ending. For each part, we separately perform three steps on the training set of the given corpus: 6 6 The corpus of Sandhaus (2008) contains lead and paragraph annotations. The lead spans either the first two paragraphs (994 editorials), the first three (5), or the first only (1). We consider the last paragraph as the ending in all cases. 1. Extract the style features from Section 3. 2. Perform a cluster analysis on the style features using cosine k -means. k is determined with the elbow method on the inertia of the clusters. 3. Derive cluster labels from the most discriminating features across clusters: For each cluster, we determine those 23 values (e.g., high tone, low authenticity) whose combination suffices to significantly distinguish a cluster from others. With high to very low , we mean here a feature has significantly higher or lower scores compared to other clusters. 7 Table 4 shows the distribution of lead, body, and ending clusters over challenging, ineffective, and reinforcing editorials. For each discourse part, the most discriminating feature is tone , followed by authenticity . The former combines positive (higher scores) and neg-7 For each feature (e.g., tone ), we measured significance using Anova (in case of homogeneity and normality) or Kruskal (otherwise). In the case of p < 0.05, we conducted posthoc analysis (independent t -test in case of normality, MannWhitney otherwise) with Bonferroni correction for each cluster pair, and we calculated the effect size r . Based on the effect size values, we deduced the labels of each cluster and the relative differences between them ( high to very low ). ative (lower scores) emotional tones (Cohn et al., 2004). The latter indicates the degree to which people authentically reveal themselves; the higher the score, the more personal, humble, or vulnerable the writer is (Newman et al., 2003). In Table 4, we observe, for example, that the lead of challenging editorials over-proportionally often shows low authenticity, or that bodies with positive tone but low authenticity tend to be ineffective. Identification of Style Patterns From Table 4, we determine the (maximum) two labels for each discourse part that are most specific to each of the three persuasive effect classes. From these, we build all possible lead-body-ending sequences, as visualized in Figure 1. According to a  -square test, the distributions of these sequences differ significantly at p < 0.05. They reveal the following patterns of NYTimes editorials for liberal readers:  Challenging editorials often begin with a polar emotional tone, followed by a negative tone. They tend to have low authenticity (i.e., not humble/personal) in the whole discourse (see Figure 2 for an example). Ineffective editorials over-proportionally often start with authenticity and dull tone. They then tend to diffuse in different directions and to have a short ending paragraph. Reinforcing editorials tend to start and end with a negative tone. They often avoid relativ-toneauthentic. While these insights are naturally still vague to some extent and require more analysis in follow-up research, they show a first way of capturing the style of editorial argumentation. This paper analyzes the importance of news editorials style in achieving persuasive effects on readers with different political ideologies. We find evidence that style has a significant influence on how a (liberal) editorial affects a (liberal) reader. Inspired by the theory of the high importance of the lead and ending in writing editorials (Rich, 2015), we also reveal common effective and ineffective style sequences (lead-body-ending) statistically. Our findings help to understand how effective argumentation works in the political sphere of editorial argumentation  and how to generate such argumentation. In related work, El Baff et al. (2019) revealed the impact of style features on generating pathosand logos-oriented short argumentative texts based on the rhetorical strategies discussed by Wachsmuth et al. (2018). With the findings of this paper, we go beyond, defining the basis of a style-dependent generation model for more sophisticated argumentation, as found in news editorials.
abstain abstain objective objective result method abstain abstain abstain abstain abstain abstain method method method method result abstain abstain result abstain other other other method other other other other other abstain method other other method other other objective method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain method method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain method abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain result result result method abstain result