In the context of fake news, bias, and propaganda, we study two important but relatively under-explored problems: ( i ) trustworthiness estimation (on a 3-point scale) and ( ii ) political ideology detection (left/right bias on a 7-point scale) of entire news outlets, as opposed to evaluating individual articles. In particular, we propose a multi-task ordinal regression framework that models the two problems jointly. This is motivated by the observation that hyper-partisanship is often linked to low trustworthiness, e.g., appealing to emotions rather than sticking to the facts, while center media tend to be generally more impartial and trustworthy. We further use several auxiliary tasks, modeling centrality, hyper-partisanship, as well as left-vs.-right bias on a coarse-grained scale. The evaluation results show sizable performance gains by the joint models over models that target the problems in isolation. Recent years have seen the rise of social media, which has enabled people to virtually share information with a large number of users without regulation or quality control. On the bright side, this has given an opportunity for anyone to become a content creator, and has also enabled a much faster information dissemination. However, it has also opened the door for malicious users to spread disinformation and misinformation much faster, enabling them to easily reach audience at a scale that was never possible before. In some cases, this involved building sophisticated profiles for individuals based on a combination of psychological characteristics, meta-data, demographics, and location, and then micro-targeting them with personalized fake news with the aim of achieving some political or financial gains (Lazer et al., 2018; Vosoughi et al., 2018). A number of fact-checking initiatives have been launched so far, both manual and automatic, but the whole enterprise remains in a state of crisis: by the time a claim is finally fact-checked, it could have reached millions of users, and the harm caused could hardly be undone. An arguably more promising direction is to focus on fact-checking entire news outlets, which can be done in advance. Then, we could fact-check the news before they were even written: by checking how trustworthy the outlets that published them are. Knowing the reliability of a medium is important not only when fact-checking a claim (Popat et al., 2017; Nguyen et al., 2018), but also when solving article-level tasks such as fake news and click-bait detection (Brill, 2001; Finberg et al., 2002; Hardalov et al., 2016; Karadzhov et al., 2017; De Sarkar et al., 2018; Pan et al., 2018; Perez-Rosas et al., 2018) Political ideology (or left/right bias) is a related characteristic, e.g., extreme left/right media tend to be propagandistic, while center media are more factual, and thus generally more trustworthy. This connection can be clearly seen in Figure 1. Despite the connection between factuality and bias, previous research has addressed them as independent tasks, even when the underlying dataset had annotations for both (Baly et al., 2018). In contrast, here we solve them jointly. Our contributions can be summarized as follows:  We study an under-explored but arguably important problem: predicting the factuality of reporting of news media. Moreover, unlike previous work, we do this jointly with the task of predicting political bias. As factuality and bias are naturally defined on an ordinal scale (factuality: from low to high , and bias: from extreme-left to extreme-right ), we address them as ordinal regression. Using multi-task ordinal regression is novel for these tasks, and it is also an under-explored direction in machine learning in general. We design a variety of auxiliary subtasks from the bias labels: modeling centrality, hyper-partisanship, as well as left-vs.-right bias on a coarse-grained scale. Factuality of Reporting Previous work has modeled the factuality of reporting at the medium level by checking the general stance of the target medium with respect to known manually fact-checked claims, without access to gold labels about the overall medium-level factuality of reporting (Mukherjee and Weikum, 2015; Popat et al., 2016, 2017, 2018). The trustworthiness of Web sources has also been studied from a Data Analytics perspective, e.g., Dong et al. (2015) proposed that a trustworthy source is one that contains very few false claims. In social media, there has been research targeting the user, e.g., finding malicious users (Mihaylov and Nakov, 2016; Mihaylova et al., 2018; Mihaylov et al., 2018), sockpuppets (Maity et al., 2017), Internet water army (Chen et al., 2013), and seminar users (Darwish et al., 2017). Unlike the above work, here we study source reliability as a task in its own right, using manual gold annotations specific for the task and assigned by independent fact-checking journalists. Moreover, we address the problem as one of ordinal regression on a three-point scale, and we solve it jointly with political ideology prediction in a multi-task learning setup, using several auxiliary tasks. Predicting Political Ideology In previous work, political ideology, also known as media bias, was used as a feature for fake news detection (Horne et al., 2018a). It has also been the target of classification, e.g., Horne et al. (2018b) predicted whether an article is biased ( political or bias ) vs. unbiased. Similarly, Potthast et al. (2018) classi-fied the bias in a target article as ( i ) left vs. right vs. mainstream, or as ( ii ) hyper-partisan vs. mainstream. Left-vs-right bias classification at the article level was also explored by Kulkarni et al. (2018), who modeled both the textual and the URL contents of the target article. There has been also work targeting bias at the phrase or the sentence level (Iyyer et al., 2014), focusing on political speeches (Sim et al., 2013) or legislative documents (Gerrish and Blei, 2011), or targeting users in Twitter (Preotiuc-Pietro et al., 2017). Another line of related work focuses on propaganda, which can be seen as a form of extreme bias (Rashkin et al., 2017; Barron-Cedeno et al., 2019a,b). See also a recent position paper (Pitoura et al., 2018) and an overview paper on bias on the Web (Baeza-Yates, 2018). Unlike the above work, here we focus on predicting the political ideology of news media outlets. In our previous work (Baly et al., 2018), we did target the political bias of entire news outlets, as opposed to working at the article level (we also modeled factuality of reporting, but as a separate task without trying multi-task learning). In addition to the text of the articles published by the target news medium, we used features extracted from its corresponding Wikipedia page and Twitter pro-file, as well as analysis of its URL structure and traffic information about it from Alexa rank. In the present work, we use a similar set of features, but we treat the problem as one of ordinal regression. Moreover, we model the political ideology and the factuality of reporting jointly in a multitask learning setup, using several auxiliary tasks. Multitask Ordinal Regression Ordinal regression is well-studied and is commonly used for text classification on an ordinal scale, e.g., for sentiment analysis on a 5-point scale (He et al., 2016; Rosenthal et al., 2017a). However, multi-task ordinal regression remains an understudied problem. Yu et al. (2006) proposed a Bayesian framework for collaborative ordinal regression, and demonstrated that modeling multiple ordinal regression tasks outperforms single-task models. Walecki et al. (2016) were interested in jointly predicting facial action units and their intensity level. They argued that, due to the high number of classes, modeling these tasks independently would be inefficient. Thus, they proposed the copula ordinal regression model for multi-task learning and demonstrated that it can outperform various single-task setups. We use this model in our experiments below. Balikas et al. (2017) used multi-task ordinal regression for the task of fine-grained sentiment analysis. In particular, they introduced an auxiliary coarse-grained task on a 3-point scale, and demonstrated that it can improve the results for sentiment analysis on the original 5-point scale. Inspired by this, below we experiment with different granularity for political bias; however, we explore a larger space of possible auxiliary tasks. Copula Ordinal Regression We use the Copula Ordinal Regression (COR) model, which was originally proposed by Walecki et al. (2016) to estimate the intensities of facial action units (AUs). The model uses copula functions and conditional random fields (CRFs) to approximates the learning of the joint probability distribution function (PDF) of the facial AUs (random variables), using the bivariate joint distributions capturing dependencies between AU pairs. It was motivated by the fact that ( i ) many facial AUs co-exist with different levels of intensity, ( ii ) some AUs co-occur more often than others, and ( iii ) some AUs depend on the intensity of other units. We can draw an analogy between modeling facial AUs and modeling news media, where each medium expresses a particular bias (political ideology) and can also be associated with a particular level of factuality. Therefore, bias and factuality can be analogous to the facial AUs in (Walecki et al., 2016), and represent two aspects of news reporting, each being modeled on a multi-point ordinal scale. In particular, we model bias on a 7-point scale ( extreme-left , left , center-left , center , center-right , right , and extreme-right ), and factuality on a 3-point scale ( low , mixed , and high ). In our case, we train the COR model to predict the joint PDF between political bias and factuality of reporting. This could potentially work well given the inherent inter-dependency between the two tasks as we have seen on Figure 1. Auxiliary Tasks We use a variety of auxiliary tasks, derived from the bias labels. This includes converting the 7-point scale to ( i ) 5-point and 3-point scales, similarly to (Balikas et al., 2017), and to ( ii ) a 2-point scale in two ways to model extreme partisanship, and centrality. Here is the list of the auxiliary tasks we use with precise defini-tion of the label mappings:  Bias5-way: Predict bias on a 5-pt scale; 1: extreme-left , 2: left , 3: { center-left, center, center-right } , 4: right , and 5: extreme-right . Bias3-way: Predict bias on a 3-pt scale; 1: { extreme-left, left } , 2: { center-left, center, center-right } , and 3: { right, extreme-right } . Bias-extreme: Predict extreme vs. nonextreme partisanship on a 2-pt scale; 1: { extreme-left, extreme-right } , 2: { left, center-left, center, center-right, right } . Bias-center: Predict center vs. non-center political ideology on a 2-pt scale, ignoring polarity: 1: { extreme-left, left, right, extreme-right } , 2: { center-left, center, center-right } . Features We used the features from (Baly et al., 2018) 1 . We gathered a sample of articles from the target medium, and we calculated features such as POS tags, linguistic cues, sentiment scores, complexity, morality, as well as embeddings. We also used the Wikipedia page of the medium (if any) to generate document embedding. Then, we collected metadata from the medium's Twitter account (if any), e.g., whether is is verified, number of followers, whether the URL in the Twitter page matches the one of the medium. Finally, we added Web-based features that ( i ) model the orthographic structure of the medium's URL address, and ( ii ) analyze the Web-traffic information about the medium's website, as found in Alexa rank. 2 4 Experiments and Evaluation Data We used the MBFC dataset (Baly et al., 2018) that has 1,066 news media manually annotated for factuality (3-pt scale: high , mixed , low ) and political bias (7-pt scale: from extreme-left to extreme-right ). This dataset was annotated by volunteers using a detailed methodology 3 that is designed to guarantee annotation objectivity. 1 https://github.com/ramybaly/ News-Media-Reliability 2 https://www.alexa.com/siteinfo 3 For details, see https://mediabiasfactcheck. Furthermore, readers can provide their own feedback on existing annotations, and in case of a large discrepancy, annotation is adjusted after a thor-ough review. Therefore, we believe the annotation quality is good enough to experiment with. We noticed that 117 media had low factuality because they publish satire and pseudo-science , neither of which has a political perspective. Since we are interested in modeling the relation between factuality and bias, we excluded those websites, thus ending up with 949 news media. Some examples from this dataset are shown in Table 1 with both factuality and bias labels, in addition to their corresponding Twitter handles and Wikipedia pages. Overall, 64% of the media in our dataset have Wikipedia pages, and 65% have Twitter accounts. Table 2 further provides detailed statistics about the label distribution in the MBFC dataset. Experimental Setup We used the implementation 4 of the Copula Ordinal Regression (COR) model as described in (Walecki et al., 2016). In our experiments, we used 5-fold cross-validation, where for each fold we split the training dataset into a training part and a validation part, and we used the latter to fine-tune the model's hyper-parameters, optimizing for Mean Absolute Error (MAE). MAE is an appropriate evaluation measure given the ordinal nature of the tasks. These hyper-parameters include the copula function ( Gumbel vs. Frank ), the marginal distribution ( normal vs. sigmoid ), the number of training iterations, the optimizer ( gradient descent , BFGS ), and the connection density of the CRFs. We report both MAE and MAEM , which is a variant of MAE that is more robust to class imbalance. See (Bac-cianella et al., 2009; Rosenthal et al., 2017b) for more details about MAEM vs. MAE. We compare the results to two baselines: ( i ) majority class, and ( ii ) single-task ordinal regression. Results and Discussion Table 3 shows the evaluation results for the COR model when trained to jointly model the main task ( shown in the columns ) using combinations of auxiliary tasks ( shown in the rows ). We can see that the single-task ordinal regression model performs much better than the majority class baseline based on both evaluation measures. We can further see that the performance on the main task improves when jointly modeling several auxiliary tasks. This improvement depends on the auxiliary tasks in use. For factuality prediction, it turns out that the combination of bias-center + bias-extreme yields the best overall MAE of 0.481. This makes sense and aligns well with the intuition that knowing whether a medium is centric or hyper-partisan is important to predict the factuality of its reporting. For instance, a news medium without a political ideology tends to be more trustworthy compared to an extremely biased one, regardless of their polarity (left or right), as we should expect based on the data distribution shown in Figure 1 above. For bias prediction (at a 7-point left-to-right scale), a joint model that uses political bias at different levels of granularity (5-point and 3-point) as auxiliary tasks yields the best overall MAE of 1.479. This means that jointly modeling bias with the same information at coarser levels of granularity, i.e., adding 3-point and 5-point as auxiliary tasks, reduces the number of gross mistakes. E.g., predicting extreme-left instead of extreme-right , since the model is encouraged by the auxiliary tasks to learn the correct polarity, regardless of its intensity. We can see that factuality is not very useful as an auxiliary task by itself (MAE=1.584 and MAEM =1.695). In other words, a medium with low factuality could be extremely biased to either the right or to the left. Therefore, relying on factuality alone to predict bias might introduce severe errors, e.g., confusing extreme-left with extreme-right, thus leading to higher MAE scores. This can be remedied by adding factuality to the mix of other auxiliary tasks to model the main task (7-point bias prediction). The results of these experiments, shown in parentheses in Table 3, indicate that adding factuality to any combination of auxiliary tasks consistently yields lower MAE scores. In particular, modeling the combination of factuality + bias5-way + bias3-way yields the best results (MAE=1.475 and MAEM =1.623). This result indicates that factuality provides complementary information that can help predict bias. We ran a two-tailed t-test for statistical significance, which is suitable for an evaluation measure such as MAE, to confirm the improvements that were introduced by the multi-task setup. We found that the best models (shown in bold in Table 3) outperformed both the corresponding majority class baselines with a p-value  0.001, and the corresponding single-task ordinal regression baselines with a p-value  0.02. Finally, we compared the above results to our previous work (Baly et al., 2018) by independently training a Support Vector Machine (SVM) classi-fier for each task, using the same features. The resulting MAE was 0.450 for factuality and 1.184 for bias prediction, which is slightly better then our results (yet, very comparable for factual-ity). However, our goal here is to emphasize the advantages of modeling the two tasks jointly. We have presented a multi-task ordinal regression framework for jointly predicting trustworthiness and political ideology of news media sources, using several auxiliary tasks, e.g., based on a coarser-grained scales or modeling extreme partisanship. Overall, we have observed sizable performance gains in terms of reduced MAE by the multi-task ordinal regression models over single-task models for each of the two individual tasks. In future work, we want to try more auxiliary tasks, and to experiment with other languages. We further plan to go beyond left vs. right , which is not universal and can exhibit regional specificity (Tavits and Letki, 2009), and to model other kinds of biases, e.g., eurosceptic vs. europhile , nationalist vs. globalist , islamist vs. secular , etc. This research is part of the Tanbih project, 5 which aims to limit the effect of fake news, propaganda and media bias by making users aware of what they are reading. The project is developed in collaboration between the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) and the Qatar Computing Research Institute (QCRI), HBKU.
objective objective abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain objective objective method method objective method objective other other other method objective other other other other other other other method abstain method method method other other other other other other method other other objective method abstain abstain method abstain method method method method abstain method abstain abstain abstain method method method method method method abstain other abstain method method method abstain method abstain method method abstain abstain method abstain method abstain method method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain method method method method method method result method abstain other other