how was labeling done?
The annotation instructions were: “Given a topic and its description on Wikipedia, mark if this is a topic that people are likely to argue about.”. After that, annotations were normalized to controversiality scores on an integer scale of 0 - 10
With the growing interest in social applications of Natural Language Processing and Computational Argumentation, a natural question is how controversial a given concept is. Prior works relied on Wikipedia's metadata and on content analysis of the articles pertaining to a concept in question. Here we show that the immediate textual context of a concept is strongly indicative of this property, and, using simple and language-independent machine-learning tools, we leverage this observation to achieve state-of-the-art results in controversiality prediction. In addition, we analyze and make available a new dataset of concepts labeled for controversiality. It is significantly larger than existing datasets, and grades concepts on a 0-10 scale, rather than treating controversiality as a binary label.
Indicating that a web page is controversial, or disputed - for example, in a search result - facilitates an educated consumption of the information therein, suggesting the content may not represent the “full picture”. Here, we consider the problem of estimating the level of controversiality associated with a given Wikipedia concept (title). We demonstrate that the textual contexts in which the concept is referenced can be leveraged to facilitate this. The definition of which concepts are controversial is controversial by itself; an accurate definition of this elusive notion attracted the attention of researchers from various fields, see for example some recent attempts in BIBREF0, BIBREF1, BIBREF2. Most people would agree, for example, that Global warming is a controversial concept, whereas Summer is not. However, the concept Pollution may be seen as neutral by some, yet controversial by others, who associate it with environmental debates. In other words, different people may have different opinions, potentially driven by different contexts salient in their mind. Yet, as reported in the sequel, an appreciable level of agreement can be reached, even without explicit context. Focusing here on Wikipedia concepts, we adopt as an initial “ground truth” the titles listed on the Wikipedia list of controversial issues, which is curated based on so-called “edit wars”. We then manually annotate a set of Wikipedia titles which are locked for editing, and evaluate our system on this much larger and more challenging dataset. To estimate the level of controversy associated with a Wikipedia concept, we propose to simply examine the words in the sentences in which the concept is referenced. Because a concept can often be found in multiple contexts, the estimation can be seen as reflecting the “general opinion” about it in the corpus. This contrasts previous works, which consider this a binary problem, and employ a complex combination of features extracted from Wikipedia's article contents and inter-references, and more extensively – from the rich edit history thereof.