what results did their system obtain?
The best model achieved a 0.9320 F-score
This paper addresses the important problem of discerning hateful content in social media. We propose a detection scheme that is an ensemble of Recurrent Neural Network (RNN) classifiers, and it incorporates various features associated with user-related information, such as the users' tendency towards racism or sexism. These data are fed as input to the above classifiers along with the word frequency vectors derived from the textual content. Our approach has been evaluated on a publicly available corpus of 16k tweets, and the results demonstrate its effectiveness in comparison to existing state of the art solutions. More specifically, our scheme can successfully distinguish racism and sexism messages from normal text, and achieve higher classification quality than current state-of-the-art algorithms.
Social media is a very popular way for people to express their opinions publicly and to interact with others online. In aggregation, social media can provide a reflection of public sentiment on various events. Unfortunately, many users engaging online, either on social media, forums or blogs, will often have the risk of being targeted or harassed via abusive language, which may severely impact their online experience and the community in general. The existence of social networking services creates the need for detecting user-generated hateful messages prior to publication. All published text that is used to express hatred towards some particular group with the intention to humiliate its members is considered a hateful message. Although hate speech is protected under the free speech provisions in the United States, there are other countries, such as Canada, France, United Kingdom, and Germany, where there are laws prohibiting it as being promoting violence or social disorder. Social media services such as Facebook and Twitter have been criticized for not having done enough to prohibit the use of their services for attacking people belonging to some specific race, minority etc. BIBREF0 . They have announced though that they would seek to battle against racism and xenophobia BIBREF1 . Nevertheless, the current solutions deployed by them have attempted to address the problem with manual effort, relying on users to report offensive comments BIBREF2 . This not only requires a huge effort by human annotators, but it also has the risk of applying discrimination under subjective judgment. Moreover, a non-automated task by human annotators would have strong impact on system response times, since a computer-based solution can accomplish this task much faster than humans. The massive rise in the user-generated content in the above social media services, with manual filtering not being scalable, highlights the need for automating the process of on-line hate-speech detection. Despite the fact that the majority of the solutions for automated detection of offensive text rely on Natural Language Processing (NLP) approaches, there is lately a tendency towards employing pure machine learning techniques like neural networks for that task. NLP approaches have the drawback of being complex, and to a large extent dependent on the language used in the text. This provides a strong motivation for employing alternative machine learning models for the classification task. Moreover, the majority of the existing automated approaches depend on using pre-trained vectors (e.g. Glove, Word2Vec) as word embeddings to achieve good performance from the classification model. That makes the detection of hatred content unfeasible in cases where users have deliberately obfuscated their offensive terms with short slang words. There is a plethora of unsupervised learning models in the existing literature to deal with hate-speech BIBREF3 , as well as in detecting the sentiment polarity in tweets BIBREF4 . At the same time, the supervised learning approaches have not been explored adequately so far. While the task of sentence classification seems similar to that of sentiment analysis; nevertheless, in hate-speech even negative sentiment could still provide useful insight. Our intuition is that the task of hate-speech detection can be further benefited by the incorporation of other sources of information to be used as features into a supervised learning model. A simple statistical analysis on an existing annotated dataset of tweets by BIBREF5 , can easily reveal the existence of significant correlation between the user tendency in expressing opinions that belong to some offensive class (Racism or Sexism), and the annotation labels associated with that class. More precisely, the correlation coefficient value that describes such user tendency was found to be 0.71 for racism in the above dataset, while that value reached as high as 0.76 for sexism. In our opinion, utilizing such user-oriented behavioural data for reinforcing an existing solution is feasible, because such information is retrieva2ble in real-world use-case scenarios like Twitter. This highlights the need to explore the user features more systematically to further improve the classification accuracy of a supervised learning system. Our approach employs a neural network solution composed of multiple Long-Short-Term-Memory (LSTM) based classifiers, and utilizes user behavioral characteristics such as the tendency towards racism or sexism to boost performance. Although our technique is not necessarily revolutionary in terms of the deep learning models used, we show in this paper that it is quite effective. Our main contributions are: INLINEFORM0 ) a deep learning architecture for text classification in terms of hateful content, which incorporates features derived form the users' behavioural data, INLINEFORM1 ) a language agnostic solution, due to no-use of pre-trained word embeddings, for detecting hate-speech, INLINEFORM2 ) an experimental evaluation of the model on a Twitter dataset, demonstrating the top performance achieved on the classification task. Special focus is given to investigating how the additional features concerning the users' tendency to utter hate-speech, as expressed by their previous history, could leverage the performance. To the best of our knowledge, there has not been done any previous study on exploring features related to the users tendency in hatred content that used a deep learning model. The rest of the paper is organized as follows. In Section SECREF2 we describe the problem of hate speech in more detail, and we refer to the existing work in the field in Section SECREF3 . In Section SECREF4 we present our proposed model, while in Section SECREF5 we refer to the dataset used, the evaluation tests we performed and we discuss the results received. Finally, in Section SECREF6 we summarize our contributions and discuss the future work.