what was the baseline?
Random labeling
Sentiment analysis is essential in many real-world applications such as stance detection, review analysis, recommendation system, and so on. Sentiment analysis becomes more difficult when the data is noisy and collected from social media. India is a multilingual country; people use more than one languages to communicate within themselves. The switching in between the languages is called code-switching or code-mixing, depending upon the type of mixing. This paper presents overview of the shared task on sentiment analysis of code-mixed data pairs of Hindi-English and Bengali-English collected from the different social media platform. The paper describes the task, dataset, evaluation, baseline and participant's systems.
The past decade witnessed rapid growth and widespread usage of social media platforms by generating a significant amount of user-generated text. The user-generated texts contain high information content in the form of news, expression, or knowledge. Automatically mining information from user-generated data is unraveling a new field of research in Natural Language Processing (NLP) and has been a difficult task due to unstructured and noisy nature. In spite of the existing challenges, much research has been conducted on user-generated data in the field of information extraction, sentiment analysis, event extraction, user profiling and many more. According to Census of India, there are 22 scheduled languages and more than 100 non scheduled languages in India. There are 462 million internet users in India and most people know more than one language. They express their feelings or emotions using more than one languages, thus generating a new code-mixed/code-switched language. The problem of code-mixing and code-switching are well studied in the field of NLP BIBREF0 , BIBREF1 . Information extraction from Indian internet user-generated texts become more difficult due to this multilingual nature. Much research has been conducted in this field such as language identification BIBREF2 , BIBREF3 , part-of-speech tagging BIBREF4 . Joshi et al. JoshiPSV16 have performed sentiment analysis in Hindi-English (HI-EN) code-mixed data and almost no work exists on sentiment analysis of Bengali-English (BN-EN) code-mixed texts. The Sentiment Analysis of Indian Language (Code-Mixed) (SAIL _Code-Mixed) is a shared task at ICON-2017. Two most popular code-mixed languages namely Hindi and Bengali mixed with English were considered for the sentiment identification task. A total of 40 participants registered for the shared task and only nine teams have submitted their predicted outputs. Out of nine unique submitted systems for evaluation, eight teams submitted fourteen runs for HI-EN dataset whereas seven teams submitted nine runs for BN-EN dataset. The training and test dataset were provided after annotating the languages and sentiment (positive, negative, and neutral) tags. The language tags were automatically annotated with the help of different dictionaries whereas the sentiment tags were manually annotated. The submitted systems are ranked using the macro average f-score. The paper is organized as following manner. Section SECREF2 describes the NLP in Indian languages mainly related to code-mixing and sentiment analysis. The detailed statistics of the dataset and evaluation are described in Section SECREF3 . The baseline systems and participant's system description are described in Section SECREF4 . Finally, conclusion and future research are drawn in Section SECREF5 .