This paper explores a new natural language processing task, review-driven multi-label music style classification. This task requires systems to identify multiple styles of music based on its reviews on websites. The biggest challenge lies in the complicated relations of music styles. To tackle this problem, we propose a novel deep learning approach to automatically learn and exploit style correlations. Experiment results show that our approach achieves large improvements over baselines on the proposed dataset. Furthermore, the visualized analysis shows that our approach performs well in capturing style correlations. 1 1 Introduction As music style (e.g., Jazz, Pop, and Rock) is one of the most frequently used labels for music, music style classification is an important task for applications of music recommendation, music information retrieval, etc. Several methods have been proposed for automatic music style classification (Qin and Ma, 2005; Zhou et al., 2006; Wang et al., 2009; Choi et al., 2017). Most of them mainly focus on using audio information to identify styles. Motivated by the fact that a pieces of music could has different styles, several studies (Wang et al., 2009; Oramas et al., 2017) also aim at multi-label music style classification. Although these methods make promising progress, they are limited in two aspects. First, not all audio data is available in real-world applications because of copyright restrictions, which limits the generalization ability. Second, some of them are based on a strong assumption that a piece of music should be assigned with only one style. Different from these studies, we focus on using  Equal Contribution 1 The code and the dataset are available at https:// github.com/lancopku/RMSC easily obtained reviews in conjunction with multilabel music style classification. The motivation comes from the fact that lots of user reviews contain rich style-related information, which can be used for music style classification. The major challenge of this task lies in the complicated correlations of music styles. For example, Soul Music 2 contains elements of R&B and Jazz. These three labels can be used alone or in combination. Traditional multi-label classification methods may mistake the true label [Soul Music, R&B, Jazz] for the false label [R&B, Jazz]. If well learned, style relations are useful knowledge for improving the performance, e.g., increasing the probability of Soul Music if we find that it is heavily linked with two high probability labels: R&B and Jazz. Therefore, to better exploit style correlations, we propose a novel deep learning approach with two parts: a label-graph based neural network, and a soft training mechanism with correlation based continuous label representation. Our contributions are listed as follows:  To the best of our knowledge, this work is the first to explore review-driven multi-label music style classification. To learn the relations among music styles, we propose a label-graph based neural network and a soft training mechanism with correlation-based label representation. This paper is related with music style classification and multi-label classification. In this section, we give a detailed introduction about the related studies. 2 Soul Music is a popular music genre that originated in the United States in the late 1950s and early 1960s. It contains elements of African-American Gospel Music, R&B and Jazz. Music Title Mozart: The Great Piano Concertos, Vol.1 Previous work mainly focuses on using audio information to identify music styles. Traditional machine learning algorithms are adopted in these studies, such as Support Vector Machine (SVM) (Xu et al., 2003), Hidden Markov Model (HMM) (Chai and Vercoe, 2001; Pikrakis et al., 2006), and Decision Tree (DT) (Zhou et al., 2006). In addition to audio information, Fell and Sporleder (2014) also propose to classify music by statistical analysis of lyrics. Motivated by the fact that a piece of music could has different styles, several studies (Wang et al., 2009; Oramas et al., 2017) also aim at multi-label music style classification. Different from these studies, we focus on using easily obtained reviews in conjunction with multi-label music style classification. Multi-label classification has been widely applied to diverse problems, including image classification (Qi et al., 2007; Wang et al., 2008), audio classification (Boutell et al., 2004; Sanden and Zhang, 2011), web mining (Kazawa et al., 2004), information retrieval (Zhu et al., 2005; Gopal and Yang, 2010), etc. Compared with the existing multilabel learning methods (Wei et al., 2018; Li et al., 2018b,a; Yang et al., 2018; Lin et al., 2018), our method has the following novelties: a label graph that explicitly models the relations of styles; a soft training mechanism that introduces correlation-based continuous label representation. Given several reviews from a piece of music, this task requires models to predict a set of music styles. Assume that X = { x 1 , . . . , x i , . . . , x K } denotes the input K reviews, and x i = x i, 1 , . . . , x i,J represents the i th review with J words. The term Y = { y 1 , y 2 , . . . , y M } denotes the gold set with M labels, and M varies in different samples. The target of review-driven multilabel music style classification is to learn the mapping from input reviews to style labels. The dataset is collected from a popular Chinese music review website, 3 where registered users are allowed to comment on all released music albums. Each sample includes a music title, a set of human annotated styles, and associated reviews. An example is shown in Table 1. In order to build a high-quality dataset, we refer to the literature about music styles. We merge similar music styles and delete music styles that violate the music classification list. 22 styles are defined in our dataset. 4 For user reviews, we first delete reviews with too little information by rule-based methods and then select top 40 voted reviews. Music samples with too few reviews are also deleted. The constructed datataset contains over 7.1k samples, 288K reviews, and 3.6M words. The proposed approach contains two parts: a label-graph based neural network and a soft training mechanism with continuous label representation. An illustration of the proposed method is shown in Figure 1. 4.1 Label-Graph Based Neural Network The first layer is a hierarchical attention layer (Yang et al., 2016) that lets the model to pay more or less attention to individual words 3 https://music.douban.com 4 Alternative Music, Britpop, Classical Music, Country Music, Dark Wave, Electronic Music, Folk Music, Heavy Metal Music, Hip-Hop, Independent Music, Jazz, J-Pop, New-Age Music, OST, Piano Music, Pop, Post-Punk, PostRock, Punk, R&B, Rock, and Soul Music. and reviews when constructing raw label probability distribution z . Label Graph. To explicitly take advantage of the label correlations when classifying music styles, we add a label graph layer to the network. This layer takes z as input and generates a soft label probability distribution e . Formally, we denote G  R m  m as the label graph, where m is the number of labels, G is initialized by an identity matrix. An element G [ l i , l j ] is a real-value score indicating how likely label l i and label l j are related. The graph G is a part of parameters and can be learned by back-propagation. Then, given the raw label probability distribution z and the label graph G , the output of this layer is: e = sigmoid( z  G ) . The probability of l j is not only determined by the current classification result, but also determined by other labels probabilities and their correlations to l j . For example, the probability of a label heavily linked with many high-probability labels will be increased. Given a predicted label probability distribution e and a target discrete label representation y , the typical loss function is computed as where  denotes all parameters, and m is the number of the labels. The function H denotes the cross entropy between two distributions. However, the widely used discrete label representation does not apply to the task of music style classification, because the music styles are not mutually exclusive and highly related to each other. The discrete distribution without label relations makes the model over-distinguish the related labels. Therefore, it is hard for the model to learn the label correlations. Instead, we propose a soft training method by combining a discrete label representation y with a correlated-based continuous label representation y (cid:48) . The probability gap between two similar labels in y (cid:48) should not be large. A straight-forward approach to produce the continuous label representation is to use the label graph matrix G to transform the discrete representation y into a continuous form: y (cid:48) = y  G . Based on the discrete label representation y and continuous label representation y (cid:48) , we define the loss function as Loss (  ) = H ( e , y ) + H ( e , y (cid:48) ) , where the loss H ( e , y ) aims to correctly classify labels, and the loss H ( e , y (cid:48) ) aims to avoid the over-distinguishing problem and to better learn label correlations. In this section, we evaluate our approach on the proposed dataset. We first introduce evaluation metrics, then show experiment results and give a detailed analysis. The training details can be found at Appendices. Multi-label classification requires different evaluation metrics compared with traditional single-label classification. In this paper, we use the following widely-used evaluation metrics for multilabel classification. F1-score: We calculate macro F1 and micro F1, respectively. Macro F1 computes the metric independently for each label and then takes the average as the final score. Micro F1 aggregates the contributions of all labels to compute the average score. One-Error: One-error evaluates the fraction of examples whose top-ranked label is not in the gold label set. Hamming Loss: Hamming loss counts the fraction of the wrong labels to the total number of labels. We implement several widely-used multi-label classification methods as baselines, such as ML-KNN (Zhang and Zhou, 2007), Binary Relevance (Tsoumakas et al., 2010), Classifier Chains (Read et al., 2011), Label Powerset (Tsoumakas and Vlahavas, 2007). The details of baselines can be found at Appendices. The results on the test set are summarized in Table 2. The proposed approach significantly outperforms the baselines, with micro F1 of 64.5, macro F1 of 54.4, and one-error of 22.6, improving the metrics by 10.6, 21.4, and 7.9 respectively. The improvements are attributed to two parts, a hierarchical attention network and a label correlation mechanism. Only using the hierarchical attention network outperforms the baselines, which shows the effectiveness of hierarchically paying attention to different words and sentences. The greater F1-score is achieved by adding the proposed label graph, which demonstrates that the proposed label graph helps a lot by taking advantage of label correlations. It can be clearly seen that with the help of soft training, the proposed method achieves the best performance. Especially, the micro F-score is improved from 62.8 to 64.5, and the one-error is reduced from 23.4 to 22.6. With the new loss function, the model not only knows how to distinguish the right labels from the wrong ones, but also can learn the label correlations that are useful knowledge, especially when the input data contains too much style unrelated words for the model to extract all necessary information. Figure 2 shows a whole heatmap of the automatically learned label graph, for convenience of display, we have subtracted the label graph by an identity matrix. We can see from the picture that some obvious music style relations are well captured. For Country Music, the most related label is Folk Music. In reality, these two music styles are highly similar and the boundary between them is not well-defined. For three kinds of rock music, Heavy Metal Music, Britpop Music, and Alternative Music, the label graph correctly captures that the most related label for them is Rock. For a more complicated relation where Soul Music is highly linked with two different labels, R&B and Jazz, the label graph also correctly capture such relation. These examples demonstrate that the proposed approach performs well in capturing relations among music styles. For clearer understanding, we compare several examples generated with and without the label correlation mechanism in Table 3. By comparing gold labels and predicted labels generated by different methods, we find that the proposed label correlation mechanism identifies the related styles more precisely. This is mainly attributed to the learned label correlations. For example, the correct prediction in the first example shows that the label correlation mechanism captures the close relation between Britpop and Rock, which helps the model to generate an appropriate prediction. Although the proposed method has achieved sig-nificant improvements, we also notice that there are some failure cases. In this section, we give the detailed error analysis. First, the proposed method performs worse on the styles with low frequency in the training set. 5 Britpop is a style of British Rock. 6 Hip-Hop is a mainstream Pop style. 7 Rhythm and Blues, often abbreviated as R&B, is a genre of popular music. 8 New-Age Music is a genre of music intended to create artistic inspiration, relaxation, and optimism. It is used by listeners for yoga, massage, and meditation. Table 4 compares the performance on the top 5 music styles of highest and lowest frequencies. As we can see, the top 5 fewest music styles get much worse results than top 5 most music styles. This is because the label distribution is highly imbal-anced where unpopular music styles have too little training data. Second, we find that some music items are wrongly classified into the styles that are similar with the gold styles. For example, a sample with a gold set [Country Music] is wrongly classified into [Folk] by the model. The reason is that some music styles share many common elements and only subtly differ from each other. It poses a great challenge for the model to distinguish them. For future work, we would like to research how to effectively address this problem. In this paper, we focus on classifying multi-label music styles with user reviews. To meet the challenge of label correlations, we propose a label-graph neural network and a soft training mechanism. Experiment results have showed the effectiveness of the proposed approach. The visualization of label graph also shows that our method performs well in capturing label correlations. We thank all reviewers for providing the thoughtful and constructive suggestions. This work was supported in part by National Natural Science Foundation of China (No. 61673028). Email correspondence to Xu Sun.
objective abstain abstain objective objective result abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain result objective objective objective abstain abstain other other other other other other other abstain other method abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain other abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain method objective abstain result other other other