Hi there,
I’m trying to do a sentiment analysis algorithm for music with lyrics as an input.
I tried some models but it seems that I can’t get a good one.Here are some details about the problem:
1.The dataset(around 11k samples):
-80% training - 20% validation
-I am using as input 4 lyrics for each song. The lyrics are the ones with the biggest frequency in the song(I have chosen them without duplicates)
-I am preprocessing them(use lower case, remove non-letters and useless whitespaces.
-I am trying to predict 2 numbers : arousal and valence. I can compute the sentiment knowing these
-I also tried the classification problem(transformed the labels directly into sentiments) with 12 classes, but I still didn t get good results.
-after preprocessing I am embedding the words in every lyric. For example if I have 50 words after embedding I get a 50 x embedding_size tensor. The embedding are also trained. I was thinking about using some pre-trained ones like word2vec or GloVe.
2.The arhitecture - I had these approaches:
-using only a GRU, then an output linear layer
-convolutional layer with output linear layer
-prenet using a 2 layered NN with a bottleneck(256 - 128 size), CNN and then linear layer
-CNN, GRU, linear layer
-module with a bank of convolutions of different kernel sizes(2,3,4) followed by GRU and linear layer
-a few more with similar results
As a note, I mainly use ReLU as activation and tanh in the final layer. After applying tanh I multiply the result by 3 to get numbers between -3,3(valence and arousal are between these values).
I use MSE as a loss function. I noticed that the loss goes around 1(maybe 0.6 in happy cases) and then stays there. The accuracy can’t get better than 20% and it usually goes to 5-10% after some iterations. The train loss and validation loss are usually decreasing.
I am training with mini-batches. I used as an optimizer SGD or Adam. I tried a lot of learning rates,but no value helped or proved better results.
I looked over my code and I don’t think there is a bug. Here is my code : https://github.com/BogdanAngh/Poorify/tree/RegressionModel/AI 1 -> everything starts in train.py
I am wondering if my arhitecture is bad or the problem is too hard with what I proposed. I used arhitectures which I met some read papers.
I need some advices because I don’t know where to move from here. Thank you so much for reading this, I know it s pretty long