Multimodal Sentiment Analysis To Explore the Structure of Emotions
We propose a novel approach to multimodal sentiment analysis using deep neural networks combining visual recognition and natural language processing. Our goal is different than the standard sentiment analysis goal of predicting whether a sentence expresses positive or negative sentiment; instead, we aim to infer the latent emotional state of the user. Thus, we focus on predicting the emotion word tags attached by users to their Tumblr posts, treating these as “self-reported emotions.” We demonstrate that our multimodal model combining both text and image features outperforms separate models based solely on either images or text. Our model’s results are interpretable, automatically yielding sensible word lists associated with emotions. We explore the structure of emotions implied by our model and compare it to what has been posited in the psychology literature, and validate our model on a set of images that have been used in psychology studies. Finally, our work also provides a useful tool for the growing academic study of images— both photographs and memes—on social networks.
 The paper presents a multi-modal CNN model for sentiment analysis that combines images and text. The model is trained on a new dataset collected from Tumblr. Positive aspects: + Emphasis in model interpretability and its connection to psychological findings in emotions + The idea of using Tumblr data seems interesting, allowing to work with a large set of emotion categories, instead of considering just the binary task positive vs. negative. Weaknesses: - A deeper analysis of previous work on the combination of image and text for sentiment analysis (both datasets and methods) and its relation with the presented work is necessary. - The proposed method is not compared with other methods that combine text and image for sentiment analysis. - The study is limited to just one dataset. The paper presents interesting ideas and findings in an important challenging area. The main novelties of the paper are: (1) the use of Tumblr data, (2) the proposed CNN architecture, combining images and text (using word embedding. I missed a "related work section", where authors clearly mention previous works on similar datasets. Some related works are mentioned in the paper, but those are spread in different sections. It*s hard to get a clear overview of the previous research: datasets, methods and contextualization of the proposed approach in relation with previous work. I think authors should cite Sentibanks. Also, at some point authors should compare their proposal with previous work. More comments: - Some figures could be more complete: to see more examples in Fig 1, 2, 3 would help to understand better the dataset and the challenges. - In table 4, for example, it would be nice to see the performance on the different emotion categories. - It would be interesting to see qualitative visual results on recognitions. I like this work, but I think authors should improve the aspects I mention for its publication.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Multimodal Sentiment Analysis To Explore the Structure of Emotions
We propose a novel approach to multimodal sentiment analysis using deep neural networks combining visual recognition and natural language processing. Our goal is different than the standard sentiment analysis goal of predicting whether a sentence expresses positive or negative sentiment; instead, we aim to infer the latent emotional state of the user. Thus, we focus on predicting the emotion word tags attached by users to their Tumblr posts, treating these as “self-reported emotions.” We demonstrate that our multimodal model combining both text and image features outperforms separate models based solely on either images or text. Our model’s results are interpretable, automatically yielding sensible word lists associated with emotions. We explore the structure of emotions implied by our model and compare it to what has been posited in the psychology literature, and validate our model on a set of images that have been used in psychology studies. Finally, our work also provides a useful tool for the growing academic study of images— both photographs and memes—on social networks.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The paper presents a multi-modal CNN model for sentiment analysis that combines images and text. The model is trained on a new dataset collected from Tumblr. Positive aspects: + Emphasis in model interpretability and its connection to psychological findings in emotions + The idea of using Tumblr data seems interesting, allowing to work with a large set of emotion categories, instead of considering just the binary task positive vs. negative. Weaknesses: - A deeper analysis of previous work on the combination of image and text for sentiment analysis (both datasets and methods) and its relation with the presented work is necessary. - The proposed method is not compared with other methods that combine text and image for sentiment analysis. - The study is limited to just one dataset. The paper presents interesting ideas and findings in an important challenging area. The main novelties of the paper are: (1) the use of Tumblr data, (2) the proposed CNN architecture, combining images and text (using word embedding. I missed a "related work section", where authors clearly mention previous works on similar datasets. Some related works are mentioned in the paper, but those are spread in different sections. It*s hard to get a clear overview of the previous research: datasets, methods and contextualization of the proposed approach in relation with previous work. I think authors should cite Sentibanks. Also, at some point authors should compare their proposal with previous work. More comments: - Some figures could be more complete: to see more examples in Fig 1, 2, 3 would help to understand better the dataset and the challenges. - In table 4, for example, it would be nice to see the performance on the different emotion categories. - It would be interesting to see qualitative visual results on recognitions. I like this work, but I think authors should improve the aspects I mention for its publication.
assistant
 The paper presents a multi-modal CNN model for sentiment analysis that combines images and text. The model is trained on a new dataset collected from Tumblr. Positive aspects: + Emphasis in model interpretability and its connection to psychological findings in emotions + The idea of using Tumblr data seems interesting, allowing to work with a large set of emotion categories, instead of considering just the binary task positive vs. negative. Weaknesses: - A deeper analysis of previous work on the combination of image and text for sentiment analysis (both datasets and methods) and its relation with the presented work is necessary. - The proposed method is not compared with other methods that combine text and image for sentiment analysis. - The study is limited to just one dataset. The paper presents interesting ideas and findings in an important challenging area. The main novelties of the paper are: (1) the use of Tumblr data, (2) the proposed CNN architecture, combining images and text (using word embedding. I missed a "related work section", where authors clearly mention previous works on similar datasets. Some related works are mentioned in the paper, but those are spread in different sections. It*s hard to get a clear overview of the previous research: datasets, methods and contextualization of the proposed approach in relation with previous work. I think authors should cite Sentibanks. Also, at some point authors should compare their proposal with previous work. More comments: - Some figures could be more complete: to see more examples in Fig 1, 2, 3 would help to understand better the dataset and the challenges. - In table 4, for example, it would be nice to see the performance on the different emotion categories. - It would be interesting to see qualitative visual results on recognitions. I like this work, but I think authors should improve the aspects I mention for its publication.