We propose a novel approach to multimodal sentiment analysis using deep neural networks combining visual recognition and natural language processing. Our goal is different than the standard sentiment analysis goal of predicting whether a sentence expresses positive or negative sentiment; instead, we aim to infer the latent emotional state of the user. Thus, we focus on predicting the emotion word tags attached by users to their Tumblr posts, treating these as "self-reported emotions." We demonstrate that our multimodal model combining both text and image features outperforms separate models based solely on either images or text. Our model's results are interpretable, automatically yielding sensible word lists associated with emotions. We explore the structure of emotions implied by our model and compare it to what has been posited in the psychology literature, and validate our model on a set of images that have been used in psychology studies. Finally, our work also provides a useful tool for the growing academic study of imagesboth photographs and memes-on social networks.
This paper presents a method for classifying Tumblr posts with associated images according to associated single emotion word hashtags.  The method relies on sentiment pre-processing from GloVe and image pre-processing from Inception.  
 
My strongest criticism for this paper is against the claim that Tumblr post represent self-reported emotions and that this method sheds new insight on emotion representation and my secondary criticism is a lack of novelty in the method, which seems to be simply a combination of previously published sentiment analysis module and previously published image analysis module, fused in an output layer. 
The authors claim that the hashtags represent self-reported emotions, but this is not true in the way that psychologists query participants regarding emotion words in psychology studies.  Instead these are emotion words that a person chooses to broadcast along with an associated announcement.  As the authors point out, hashtags and words may be used sarcastically or in different ways from what is understood in emotion theory.  It is quite common for everyday people to use emotion words this way e.g. using #love to express strong approval rather than an actual feeling of love.   
In their analysis the authors claim:
“The 15 emotions retained were those with high relative frequencies on Tumblr among the PANAS-X scale (Watson & Clark, 1999)”.
However five of the words the authors retain: bored, annoyed, love, optimistic, and pensive are not in fact found in the PANAS-X scale:
Reference: The PANAS-X Scale: https://wiki.aalto.fi/download/attachments/50102838/PANAS-X-scale_spec.pdf Also the longer version that the authors cited: 
https://www2.psychology.uiowa.edu/faculty/clark/panas-x.pdf
It should also be noted that the PANAS (Positive and Negative Affect Scale) scale and the PANAS-X (the “X” is for eXtended) scale are questionnaires used to elicit from participants feelings of positive and negative affect, they are not collections of "core" emotion words, but rather words that are colloquially attached to either positive or negative sentiment.  For example PANAS-X includes words like:“strong” ,“active”, “healthy”, “sleepy” which are not considered emotion words by psychology.  
If the authors stated goal is "different than the standard sentiment analysis goal of predicting whether a sentence expresses positive or negative sentiment" they should be aware that this is exactly what PANAS is designed to do - not to infer the latent emotional state of a person, except to the extent that their affect is positive or negative.
The work of representing emotions had been an field in psychology for over a hundred years and it is still continuing.  https://en.wikipedia.org/wiki/Contrasting_and_categorization_of_emotions.
One of the most popular theories of emotion is the theory that there exist “basic” emotions: Anger, Disgust, Fear, Happiness (enjoyment), Sadness and Surprise (Paul Ekman, cited by the authors).  These are short duration sates lasting only seconds.  They are also fairly specific, for example “surprise” is sudden reaction to something unexpected, which is it exactly the same as seeing a flower on your car and expressing “what a nice surprise.”  The surprise would be the initial reaction of “what’s that on my car?  Is it dangerous?” but after identifying the object as non-threatening, the emotion of “surprise” would likely pass and be replaced with appreciation.  
The Circumplex Model of Emotions (Posner et al 2005) the authors refer to actually stands in opposition to the theories of Ekman.  From the cited paper by Posner et al : 
"The circumplex model of affect proposes that all affective states arise from cognitive interpretations of core neural sensations that are the product of two independent neurophysiological systems. This model stands in contrast to theories of basic emotions, which posit that a discrete and independent neural system subserves every emotion."
From my reading of this paper, it is clear to me that the authors do not have a clear understanding of the current state of psychology’s view of emotion representation and this work would not likely contribute to a new understanding of the latent structure of peoples’ emotions.
In the PCA result, it is not "clear" that the first axis represents valence, as "sad" has a slight positive on this scale and "sad" is one of the emotions most clearly associated with negative valence.
With respect to the rest of the paper, the level of novelty and impact is "ok, but not good enough."  This analysis does not seem very different from Twitter analysis, because although Tumblr posts are allowed to be longer than Twitter posts, the authors truncate the posts to 50 characters.  Additionally, the images do not seem to add very much to the classification.  The authors algorithm also seems to be essentially a combination of two other, previously published algorithms.
For me the novelty of this paper was in its application to the realm of emotion theory, but I do not feel there is a contribution here.  This paper is more about classifying Tumblr posts according to emotion word hashtags than a paper that generates a new insights into emotion representation or that can infer latent emotional state.