When assigning quantitative labels to a dataset, different methodologies may rely on different scales. In particular, when assigning polarities to words in a sentiment lexicon, annotators may use binary, categorical, or continuous labels. Naturally, it is of interest to unify these labels from disparate scales to both achieve maximal coverage over words and to create a single, more robust sentiment lexicon while retaining scale coherence. We introduce a generative model of sentiment lexica to combine disparate scales into a common latent representation. We realize this model with a novel multi-view variational autoencoder (VAE), called SentiVAE. We evaluate our approach via a downstream text classification task involving nine English-Language sentiment analysis datasets; our representation outperforms six individual sentiment lexica, as well as a straightforward combination thereof. Sentiment lexica provide an easy way to automatically label texts with polarity values, and are also frequently transformed into features for supervised models, including neural networks (Palogiannidi et al., 2016; Ma et al., 2018). Indeed, given their utility, a veritable cottage industry has emerged focusing on the design of sentiment lexica. In practice, using any single lexicon, unless specifically and carefully designed for the particular domain of interest, has several downsides. For example, any lexicon will typically have low coverage compared to the language's entire vocabulary, and may have misspecified labels for the domain. In many cases, it may therefore be desirable to combine multiple sentiment lexica into a single representation. Indeed, some research on unifying Figure 1: A depiction of the encoder portion of SentiVAE. such lexica has emerged (Emerson and Declerck, 2014; Altrabsheh et al., 2017), borrowing ideas from crowdsourcing (Raykar et al., 2010; Hovy et al., 2013). However, this is a non-trivial task, because lexica can use binary, categorical, or continuous scales to quantify polarityin addition to different interpretations for eachand thus cannot easily be combined. In Fig. 1, we show an example of the same word labeled using different lexica to illustrate the nature of the challenge. To combine sentiment lexica with disparate scales, we introduce SentiVAE, a novel multiview variant of the variational autoencoder (VAE) (Kingma and Welling, 2014). SentiVAE, visualized as a graphical model in Fig. 2, differs from the original VAE in two ways: (i) it uses a Dirichlet latent variable (rather than a Gaussian) for each word in the combined vocabulary, and (ii) it has multiple emission distributionsone for each lexicon. Because the latent variables are shared across the lex-Lexicon Source N Dom SentiWordNet WordNet 14107 [  1 , 1] 2 MPQA Newswire 4397 { 0 , 1 } SenticNet  100000 [  1 , 1] Hu-Liu Product reviews 6790 { 0 , 1 } GI  4206 { 0 , 1 } VADER Social media 7489 { 0 ,..., 8 } 10 Table 1: Descriptive statistics for the sentiment lexica. ica, we are able to derive a common latent representation of the words' polarities. The resulting model is spiritually related to a multi-view learning approach (Sun, 2013), where each view corresponds to a different lexicon. Experimentally, we use SentiVAE to combine six commonly used English-language sentiment lexica with disparate scales. We evaluate the resulting representation via a text classification task involving nine English-language sentiment analysis datasets. For each dataset, we transform each text into an average polarity value using either our representation, one of the six commonly used sentiment lexica, or a straightforward combination thereof. We then train a classifier to predict the overall sentiment of each text from its average polarity value. We find that our representation outperforms the individual lexica, as well as the straightforward combination for some datasets. Our representation is particularly efficacious for datasets from domains that are not well-supported by standard sentiment lexica. 1 The existing research that is most closely related to our work is SentiMerge (Emerson and De-clerck, 2014), a Bayesian approach for aligning sentiment lexica with different continuous scales. SentiMerge consists of two steps: (i) aligning the lexica via rescaling, and (ii) combining the rescaled lexica using a Gaussian distribution. The authors perform token-level evaluation using a single sentiment analysis dataset where each token is labeled with its contextually dependent sentiment. Because SentiMerge can only combine lexica with continuous scales, we do not include it in our evaluation. We use the following commonly used English-language sentiment lexica: SentiWordNet (Bac-cianella et al., 2010), MPQA (Wilson et al., 2005), SenticNet 5 (Cambria et al., 2014), Hu-Liu (Hu and Liu, 2004), GI (Stone et al., 1962), and VADER (Hutto and Gilbert, 2014). Descriptive statistics for each lexicon are shown in Tab. 1. Each word in SentiWordNet is labeled with two real values, each in the interval [0 , 1] , corresponding to the strength of positive and negative sentiment (e.g., the label (0 0) is neutral, while the label (1 0) is maximally positive). Each word in VADER is labeled by ten different human evaluators, with each evaluator providing a polarity value on a nine-point scale (where the midpoint is neutral), yielding a 10-dimensional label. MPQA, Hu-Liu, and GI all use binary scales. Lastly, each word in SenticNet is labeled with a real value in the interval [  1 , 1] , where 0 is neutral. We first describe a figurative generative process for a single sentiment lexicon d  D , where D is a set of sentiment lexica. Imagine there is a true (latent) polarity value z w associated with each word w in the lexicon's vocabulary. When the lexicon's creator labels that word according to their chosen scale (e.g., thumbs-up or thumbs-down, a real value in the interval [0 , 1] ), they deterministically transform this true value to their chosen scale via a function f (  ;  d ) . 2 Sometimes, noise is introduced during this labeling process, corrupting the label as it leaves the ethereal realm and producing the (observed) polarity label x wd . They then add this potentially noisy label to the lexicon. Given a lexicon of observed polarity labels, the latent polarity values can be inferred using a VAE. The original VAE posits a generative model of observed data X and latent variables Z : P ( X , Z ) = P ( X | Z ) P ( Z ) . Inference of Z then proceeds by approximating the (intractable) posterior P ( Z | X ) with a Gaussian distribution, factorized over the individual latent variables. A parameterized encoder function compresses X into Z , while a parameterized decoder function reconstructs X from Z . SentiVAE extends the original VAE model to combine multiple lexica with disparate scales, producing a common latent representation of the polarity value for each word in the combined vocabulary. Generative process. Given a set of sentiment lexica D with a combined vocabulary W , SentiVAE posits a common latent representation z w of the polarity value for each word w  W , where z w is a three-dimensional categorical distribution over 2 Parameterized by lexicon-specific weights  d . the sentiments positive , negative , and neutral . The generative process starts by drawing each latent polarity value z w from a three-dimensional Dirichlet prior, parameterized by  w = (1 , 1 , 1) : z w  Dir (  w ) . (1) If the word is uncontroversial, 3 we spur this prior somewhat using the number of lexica in which the word appears c ( w ) . Specifically, we add c ( w ) to the parameter for the sentiment associated with that word in the lexica, e.g.,  SUPERB = (1 + c ( SUPERB ) , 1 , 1) . This has the effect of regularizing the inferred latent polarity value toward the desired distribution over sentiments. Having generated z w , the process proceeds by decoding z w into each lexicon's chosen scale. First, for each lexicon d  D , z w is deterministically transformed via neural network f (  ;  d ) with a single 32-dimensional hidden layer, parameterized by lexicon-specific weights  d :  wd = f ( z w ;  d ) . The dimensionality of  wd and the emission distribution P d are lexicon-specific. For SentiWordNet, P d 3 We say that a word is uncontroversial if there is strong agreement across the sentiment lexica in which it appears. Even without this spurring, the inferred latent representation typically separates into the three sentiment classes, but performance on our text classification task is somewhat diminished. is a two-dimensional Gaussian with mean  wd and a diagonal covariance matrix equal to 0 . 01 I ; for VADER, P d consists of ten nine-dimensional categorical distributions, collectively parameterized by  wd ; for MPQA, Hu-Liu, and GI, P d is a Bernoulli distribution, parameterized by  wd ; and for SenticNet, P d is a univariate Gaussian with mean and variance each an element in a two-dimensional  wd . Inference. Inference involves forming the posterior distribution over the latent polarity values Z given the observed polarity labels X . Because computing the normalizing constant P ( X ) is intractable, we instead approximate the posterior with a family of distributions Q  ( Z ) , indexed by variational parameters  . Specifically, we use Q  ( Z ) = (cid:89) w W Q  w ( z w ) = (cid:89) w W Dir (  w ) . To construct  w , we first define a neural network g (  ;  d ) , with a single 32-dimensional hidden layer, which encodes x wd into a three-dimensional vector. The output of this neural network is then transformed via a softmax as follows:  wd = softmax (cid:0) g ( x wd ;  d ) (cid:1) (5)  w = 1 + (cid:88) d D  wd . The intuition behind  w can be understood by appealing to the pseudocount interpretation of Dirichlet parameters. Each lexicon contributes exactly one pseudocount, divided among positive , negative , and neutral , to what would otherwise be a symmetric, uniform Dirichlet distribution. As a consequence of this construction, words that appear in more lexica will have more concentrated Dirichlets. Intuitively, this property is appealing. We optimize the resulting ELBO objective (Blei et al., 2017) with respect to the variational parameters via stochastic variational inference (Hoffman IMDB 2C Yelp 5C Yelp 3C SemEval 3C MultiDom 2C ACL 5C ACL 3C ICLR 10C ICLR 3C SentiVAE EQ [ z w ] 72.7 49.8 57.5 46.0 70.8 66.7 73.3 92.6 87.0 SentiVAE  w 73.4 49.7 59.4 52.2 74.7 73.3 80.0 92.6 86.5 SentiWordNet 63.4 36.0 47.6 32.2 62.0 60.0 53.3 89.1 83.5 MPQA 65.4 44.0 53.0 29.9 67.4 60.0 53.3 89.1 83.5 SenticNet 60.5 38.4 43.4 37.2 62.3 60.0 53.3 89.1 83.9 Hu-Liu 67.2 46.6 56.4 31.5 69.4 60.0 53.3 89.1 83.5 GI 58.4 40.7 47.9 31.3 61.6 60.0 53.3 89.1 83.5 VADER 71.7 46.8 59.3 38.5 73.5 66.7 66.7 94.3 86.1 Combined 75.6 51.0 64.1 50.6 75.4 66.7 66.7 93.9 86.1 Table 3: Classification accuracies for our representation, six lexica, and a straightforward combination thereof. et al., 2013) using Adam (Kingma and Ba, 2015) in the Pyro framework (Bingham et al., 2018). The standard reparameterization trick used in the original VAE does not apply to models with Dirichlet-distributed latent variables, so we use the generalized reparameterization trick of Ruiz et al. (2016). To evaluate our approach, we first use SentiVAE to combine the six lexica described in 2. For each word w in the combined vocabulary, we obtain an estimate of z w by taking the mean of Q  w ( z w ) = Dir (  w ) i.e., by normalizing  w . We compare this representation to using  w directly, because  w contains information about Sen-tiVAE's certainty about the word's latent polarity value. We evaluate our common latent representation via a text classification task involving nine English-language sentiment analysis datasets: IMDB (Maas et al., 2011), Yelp (Zhang et al., 2015), SemEval 2017 Task 4 (SemEval, Rosenthal et al. (2017)), multi-domain sentiment analysis (MultiDom, Blitzer et al. (2007)), and PeerRead (Kang et al., 2018) with splits ACL 2017 and ICLR 2017 (Kang et al., 2018). Each dataset consists of multiple texts (e.g., tweets, articles), each labeled with an overall sentiment (e.g., positive ). Descriptive statistics for each dataset are shown in Tab. 2. For the datasets with more than three sentiment labels, we consider two versionsthe original and a version with only three (bucketed) sentiment labels. For each dataset, we transform each text into an average polarity value using either our representation, one of the six lexica, 4 or a straightforward combination thereof, where the polarity value for 4 We bucket the upper four and lower four points of VADER's nine-point scale, to yield a three-point scale. Without this bucketing, our representation outperforms VADER on four of the nine datasets. We do not bucket VADER when using it in SentiVAE or in the straightforward combination. each word in the (combined) vocabulary is a 16-dimensional vector that consists of a concatenation of polarity values. (Unlike SentiVAE, this concatenation does not yield a single sentiment lexicon that retains scale coherence, while achieving maximal coverage over words.) Specifically, we replace each token with its corresponding polarity value, and then average the these values (Go et al., 2009; Ozdemir and Bergler, 2015; Kiritchenko et al., 2014). We then use the training portion of the dataset to learn a logistic regression classifier to predict the overall sentiment of each text from its average polarity value. Finally, we use the testing portion to compute the accuracy of the classifier. Results. The results in Tab. 3 show that our representation using  w outperforms the individual lexica for all but one dataset, and that our representation using the mean of Q  w ( z w ) outperforms them for six datasets. This is likely because SentiVAE has a richer representation of sentiment than any individual lexicon, and it has greater coverage over words (see Tab. 4). The results in Tab. 5 support the former reason: even when we limit the words in our representation to match those in an individual lexicon, our representation still outperforms the individual lexicon. Unsurprisingly, our representation especially outperforms lexica with unidimensional scales. We also find that our representation outperforms the straightforward combination for datasets from domains that are not well supported by the individual lexica (see Tabs. 1 and 2 for lexicon and dataset sources, respectively). By combining lexica from different domains, our representation captures a general notion of sentiment that is not tailored to any specific domain. latent representation, and realized this model with a novel multi-view variational autoencoder, called SentiVAE. We then used SentiVAE to combine six commonly used English-language sentiment lexica with binary, categorical, and continuous scales. Via a downstream text classification task involving nine English-language sentiment analysis datasets, we found that our representation outperforms the individual lexica, as well as a straightforward combination thereof. We also found that our representation is particularly efficacious for datasets from domains that are not well-supported by standard sentiment lexica. Finally, we note that our approach is more general than SentiMerge (Emerson and De-clerck, 2014). While SentiMerge can only combine sentiment lexica with continuous scales, SentiVAE is designed to combine lexica with disparate scales. We would like to thank to Adam Forbes for the design of Fig. 1. We further acknowledge the support of the NVIDIA Corporation with the donation of the Titan Xp GPU used to conduct this research.
abstain abstain abstain method objective result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain other abstain abstain method result method method result abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain other method abstain result abstain other abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result result result abstain abstain method result result abstain abstain other other