summaries: The paper aims at extending GloVe word embedding model so that the resulting embeddings should capture sentiments (e.g. "good" is positive while "bad" is negative). The key idea is to employ an extension term to deal with the fact that some words appearing in text with sentiment information. Furthermore, to deal with the fact that many words an infrequent, besides maximum likelihood estimation, the paper proposes to use bayesian estimation. In the experiments, Stanford sentiment tree (SST) corpus is used. The word embeddings from the two models (each trained on different estimation methods) show their capability of expressing sentiments, compared with popular methods like Glove, word2vec.  This paper proposes a method to learn word embedding by incorporating additional sentiment information. The proposed method extends from D-GloVe by adding the probability of positive sentiment to the loss function. The paper presents three experiments: word similarity, word-level sentiment analysis, and sentence-level sentiment analysis. The experiments show that the method performs comparably with other baseline methods and outperforms in the low-frequency sentence setting (i.e. sentence containing lower frequency words). The paper proposed a word embedding model to incorporate the sentiment information. The paper provided both maximum likelihood estimation and maximum posterior estimation for the proposed framework. Improved experiment results on word similarity and low frequency embeddings are presented. Overall, the paper incorporates the sentiment information in a neat way. And my main concern is the around the Bayesian inference and the prior knowledge distilled into the model.  Detail comments are as following, 
abstractText: The word embedding models have achieved state-of-the-art results in a variety of natural language processing tasks. Whereas, current word embedding models mainly focus on the rich semantic meanings while are challenged by capturing the sentiment information. For this reason, we propose a novel sentiment word embedding model. In line with the working principle, the parameter estimating method is highlighted. On the task of semantic and sentiment embeddings, the parameters in the proposed model are determined by using both the maximum likelihood estimation and the Bayesian estimation. Experimental results show the proposed model significantly outperforms the baseline methods in sentiment analysis for low-frequency words and sentences. Besides, it is also effective in conventional semantic and sentiment analysis tasks.
references.authors: Ronan Collobert Jason Weston
references.title: A unified architecture for natural language processing: Deep neural networks with multitask learning
references.venue: In Proceedings of the 25th international conference on Machine learning,
references.authors: Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova
references.title: Bert: Pretraining of deep bidirectional transformers for language understanding
references.venue: arXiv preprint arXiv:1810.04805,
references.authors: Petar M Djuric Yufei Huang
references.title: Estimation of a bernoulli parameter p from imperfect trials
references.venue: IEEE Signal Processing Letters,
references.authors: Thomas S Ferguson
references.title: A bayesian analysis of some nonparametric problems
references.venue: The annals of statistics,
references.authors: Shoaib Jameel Steven Schockaert
references.title: D-glove: A feasible least squares model for estimating word embedding densities
references.authors: Shoaib Jameel Zihao Fu Bei Shi Wai Lam Steven Schockaert
references.title: Word embedding as maximum a posteriori estimation
references.venue: In Proceedings of the AAAI Conference on Artificial Intelligence,
references.authors: Yoon Kim
references.title: Convolutional neural networks for sentence classification
references.venue: arXiv preprint arXiv:1408.5882,
references.authors: Omer Levy Yoav Goldberg
references.title: Dependency-based word embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume
references.authors: Omer Levy Yoav Goldberg Ido Dagan
references.title: Improving distributional similarity with lessons learned from word embeddings
references.venue: Transactions of the Association for Computational Linguistics,
references.authors: Yang Li Quan Pan Tao Yang Suhang Wang Jiliang Tang Erik Cambria
references.title: Learning word representations for sentiment analysis
references.venue: Cognitive Computation,
references.authors: Zhanyu Ma Yuping Lai W Bastiaan Kleijn Yi-Zhe Song Liang Wang Jun Guo
references.title: Variational bayesian learning for dirichlet process mixture of inverted dirichlet distributions in non-gaussian image feature modeling
references.venue: IEEE transactions on neural networks and learning systems,
references.authors: Andrew L Maas Raymond E Daly Peter T Pham Dan Huang Andrew Y Ng Christopher Potts
references.title: Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologiesvolume 1, pages 142–150
references.venue: Association for Computational Linguistics,
references.authors: Tomas Mikolov Kai Chen Greg Corrado Jeffrey Dean
references.title: Efficient estimation of word representations in vector space
references.venue: arXiv preprint arXiv:1301.3781,
references.authors: Bo Pang Lillian Lee
references.title: Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd annual meeting on association for computational linguistics, pages 115–124
references.venue: Association for Computational Linguistics,
references.authors: Jeffrey Pennington Richard Socher Christopher Manning
references.title: Glove: Global vectors for word representation
references.venue: In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP),
references.authors: Matthew E Peters Mark Neumann Mohit Iyyer Matt Gardner Christopher Clark Kenton Lee Luke Zettlemoyer
references.title: Deep contextualized word representations
references.venue: arXiv preprint arXiv:1802.05365,
references.authors: Seuk-Yen Phoong Mohd Tahir Ismail
references.title: A comparison between bayesian and maximum likelihood estimations in estimating finite mixture model for financial data
references.venue: Sains Malaysiana,
references.authors: Trideep Rath
references.title: Word and Relation Embedding for Sentence Representation
references.venue: PhD thesis, Arizona State University,
references.authors: Bei Shi Zihao Fu Lidong Bing Wai Lam
references.title: Learning domain-sensitive and sentimentaware word embeddings
references.venue: arXiv preprint arXiv:1805.03801,
references.authors: Duyu Tang Furu Wei Bing Qin Ming Zhou Ting Liu
references.title: Building large-scale twitterspecific sentiment lexicon: A representation learning approach
references.venue: In Proceedings of coling 2014,
references.authors: Duyu Tang Furu Wei Bing Qin Nan Yang Ting Liu Ming Zhou
references.title: Sentiment embeddings with applications to sentiment analysis
references.venue: IEEE Transactions on Knowledge and Data Engineering,
references.authors: Anqi Wu Sebastian Nowozin Edward Meeds Richard E Turner José Miguel HernándezLobato Alexander L Gaunt
references.title: Deterministic variational inference for robust bayesian neural networks. 2018
references.authors: Zhilin Yang Zihang Dai Yiming Yang Jaime Carbonell Ruslan Salakhutdinov Quoc V Le
references.title: Xlnet: Generalized autoregressive pretraining for language understanding
references.authors: Liang-Chih Yu Jin Wang K Robert Lai Xuejie Zhang
references.title: Refining word embeddings using intensity scores for sentiment analysis
references.venue: IEEE/ACM Transactions on Audio, Speech, and Language Processing,
sections.heading: 1 Introduction
sections.text: Word embeddings provide continuous low-dimensional vector representations of words from documents (Li et al., 2017). Aiming to capture semantic and syntactic contextual information from large datasets, the word embedding models are extensively employed to represent words in natural language processing tasks (Levy and Goldberg, 2014). For this reason, many modelling methods are proposed to generate dense representations of words (Rath, 2017). Seeing the flourish of word embeddings, Word2vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) are considered as the edge-cutting approaches to deal with the word contexts. C&W is another most widespread method due to the progress in neural networks (Collobert and Weston, 2008). Besides, other algorithms are integrated into the existing models. For instance, Jameel and Schockaert proposes D-GloVe by combing GloVe and Dirichlet-Multinomial language modeling (Jameel and Schockaert, 2016). More recently, the contextualized word embedding models, which improve the accuracy to a large extent, are put forward. As such, the traditional approaches are concluded as pre-trained word embeddings. Whereas, the newly proposed methods, such as ELMo by Peters M. E. et al.(Peters et al., 2018) , BERT by Devlin J. et al.(Devlin et al., 2018) and XLNet by Yang Z et al.(Yang et al., 2019) , cost large amount of computing resource for training whilst obtain a better working performance in downstream tasks. In this way, the pre-trained word embeddings still hold a great promise in handling complicated natural language processing tasks. The aforementioned models are effective in dealing with semantic-oriented tasks. Likewise, in sentiment analysis, research is still ongoing to capture sufficient sentiment information while the sentiment embeddings typically depend on the sentiment polarity labels provided by labeled corpora to guide learning processes via objective functions (Yu et al., 2017). Tang et al. propose a method for learning sentiment embeddings by regulating the C&W model, which encodes sentiment information in the continuous word representations (Tang et al., 2015). By exploiting the prior knowledge, Li et al. incorporate the sentiment information to analyze the sentiment label of each word in target and contexts (Li et al., 2017). Maas et al. apply a semi-supervised method to get sentiment information and carry out the maximum likelihood estimation for parameter determination (Maas et al., 2011).
  Notwithstanding, the pre-trained word embeddings still have challenges in tackling sentiment analysis tasks, which are concluded as the following two aspects. On the one hand, the semantically similar word may have opposite sentiment polarities. Thus the sentiment polarity identification process has to be dedicatedly designed (Tang et al., 2015) (Li et al., 2017) (Shi et al., 2018). On the other hand, the capturing of sentiment information from low-frequency words is most pronounced. Typically, the low-frequency words can be regarded as the derivation of entity nouns, new terms and some deformation high-frequency words, which also contain significant semantic information. Nevertheless, due to the low frequency, current models are absent of processing their sentiment. The objective of this work is to devise a sentiment word embedding model. Specifically, the issue of parameter setting is deeply studied. Methods for effectively estimating the involving parameters based on Bayesian estimation and maximum likelihood estimation are proposed. For low-frequency word analysis, the Bayesian estimation is applied to determine the co-occurrence probabilities and the sentiment probabilities. This work describes current parameter estimation approaches and the model of GloVe in Section 2, illustrates our sentiment word embedding model in Section 3, shows the experiments in Section 4, and presents the research findings in Section 5.
sections.heading: 2 Preliminary
sections.text: This section introduces the basic theory related to parameter estimating algorithms and the GloVe model, so as to facilitate the description of subsequent model architecture.
sections.heading: 2.1 Parameter Estimating Algorithms
sections.text: Typically, word vectors are taken as learning variables in the word embeddings, which results in the use of parameter estimating algorithms. The way of establishing objective function is therefore be employed. According to (Tang et al., 2014), the objective function of the Skip-Gram model is to maximize the average log probability, which is expressed as:
  J = 1
  T T∑ i=1 ∑ −c⩽j⩽c,j ̸=0 ln p (wi+j | ei) (1)
  where T is the number of words in the corpus and c indicates the size of window. We take ei as the embedding of target word wi and wi+j as the context of wi. The outcome p (wi+j |ei) is obtained via the hierarchical softmax. Similarly, the objective of GloVe refers to the maximum of likelihood probability and is defined as (Jameel et al., 2019):
  J = ∏ i,j N ( lnxij ;wi · w̃j + bi + b̃j , σ2 )
  (2)
  where N ( .;µ, σ2 ) represents the normal distribution with the mean µ and the variance σ2.
  In GloVe, the variance is determined by each word couple (i, j). In addition to objective function constructing, the estimation algorithms are applied to compute other parameters within word embeddings. In (Maas et al., 2011), the maximum posterior probability estimation identifies the parameter to weigh the semantic information (Maas et al., 2011). In D-GloVe, Jameel and Schockaert also use the semantic information weighing parameter, whose value corresponds to the Bayesian estimating outcome (Jameel and Schockaert, 2016).
sections.heading: 2.2 The GloVe model
sections.text: Basically, the GloVe model is a word-embedding method that combines evidence from the local context and the global counts. Typically, three distinguished words are used in this
  model, which are Wi,Wj and Wk. Both Wi and Wj are target words while Wk is the context. Let x be the matrix representing the word-word co-occurrence counts. We define the element xik as the times for word Wk appearing in the context of Wi. Correspondingly, xi = ∑ k xik indicates the frequency of each word occurs in the context of Wi. The cooccurrence probability of Wk being the context word of Wi is given as
  Pik = P (Wk|Wi) = xik/xi (3)
  The parameter Pik/Pjk is taken to determine the relation of Wi to Wk and Wj to Wk. For Wk has a similar relation to Wi and Wj , i.e. both relevant or irrelevant, the ratio approaches 1. The information in the ratio of co-occurrence probabilities is:
  F ( wTi w̃k − wTj w̃k ) = Pik/Pjk (4)
  where wϵRn refers to the target word vector and w̃ϵRn to the context vector. Commonly, GloVe extracts the semantic relation between different words by using the ratio of cooccurrence probabilities while the semantic information are identified via the maximum likelihood estimation (Maas et al., 2011).
sections.heading: 3 Methodology
sections.text: This section depicts the architecture of the sentiment word embedding, working principle of the parameter estimating process using two different estimation algorithms.
sections.heading: 3.1 Sentiment Word Embedding Model Architecture
sections.text: In sentiment analysis tasks, the sentiment information is captured during processing. Aiming to identify the sentiment polarities of different words, a word embedding model, incorporating the sentiment information, is established. Typically, we tend to characterize the proposed model by the loss function. To compute the sentiment embeddings, we define the probability of Wi being positive as Bi and negative as 1−Bi. Assuming that Wi = good and Wj = bad, the value of Bi/Bj is larger than 1, which indicates good is more positive than bad. In turn, the value (1−Bi) / (1−Bj) is less than 1 since bad shows a negative polarity. In this way, the relations of the word sentiment are expressed by the ratio of sentiment probabilities. For Bi+(1−Bi) = 1, Bi/Bj and (1−Bi) / (1−Bj) make the same sense in conveying the sentiment, we take Bi/Bj to construct the sentiment relation of Wi and Wj . More details of the words’ relation and the ratios are presented in Appendix 1. Considering the bias vector corresponds to positive sentiment polarity, we take sϵRn to indicate the bias vector to match the size of word vector. By transforming Wi and Wj into word vectors wi and wj , the difference established upon si and sj is written as:
  F (wTi si − wTj sj) = Bi/Bj (5)
  Assuming that F is confirming to the homomorphisms between groups (R,+) and (R>0,×), the semantic and sentiment information is combined to get:
  F (( wTi w̃k − wTj w̃k ) + ( wTi si − wTj sj )) =F ( wTi w̃k − wTj w̃k ) · F ( wTi si − wTj sj
  ) = Pik Pjk · Bi Bj
  (6)
  Due to properties of group homomorphism, eqn.6 is transformed into
  F ( wTi w̃k − wTj w̃k + wTi si − wTj sj ) =F (( wTi w̃k + w T i si ) − ( wTj w̃k + w T j sj ))
  = F ( wTi w̃k + w T i si )
  F ( wTj w̃k + w T j sj ) = Pik Pjk · Bi Bj
  (7)
  in line with
  F ( wTi w̃k + w T i si ) = Pik ·Bi (8)
  According to eqn.7,the basic objective function F is in the form of exponential, that is F (x) = exp (x). Thus, we apply the logarithm operation to each side and have:
  wTi w̃k + w T i si = ln (Pik ·Bi) = lnPik + lnBi (9)
  By incorporating the sentiment information, the loss function of the word embedding model is defined as
  loss (wi, w̃k, si) = V∑ i,k=1 [ wTi w̃k + w T i si − lnPik − lnBi ]2 (10) where V indicates the size of the vocabulary. The parameters wTi , w̃k and si are computed via gradient descent algorithms.
sections.heading: 3.2 Incorporating Sentiment Information
sections.text: As pointed out in the Introduction, current models use the maximum likelihood estimating algorithm for parameter determination. In this part, we preliminarily carry out the parameter estimation based on the maximum likelihood principle. For each target word Wi, xi times Bernoulli experiments are conducted to extract the context independently with V different outcomes in each experiment (Djuric and Huang, 2000). The occurrence number of the kth outcome and its probability are represented by xik and Pik. If the random variable Xi = (Xi1, Xi2, · · ·, XiV ) stands for the occurrence times of all the possibilities, i.e. Xik is the number for the kth one, the parameter Xi obeys the Multinomial distribution, i.e. Xi ∼ Multinomial
  (−→xi ,−→Pi) with −→Pi = (Pi1, Pi2, · · · , PiV ) and −→xi = (xi1, xi2, · · · , xiV ). Hence, a log-likelihood function is constructed:
   max Pi1,Pi2,··· ,Pik,··· ,PiV lnL(Pi1, Pi2, · · · , Pik, · · · , PiV ) = max Pi1,Pi2,··· ,Pik,··· ,PiV ln [(Pi1)xi1 · (Pi2)xi2 · · · (Pik)xik · · · (PiV )xiV ] = max Pi1,Pi2,··· ,Pik,··· ,PiV V∑ k=1 xik · lnPik s.t. V∑ k=1 Pik = 1
  (11)
  According to eqn.11, the objective function can be resolved as an optimal problem that equality constraints. Thus, the corresponding Lagrangian function is formulated as
  J (Pi1, Pi2, · · · , PiV , λ) = V∑
  k=1
  xik · logPik + λ ( 1−
  V∑ k=1 Pik
  ) (12)
  where we have Pik = xikλ determined by ∂J(Pi1,Pi2,··· ,PiV ,λ) ∂Pik = xikPik − λ = 0. Likewise, λ = ∑V k=1 xik = xi is calculated with respect to ∑V k=1 Pik = ∑V k=1 ( xik λ ) = ∑V
  k=1 xik/λ = 1. Thus, the estimation of Pik is written as
  P̂ik = xik/xi (13)
  Notably, the obtained Pik is the same with that from GloVe according to eqn.3, which demonstrates the feasibility for parameter estimation in our model. In this way, the outcome of parameter sentiment probability can also be computed by using the maximum likelihood estimator. As such, a maximum likelihood estimation-based sentiment word embedding, namely MLESWE, is put forward. In this case, the Bernoulli experiments are applied to pick up the sentiment polarity of the target word Wi and the outcome can be either positive or negative. Since Bi is the probability of Wi being positive, we designate the distribution of Wi obeys−→ ti = (ti0, ti1) where ti0 is the number of negative texts and ti1 indicates that of the positive ones. Thus, the total number of texts including Wi is expressed as ti = ti0 + ti1. Support a random variable Ti = (Ti0, Ti1) denotes the times of all the possibilities of outcomes and Ti conforms to the binomial distribution, i.e. Ti ∼ Binomial (−→ ti , −→ Bi ) where −→Bi = (Bi, 1−Bi).
  The log-likelihood function of sentiment probabilities is delivered as:
  max Bi lnL (Bi) = max Bi
  ln [ (Bi) ti1 · (1−Bi)ti0 ] = max
  Bi [ti1 · lnBi + ti0 · ln (1−Bi)] (14)
  Similarly, B̂i = ti1ti is obtained based on ∂(ln L) ∂Bi = ti1Bi − ti0 1−Bi = 0. Combining the semantic and sentiment information, the final loss function based on maximum likelihood principle is
  loss (wi, w̃k, si) = V∑ i,k=1 [ wTi w̃k + w T i si − ln xik xi − ln ti1 ti ]2 (15)
sections.heading: 3.3 Parameter Estimating using Bayesian Estimation
sections.text: The Bayesian estimating method is highlighted due to its not sensitive to initialization via proper prior distributions to parameters (Ma et al., 2018). By using the prior knowledge, the deficiency of lacking information of small datasets can be resolved, which leads to the converge to the actual value (Phoong and Ismail, 2015). Accordingly, the generalization ability of the model can be improved (Wu et al., 2018). The Bayesian approach, in this way, is able to present an elegant solution for automatically determining the parameters (Ferguson, 1973). We thus employ the Bayesian estimation for the parameter estimating of the proposed model. The Bayesian estimation-based sentiment word embedding, namely BESWE, is performed. In accordance to the assumption of maximum likelihood principle mentioned before, the prior distribution P (−→ Pi ) is assumed to obey the Dirichlet distribution of −→α = (αi, αi, · · · , αV ).
  The prior distribution is converted to P (−→ Pi ) = Dir (−→α ) = Γ( ∑ k αk)∏
  k Γ(αk)
  ∏ k P αk−1 ik , with the
  identical likelihood function:
  P (−→x i|−→P i) = Mult(−→x i,−→P i) = xi!∏V
  k (xik!) · V∏ k P xikik (16)
  Considering the Dirichlet-Multinomial conjugate structure, the posterior distribution is
  P (−→ P i|−→x i ) = Dir (−→α +−→x i) = Γ ( ∑
  k αk + xik)∏ k Γ (αk + xik) · ∏ k Pαk+xik−1ik (17)
  where αk = λ1 · nk∑ k nk
  , nk is the total number of occurrences of word Wk in the corpus and λ1 > 0 is determined by tuning data. By satisfying
  cik = EP (−→ P i|−→x i ) [lnPik] (18) we compute the Bayesian estimating outcome of lnPik in the loss function provided by eqn.10, which is also the mean value of posterior probability in line with Pik. As stated in (Jameel and Schockaert, 2016), the computation of E
  P (−→ P i|−→x i ) [lnPik] is facilitated via Taylor expansion:
  E P (−→ P i|−→x i ) [lnPik] ≈ lnEP(−→P i|−→x i) [Pik]− V ar
  P (−→ P i|−→x i ) [Pik] 2 · E2
  P (−→ P i|−→x i ) [Pik] (19)
  where we have V ar P (−→ P i|−→x i ) [Pik] = αk+xik∑ k(αk+xik)
  · ( 1− αk+xik∑
  k(αk+xik)
  ) · 1∑
  k(αk+xik)+1 and
  E P (−→ P i|−→x i ) [Pik] = αk+xik∑ k(αk+xik)
  . Note that lnPik is estimated via Bayesian principle in eqn.18 whose form is unlike that of eqn.13. Comparing to the maximum likelihood estimation, a direct outcome is obtained without using Laplace smoothing in experiment. Comparatively, P (−→ Bi ) is designed to obey Beta distribution with the parameter −→β = (β0, β1), along with
  the prior distribution given as
  P (−→ B i ) = Beta (−→ β ) = Γ (β0 + β1)
  Γ (β0) Γ (β1) (1−Bi)β0−1 ·Bβ1−1i (20)
  from which the log-likelihood function is
  P (−→ t i| −→ B i ) = b (−→ t i, −→ B i ) = Cti1ti · (1−Bi) ti0 ·Bti1i (21)
  and the posterior distribution subject to the conjugate structure of Beta-Binomial is
  P (−→ B i| −→ t i ) = Beta (−→ β + −→ t i ) = Γ (β0 + ti0 + β1 + ti1)
  Γ (β0 + ti0) Γ (β1 + ti1) · (1−Bi)β0+ti0−1 ·Bβ1+ti1−1i
  (22)
  where mk stands for the texts of the sentiment label k, λ2 > 0 is a parameter depending on tuning data and βk = λ2 · mk∑
  k mk . Thereupon, to determine the lnBi in eqn.10, we take
  the Bayesian estimation approach. The solution to the posterior probability expectation of lnBi, which is involved with Bi is characterized as
  ei = EP (−→ B i| −→ t i ) [lnBi] (23) Furthermore, the Taylor expansion is employed to update the equation:
  E P (−→ B i| −→ t i ) [lnBi] ≈ lnEP(−→B i|−→t i) [Bi]− V ar
  P (−→ B i| −→ t i ) [Bi] 2 · E2
  P (−→ B i| −→ t i ) [Bi] (24)
  where we have V ar P (−→ B i| −→ t i ) [Bi] = β1+ti1∑ k(βk+tik)
  · ( 1− β1+ti1∑
  k(βk+tik)
  ) · 1∑
  k(βk+tik)+1 and
  E P (−→ B i| −→ t i ) [Bi] = β1+ti1∑ k(βk+tik) . Hence, the final loss function of BESWE can be obtained:
  loss (wi, w̃k, si) = V∑ i,k=1 [ wTi w̃k + w T i si − cik − ei ]2 (25)
sections.heading: 4 Experiments
sections.text: In this section, the working performance of BESWE and MLESWE are evaluated. The task of word similarity analysis is carried out. To deliver the sentiment embeddings, both wordand sentence-level sentiment analysis using different models is conducted.
sections.heading: 4.1 Experiment Settings
sections.text: Datasets. The dataset SST (Stanford Sentiment Tree) is employed for the mode training. There are five classes annotations within SST, which are very negative, negative, neutral, positive and very positive. Typically, we assign the value 3 and 4 to represent the positive polarity, 0 and 1 to negative and 2 to else. For our models, the word representation dimension is 50, the learning rate is 0.05 and the iteration number is 50. Besides, the loss function is optimized with the deployment of AdaGrad. Baseline Methods. We evaluate the proposed model in comparison to other state-of-theart models. The models of word embeddings, such as C&W, word2vec and GloVe, together with models of sentiment embeddings, including SE-HyRank and DLJT2, are implemented. For the baseline methods, we use default settings in the provided implementations or described as their papers and the word representation dimension is 50. Word Similarity. Computing word similarity (Levy et al., 2015) aims to capture the general meanings of words. In this research, the word similarity tasks are conducted on the dataset EN-WS-353-ALL, EN-WS-353-SIM and SCWS, which are detailed illustrated in (Jameel et al., 2019). Word-level Sentiment Analysis. We conduct word-level sentiment analysis on two sentiment lexicons, namely MPQA and NRC. The number of positive and negative items for MPQA is 2301 and 4151 while for NRC is 2231 and 3324. The N-fold cross validation with N=5 and N=10 is performed. An SVM classifier is trained whose average accuracy is the evaluation metric. Specifically, the words from SST corpus are extracted and converted into word embeddings, which are taken as the features of SVM.
  As Bayesian estimating principle is capable of tackling low-frequency words, we distinctively pick up the words with a frequency less than 5 for analysis. Statistically, the SST corpus contains 9984 low-frequency words. Sentence-level Sentiment Analysis. Considering the sentiment analysis for sentence, the movie review polarity datasets MovieReview is employed (Pang and Lee, 2005), which contains 10622 samples with the proportion of each polarity 1:1. We use a convolutional neural network (CNN) model, namely Text-CNN, with its online implementation (Kim, 2014). Likewise, the inputs of Text-CNN are word embeddings as well. The training episode is set as 200 epochs using the default settings. Similarly, we also pick the low-frequency words with the occupation over 10% as the lowfrequency sentences for testing. There are totally 1258 sentences cater to the demands and are all sent to Text-CNN classifier for processing.
sections.heading: 4.2 Experimental Results
sections.text: Word Similarity. On the task of working performance evaluation, we first present the results for of word similarity analysis (Fig. 1). It can be observed BESWE outperforms other algorithms on all datasets, indicating that our model is capable to capture sufficient semantic information. Distinctively, the implementation of MLESWE, although not as good as BESWE, still achieves a better result on the average accuracy (i.e. Ave_Acc in Fig. 1) than the original GloVe. Yet the maximum likelihood estimating algorithm can also be applied to parameter determination of the word embeddings.
  Word-level Sentiment Analysis Results. The word-level sentiment analysis task is conducted on the dataset of single-word entries. The DLJT2 model outperforms other word embedding models by incorporating sentiment information into the learning processes, as shown in Fig.2. Compared to the state-of-the-arts, our model fails to exceed the outcome of the best method on average accuracy. Encouragingly, the BESWE model shows an even better performance in tackling the low-frequency words.
  Sentence-level Sentiment Analysis Results. The working performance of the proposed model is further evaluated on the sentence-level sentiment analysis task. From Fig.3, we see that SE-HyRank has a better performance than any other algorithms in average accuracy. Clearly, the outcome of BESWE is anyhow decent which is comparable with that of SEHyRank. Regarding low-frequency sentences, the minimum performance gap of over 9% against SE-HyRank is reported. Consequently, for the sentiment analysis of low-frequency words or low-frequency sentences, BESWE always obtain the best and most consistent results in the identification of sentiment polarity.
  Effects of λ1 and λ2. The hyperparameters in BESWE, i.e. regulatory factors λ1 and λ2, are used to represent the semantic and the sentiment information. The settings of the involving parameter can be therefore determined. The values of λ1 and λ2 are varied within the collection of {1, 0.75, 0.5, 0.25, 0.1, 0.05, 0.02, 0.01}. Firstly, the value of λ1 is set as {1, 0.75, 0.5, 0.25, 0.1, 0.05, 0.02, 0.01}. When λ2 = 1, we name BESWE as BESWE#1 and λ2 = 0.75 as BESWE#2, and so on so forth. Correspondingly, the value of λ2 is also picked from the same set and named from BESWE#9 to BESWE#16 in the same order. Totally, we get 16 different models. The results on the sentence-level sentiment analysis against different hyperparameter settings are shown in Fig.4(a) and Fig.4(c). Likewise, the results for low-frequency sentences are in Fig.4(b) and Fig.4(d). We take LowFreSentence#n to nominate the outcomes from low-frequency sentences.
  The sentence-level sentiment analysis reaches the highest accuracy 71.64% at the point λ1 = 0.05 and λ2 = 0.02. For the analysis of low-frequency sentences, the optimal values of λ1 and λ2 are 0.1 and 0.01, which results in an accuracy of 79.92%.
  The experimental results verify the effectiveness of the proposed sentiment word embedding. The BESWE model outperforms other state-of-the-art in word similarity identification. In the sentiment analysis of both word level and sentence level, our method still presents comparable outcomes. Specifically, by integrating the prior knowledge into sentiment probabilities estimating, the BESWE model is a better alternative for low-frequency-word sentiment capturing. It is reasonable to expect better performance in sentiment analysis tasks, as it is the case.
sections.heading: 5 Conclusion
sections.text: In this work, the designing and deploying of the sentiment word embeddings is deeply studied. On the foundation of current word embedding models, the estimation principle of the objective function, together with other parameters, are investigated. Motivated by the significance of sentiment information, a novel word embedding model for sentiment analysis is established. Within the proposed model, both semantic and sentiment information is integrated into the word vectors. Aiming to construct the objective function, the group homomorphism theory is applied. As for the parameter determination, the maximum likelihood estimator and the Bayesian estimator are employed. Experiments are conducted on various tasks to evaluate the working performance. In comparison to the baseline models, our model is capable of tackling word similarity tasks. For the purpose of sentiment embeddings representation, the proposed model is effective in word-level and sentence-level sentiment analysis. Specifically, it outperforms other methods on low-frequency words and sentences sentiment polarity identification to demonstrate its efficacy.
sections.heading: A Appendix I
sections.text: For Wi = good and Wj = bad, we have Bi⁄Bj > 1, (1−Bi) / (1−Bj) < 1,
  i.e. Wi = good is more positive than Wj = bad. For Wi = good and Wj = great, we have Bi/Bj ≈ 1,
  (1−Bi) / (1−Bj) ≈ 1, i.e. Wi = good and Wj = great are of positive polarities.
  For Wi = then and Wj = home, we have Bi/Bj ≈ 1,
  (1−Bi) / (1−Bj) ≈ 1, i.e. Wi = then and Wj = home are of neutral polarities.
  The specific sentiment probabilities calculated by maximum likelihood estimation are presented in Table 1.
  Likewise, the outcomes based on Bayesian estimation are in Table 2.
title: A Novel Bayesian Estimation-Based Word Embedding Model For Sentiment Analysis