Leveraging domain knowledge is an effective strategy for enhancing the quality of inferred low-dimensional representations of documents by topic models. In this paper, we develop topic modeling with knowledge graph embedding (TMKGE), a Bayesian nonparametric model to employ knowledge graph (KG) embedding in the context of topic modeling, for extracting more coherent topics. Specifi-cally, we build a hierarchical Dirichlet process (HDP) based model to flexibly borrow information from KG to improve the interpretability of topics. An efficient online variational inference method based on a stick-breaking construction of HDP is developed for TMKGE, making TMKGE suitable for large document corpora and KGs. Experiments on three pub-lic datasets illustrate the superior performance of TMKGE in terms of topic coherence and document classification accuracy, compared to state-of-the-art topic modeling methods. Topic models, such as Probabilistic Latent Semantic Analysis (PLSA) (Hofmann, 2017) and Latent Dirichlet Allocation (LDA) (Blei et al., 2003), play significant roles in helping machines interpret text documents. Topic models consider documents as a bag of words. Given the word information, topic models formulate documents as mixtures of latent topics, where these topics are generated via the multinomial distributions over words. Bayesian methods are utilized to extract topical structures from the document-word frequency representations of the text corpus. Without supervision, however, it is found that the topics generated from these models are often not interpretable (Chang et al., 2009; Mimno et al., 2011). In recent studies, incorporating knowledge of different forms as a supervision has become a powerful strategy for discovering meaningful topics (Andrzejewski et al., 2009). Most conventional approaches take prior domain knowledge into account to improve the topic coherence (Andrzejewski et al., 2009; Andrzejewski and Zhu, 2009; Hu et al., 2014; Jagarlamudi et al., 2012; Doshi-Velez et al., 2015). One commonly used domain knowledge is based on word correlations (Andrzejewski et al., 2009; Chen et al., 2013; Chen and Liu, 2014). For example, must-links and cannot-links among words are generated by domain experts to help topic modeling (Andrzejewski et al., 2009). Another useful form of knowledge for topic discoveries is based on word semantics (Andrzejewski and Zhu, 2009; Chemudugunta et al., 2008; Hu et al., 2014; Jagarlamudi et al., 2012; Doshi-Velez et al., 2015). In particular, word embedding (Pennington et al., 2014; Goldberg and Levy, 2014), in which bag of words are transformed into vector representations so that contexts are embedded into those word vectors, are used as semantic regularities to enhance topic models (Nguyen et al., 2015; Li et al., 2016; Das et al., 2015; Batmanghelich et al., 2016). Knowledge graph (KG) embedding (Bordes et al., 2013) learns a low-dimensional continuous vector space for entities and relations to preserve the inherent structure of KGs. Yao et al. (2017) proposes KGE-LDA to incorporate embeddings of KGs into topic models to extract better topic representations for documents and shows promising performance. However, KGE-LDA forces words and entities to have identical latent representations, which is a rather restrictive assumption that prevents the topic model from recovering correct underlying latent structures of the data, especially in scenarios where only partial KGs are available. This paper develops topic modeling with knowledge graph embedding (TMKGE), a hierarchical Dirichlet process (HDP) based model to extract more coherent topics by taking advantage of the KG structure. Unlike KGE-LDA, the proposed TMKGE allows for more flexible sharing of information between words and entities, by using a multinomial distribution to model the words and a multivariate Gaussian mixture to model the entities. With this approach, we introduce two proportional vectors, one for words and one for entities. In contrast, KGE-LDA only uses one, shared by both words and entities. Similar to HDP, TMKGE includes a collection of Dirichlet processes (DPs) at both corpus and document levels. The atoms of corpus-level DP form the base measure for document levels DPs of words and entities. Therefore, the atoms of corpus-level DP can represent word topics, entity mixture components, or both of them. Figure 1 provides an overview of TMKGE, where two sources of inputs, bag of words and KG embedding, extracted from corpus and KGs respectively, are passed into TMKGE. As a nonparametric model, TMKGE does not assume a fix number of topics or entity mixture components as constraints. Instead, it learns the number of topics and entity mixture components automatically from the data. Furthermore, an efficient online variational inference algorithm is developed, based on Sethuraman's stick-breaking construction of HDP (Sethuraman, 1994). We in fact construct stick-breaking inference in a mini-batch fashion (Wang et al., 2011; Bleier, 2013), to derive a more efficient and scalable coordinate-accent variational inference for TMKGE. Summary of contributions: TMKGE is a Bayesian nonparametric model to extract more coherent topics by taking advantage of knowledge graph structures. We introduce two proportional vectors for more flexible sharing of information between words and entities. We derive an efficient and scalable parameter estimation algorithm via online variational inference. Finally, we empirically demonstrate the effectiveness of TMKGE in topic discovering and document classification. Latent Dirichlet Allocation (LDA) (Blei et al., 2003) is a popular probabilistic model that learns latent topics from documents and words, by using Dirichlet priors to regularize the topic distributions. The generated topics from LDA models, however, are often not interpretable (Chang et al., 2009; Mimno et al., 2011), in part because LDA models are unsupervised without using prior knowledge or external resources. In recent years, prior knowledge are leveraged to guide the process of topic modeling (Andrze-jewski and Zhu, 2009; Hu et al., 2014; Jagarlamudi et al., 2012; Doshi-Velez et al., 2015). For example, the deep forest LDA (DF-LDA) model (Andrzejewski et al., 2009) is proposed to incorporate must-links and cannot-links among words into topic modeling. One weakness of the DF-LDA model is that the link information is domain-dependent. Later, general knowledge based LDA is introduced to leverage must-links from multiple domains (Chen et al., 2013). More recently, MetaLDA (Zhao et al., 2017) proposes to improve topic modeling by incorporating diverse meta information as priors for both document hyperparameter  and word hyperparameter  . Besides the word correlations, word semantics are also utilized as one type of useful knowledge for topic modeling (Chemudugunta et al., 2008; Hu et al., 2014; Jagarlamudi et al., 2012). Word embeddings, as a low-dimensional continuous vectors of words (Mikolov et al., 2013; Bengio et al., 2003; Pennington et al., 2014) are regarded to be an efficient representations of word semantics. Latent Feature Topic Modeling (LFTM) is proposed to use pre-trained word embeddings in topic modeling (Nguyen et al., 2015). It incorporates the embedding of a word and its topics into the traditional multinomial distribution over words as the probability function of topic modeling. Top-icVec extends LFTM by combining a word and its local contextual words together into the conventional multinomial distribution over words. It also learns embedding representations for topics (Li et al., 2016). Gaussian-LDA goes further to improve topic modeling (Das et al., 2015) by taking into considerations the continuous nature of word embeddings. Shi et al. (2017) constructs a more unified framework, STE (skip-gram topic embedding) to address the problem of polysemy. Li et al. (2019) proposes a unified framework TMSA (Topic Modeling and Sparse Autoencoder) to improve topic discovery and word embedding simultaneously via a mutual learning mechanism. Hu et al. (2016) proposes topic-based embeddings for learning from large knowledge graphs (KGE). KGE learns low-dimensional continuous vector space for both entities and relations to preserve the inherent structure of knowledge graphs. A Bayesian method is introduced by considering the embeddings of entities and relations as topics. Later, Yao et al. (2017) proposes knowledge graph embedding LDA (KGE-LDA) to encode entity embeddings learned from knowledge graphs into LDA and show that knowledge graph embeddings boost topic discoveries. Inspired by this work, we explore to utilize entity embeddings to encode prior knowledge for topic modeling. This section presents the TMKGE model and an efficient online variational inference for learning its parameters. We first provide a review of hierarchical Dirichlet process (HDP) (Teh et al., 2005). Dirichlet process (DP) (MacEachern and Muller, 1998) G  DP (  0 , G 0 ) , with a base measure G 0 and a concentration parameter  0 > 0 , is the distribution of a random probability measure G over a measurable space ( , B ) , such that for any measurable disjoint partition ( A 1 , ..., AQ ) of  , ( G ( A 1 ) , ..., G ( AQ ))  Dir (  0 G 0 ( A 1 ) , ...,  0 G 0 ( AQ )) Hierarchical Dirichlet process (HDP) (Teh et al., 2005), introduced for dealing with multiple ( D ) groups of data, is a distribution over a set of random probability measures over ( , B ) : one probability measure G d  DP (  0 , G 0 ) for each group d  { 1 , 2 , ..., D } , and a global probability measure G 0  DP (  0 , H ) with a base measure H . Stick-breaking construction Teh et al. (2005) shows that the draws from G 0 and G d can be expressed as weighted sums of point masses: G 0 =  (cid:88) k =0  k   k , G d =  (cid:88) k =0  dk   k . A more convenient stick-breaking construction, especially for deriving closed-form variational inference (Wang et al., 2011), is Sethuraman (1994)'s construction, which proceeds as follows. First, the global-level DP draw is represented as  (cid:48) k  Beta (1 ,  0 ) ,  k =  (cid:48) k k  1 (cid:89) (cid:96) =1 (1   (cid:48) (cid:96) ) , Note that the distribution for  = {  k }  k =1 is also commonly written as   GEM (  0 ) (Pit-man, 2002). Subsequently, the group-level draws are constructed as  dt  G 0 ,  (cid:48) dt = Beta (1 ,  0 ) ,  dt =  (cid:48) dt t  1 (cid:89) (cid:96) =1 (1   (cid:48) d(cid:96) ) , G d =  (cid:88) t =1  dt   dt . Alternatively, the group-level atoms {  dt }  t =1 can be represented as  dt =  c dt , where the auxiliary indicator variables c dt are independently drawn from a multinomial Mult (  ) . Teh et al. (2008) also proposes a collapsed inference method as an alternative of stick-breaking inference. However, following Fox et al. (2011), we stick to the uncollapsed HDP model considering our truncated Dirichlet process has more computational efficiency and is simple to implement. Figure 2 is the graphical representation of TMKGE. Let D denote the number of documents in the corpus, where each document d  { 1 , 2 , ..., D } contains N ( w ) d words and N ( e ) d entities. Throughout this work, superscripts ( w ) and ( e ) indicate word and entity related parameters, respectively. In each document d , the n -th word is represented by w dn , where each word belongs to a vocabulary of size V , i.e., w dn  { 1 , 2 , ..., V } . Furthermore, the P -dimensional embedding of the m -th entity is e dm , where the total number of unique entities in the corpus is E . We assume that entity embeddings are obtained from the com-plete knowledge graph, and hence they contain information independent of the corpus. In this paper, we use TransE (Bordes et al., 2013), a simple and effective tool for knowledge encoding, to calculate the embeddings of entities extracted from the documents. We should mention that we remove the normalization step of TransE and thus the output vectors ( e dm ) do not have unit (cid:96) 2 norm. TMKGE builds upon HDP for joint modeling of word topics and entity mixtures. At the corpus level, word topics and entity mixtures correspond to atoms of a Dirichlet process G 0  DP (  0 , H ) . At the document level, word topics and entity mixture components are atoms of independent DPs, with shared base measure G 0 . Mathematically, for document d , we have G ( w ) d  DP (  0 , G 0 ) , G ( e ) d  DP (  0 , G 0 ) , where G ( w ) d and G ( e ) d are word and entity related DPs. Sethuraman's construction in (1) yields G ( w ) d =  (cid:88) t =1  ( w ) dt   ( w ) dt , G ( e ) d =  (cid:88) t =1  ( e ) dt   ( e ) dt . (2) These DPs are then used to assign words and entities to topics and mixture components, respectively. Using the mixing proportions of DPs in (2), we have p ( z ( w ) dn = t ) =  ( w ) dt , p ( z ( e ) dn = t ) =  ( e ) dt . In document d , let z ( w ) dn denote the topic assigned to the n -th word, and z ( e ) dm denote the mixture component assigned to the m -th entity. For simplicity, we use index t to denote both word and entity related atoms, although they can correspond to different atoms of the global DPs. The mixing proportions of corpus-level DP are used to map the document atoms to the shared global atoms. More precisely, we introduce the word and entity atoms mapping auxiliary variables c ( w ) d = { c ( w ) dt }  t =1 and c ( e ) d = { c ( e ) dt }  t =1 . TMKGE allows flexible sharing of information between knowledge graphs and documents. This is an important advantage, as in practice only partial relational information are available, and thus strictly forcing the topics and entity mixtures to share components may lead to reducing the power of model to correctly recover the latent structure of the data. Furthermore, the nonparametric nature of the model enables the automatic discovery of number of atoms for both words and entities, at document and corpus levels. Each atom of corpus DP ( G 0 ) corresponds to a set of parameters for both words and entities. Atom k contains topic-word Dirichlet distribution  k = (  k 1 , ...,  kV ) T , and entity Gaussian mixture parameters {  k ,  k } . Given  k and topic assignment variables, the generative process for n -th word of document d is z ( w ) dn  Mult (  ( w ) d ) , ( w dn | z ( w ) dn = t, c ( w ) dt = k,  k )  Mult (  k ) . where  k and  k are the mean vector and precision matrix of multivariate Gaussian distribution. Furthermore, we impose conjugate priors on both word and entity components parameters as:  k  Dir ( , ...,  ) ,  k  N (cid:0) m 0 , (  0  k )  1 (cid:1) ,  k  Wishart (  0 , W 0 ) . In this section, inspired by (Wang et al., 2011), we propose an online variational inference algorithm for efficient learning of TMKGE model parameters. We use a fully factorized variational distribution based on stick-breaking construction, and perform online mean-field variational inference. In addition to topic parameters  k and entity mixture parameters {  k ,  k } , other parameters of interest are corpus-level stick proportions  (cid:48) = {  (cid:48) k }  k =1 , document-level stick proportions for words  (cid:48) ( w ) d = {  (cid:48) ( w ) dt }  t =1 and entities  (cid:48) ( e ) d = {  (cid:48) ( e ) dt }  t =1 , topic assignments for words z ( w ) d = { z ( w ) dn } N ( w ) d n =1 , mixture assignments for entities z ( e ) d = { z ( e ) dm } N ( e ) d m =1 , and mapping variables c ( w ) d and c ( e ) d . Denote  ( w ) and  ( e ) respectively the word and entity related parameters. Then the variational distribution factorizes as q (  (cid:48) ,  ( w ) ,  ( e ) ) = q (  (cid:48) ) q ( ( w ) ) q ( ( e ) ) . For corpus-level stick proportions, we assume a Beta distribution: q (  (cid:48) ) = K  1 (cid:89) k =1 Beta (  (cid:48) k | u k , v k ) , where the number of global atoms is truncated at K , thereby q (  (cid:48) K = 1) = 1 . For the word related parameters  ( w ) , we have q ( ( w ) ) = q ( c ( w ) ) q ( z ( w ) ) q (  (cid:48) ( w ) ) q (  ) , q ( c ( w ) ) = D (cid:89) d =1 T  1 (cid:89) t =1 Mult (  ( w ) dt ) , q ( z ( w ) ) = D (cid:89) d =1 N ( w ) d (cid:89) n =1 Mult (  ( w ) dt ) , q (  (cid:48) ( w ) ) = D (cid:89) d =1 T  1 (cid:89) t =1 Beta (  (cid:48) dt | a ( w ) dt , b ( w ) dt ) , q (  ) = K (cid:89) k =1 Dir (  k ) . The variational distributions for entity related parameters have a similar form to the above distributions, except the Gaussian mixture parameters, which are expressed as follows: q (  k ) = N (cid:0) m k , (  k  k )  1 (cid:1) , q (  k ) = Wishart (  k , W k ) . In standard variational inference theory, the evidence lower bound (ELBO), which is the lower bound to the marginal log likelihood of the observed data, is maximized to find the best variational approximation to the true intractable posterior. Given the modeling framework of TMKGE, the ELBO can be written as L ( q ) = (cid:88) d (cid:40) E (cid:104) log (cid:16) p ( w d | c ( w ) d , z ( w ) d ,  ) p ( c ( w ) d |  (cid:48) )  p ( z ( w ) d |  (cid:48) ( w ) d ) p ( e d | c ( e ) d , z ( e ) d ,  ,  )  p ( c ( e ) d |  (cid:48) ) p ( z ( e ) d |  (cid:48) ( e ) d ) p (  (cid:48) ( w ) d |  0 ) p (  (cid:48) ( e ) d |  0 ) (cid:17)(cid:105) + H (cid:0) q ( c ( w ) d ) (cid:1) + H (cid:0) q ( z ( w ) d ) (cid:1) + H (cid:0) q (  (cid:48) ( w ) d ) (cid:1) + H (cid:0) q ( c ( e ) d ) (cid:1) + H (cid:0) q ( z ( e ) d ) (cid:1) + H (cid:0) q (  (cid:48) ( e ) d ) (cid:1)(cid:41) + E (cid:104) log (cid:16) p (  (cid:48) ) p (  ) p (  ,  ) (cid:17)(cid:105) + H (cid:0) q (  (cid:48) ) (cid:1) + H (cid:0) q (  ) (cid:1) + H (cid:0) q (  ,  ) (cid:1) , where H (  ) is the entropy term for variational distribution. By taking derivatives of this lower bound with respect to each variational parameter, we derive the coordinate ascent update steps. We develop an online variational inference for TMKGE, to process large datasets (Wang et al., 2011; Hoffman et al., 2010). Given the existing corpus-level parameters, first a document d is sampled and then its optimal document-level variational parameters are computed. For word related variational parameters, these updates include a ( w ) dt = 1 + (cid:88) n  ( w ) dnt , b ( w ) dt =  0 + (cid:88) n T (cid:88) s = t +1  ( w ) dns ,  ( w ) dtk  exp (cid:16) (cid:88) n  ( w ) dns E q (cid:2) log p ( w dn |  k ) (cid:3) E q (cid:2) log  k (cid:3)(cid:17) ,  ( w ) dnt  exp (cid:16) (cid:88) k  ( w ) dtk E q (cid:2) log p ( w dn |  k ) (cid:3) E q (cid:2) log  ( w ) dt (cid:3)(cid:17) (3) where expectations are with respect to variational distributions and have closed forms. For entity related variational parameters, similar updates can be derived, with the term E q (cid:2) log p ( e dm |  k ,  k ) (cid:3) replacing E q (cid:2) log p ( w dn |  k ) (cid:3) . Following Wang et al. (2011), for the corpus-level variational parameters, we use the following gradients:  kv =   kv +  + D (cid:88) t  ( w ) dtk (cid:0) (cid:88) n  ( w ) dnt I [ w dn = v ] (cid:1) ,  m k =  m k + D (cid:80) m,t  ( e ) dtk  ( e ) dmt e dm +  0 m 0 Dr k +  0 ,  k =   k +  0 + Dr k ,  k =   k +  0 + Dr k ,  W k =  W k + (cid:16) W  1 0 + D (cid:88) m,t  ( e ) dtk  ( e ) dmt e dm e Tdm (cid:17)  1 , u k =  u k + 1 + D (cid:88) t (cid:0)  ( w ) dtk +  ( e ) dtk (cid:1) , v k =  v k +  0 + D (cid:88) t K (cid:88) (cid:96) = k +1 (cid:0)  ( w ) dt(cid:96) +  ( e ) dt(cid:96) (cid:1) , (4) where r k is defined as (cid:80) m,t  ( e ) dtk  ( e ) dmt . The corpus-level parameters are then updated using these gradients (among them, the first, the fifth and the sixth are natural gradients while the other four are approximations from the posterior of Gaussian Wishart scale matrix W . It appears difficult to obtain natural gradients for those four.) and a learning rate parameter (cid:15) t . For instance, for topic-words distribution parameters we have    + (cid:15) t 0   . (5) number of top words and PMI scores model parameters data source 5 10 15 20 25 30 TMKGE K=300, T=20 20 Newsgroups 20.8 91.1 210.0 380.0 602.0 876.0 HDP K=300, T=20 20.0 91.6 212.6 384.1 598.4 868.7 LDA K=100 13.5 64.6 163.4 285.0 455.2 671.1 KGE-LDA K=30 18.9 69.8 187.5 320.6 482.7 616.5 TMKGE K=300, T=20 NIPS 16.6 97.1 160.3 299.6 474.5 685.5 HDP K=300, T=20 16.7 66.8 157.2 280.2 444.0 643.1 LDA K=100 13.9 67.6 161.9 297.0 471.2 681.1 KGE-LDA K=30 14.3 97.2 163.4 285.3 453.3 645.4 TMKGE K=300, T=20 Ohsumed 21.6 123.3 237.3 407.7 624.2 895.5 HDP K=300, T=20 15.6 70.7 168.2 338.9 582.9 864.9 LDA K=100 11.9 65.6 131.9 257.0 481.2 691.1 KGE-LDA K=30 15.6 116.5 185.4 354.2 585.4 795.6 Table 1: Topic Coherence of all models on three datasets with different number of top words. The rest of corpus-level variational parameters in (4) can be similarly updated. To ensure that the parameters converge to a stationary point, the learning rate satisfies (Hoffman et al., 2010; Sato, 2001) (cid:80)  t 0 =1 (cid:15) t 0 =  and (cid:80)  t 0 =1 (cid:15) 2 t 0 <  . Following Wang et al. (2011), we use (cid:15) t 0 = (  0 + t 0 )   , where   (0 . 5 , 1] and  0 > 0 . To improve the stability of online variational inference, we use a mini-batch of documents to compute the natural gradients. That is, the contribution of the single document d in (4) is replaced by sum of contributions of documents in the mini-batch S , and the factor D is replaced by D/ |S| . The overall scheme of online variational inference for TMKGE is shown in Algorithm 1. proposed TMKGE framework. while Stopping criterion is not met do Sample a random document d from the corpus. Update a ( w ) d , b ( w ) d ,  ( w ) d and  ( w ) d using (3). Update a ( e ) d , b ( e ) d ,  ( e ) d and  ( e ) d similar to (3). Compute the natural gradients using (4). Set (cid:15) t 0 = (  0 + t 0 )   and t 0  t 0 + 1 . Update all corpus-level parameters as (5). We evaluate TMKGE on two experimental tasks and compare its performance to those of LDA, HDP and KGE-LDA. For LDA and HDP, we use the online variational inference implementations. More precisely, we will evaluate our framework by the test whether it finds coherent and meaningful topics and the test whether it can achieve good performance in document classification. We run our experiments on three popular datasets; 20 Newsgroups, NIPS and the Ohsumed corpus. The 20 Newsgroups dataset contains 18,846 documents evenly categorized into 20 different categories. The NIPS dataset contains 1,740 papers from the NIPS conference. The Ohsumed corpus is from the MEDLINE database. We consider the 13,929 unique Cardiovascular diseases abstracts in the first 20,000 abstracts of the year 1996. Each document in the set has one or more associated categories from the 23 disease categories. The documents belonging to multiple categories are eliminated so that 7,400 documents belonging to only one category remain. The datasets are tokenized with Stanford CoreNLP (Manning et al., 2014). After standard pre-processing (such as removing stop words), there are 20,881 distinct words in the 20 Newsgroups dataset, 14,482 distinct words in the NIPS dataset and 8,446 distinct words in the Ohsumed dataset. The knowledge graph we employ is WordNet (Miller, 1995). WordNet is a large lexical knowledge graph. Entities in WordNet are synonyms which express distinct concepts. Relations in WordNet mainly involve conceptual-semantic and lexical relations. We use a subset of WordNet (WN18) introduced in Bordes et al. (2011) and employed in Yao et al. (2017) as well. WN18 contains 151,442 triplets with 40,943 entities and 18 relations. We link tokenized words to entities in WN18 via NLTK (Bird and Loper, 2004). In the experiments, for each method, we report the results based on the hyperparameter settings that obtain the best performances. For TMKGE and HDP, we report the results for K = 300 , T = 20 and K = 100 , T = 10 cases. For LDA and KGE-LDA, respectively, we have K = 100 and K = 30 . Throughout this work we fix the dimension of entity embedding as P = 5 . For online variational inference, we run the algorithms for 1000 iterations, with mini-batch size of 100. We assess the performance of the proposed TMKGE model based on topic coherence. Topic coherence has been shown to be more consistent with human judgment than other typical topic model metrics such as perplexity (Chang et al., 2009; Newman et al., 2010). We perform both quantitative and qualitative analysis of the topics discovered by TMKGE, and compare its performance to those of LDA, HDP and KGE-LDA. We evaluate the coherence of discovered topics by the point-wise mutual information (PMI) Topic Coherence metric. The PMI Topic Coherence is implemented following Newman et al. (2010): PMI ( k ) = N (cid:88) j =2 j  1 (cid:88) i =1 log p ( w i , w j ) p ( w i ) p ( w j ) where k refers to a topic, N refers to the number of top words of k , p ( w i ) is the probability that w i appears in a document, p ( w i , w j ) is the probability that w i and w j co-occur in the same document. A higher PMI score implies a more coherent topic. Following KGE-LDA, 4,776,093 Wikipedia articles are employed for obtaining topic coherence scores. Different from Yao et al. (2017), which used a fixed value of N (the number of top words, e.g. N = 5 or N = 10 ), we vary N in a range from 5 to 30 . (Lau and Baldwin, 2016) suggests that calculating topic coherence over several different cardinalities and averaging results in a substantially more stable evaluation. Table 1 shows the average topic coherence for different methods and datasets. We can observe that for the three datasets, TMKGE achieves highest topic coherence in almost all top word sizes. In the few cases which TMKGE does not rank highest, there only exist subtle differences with the top performing result. This shows that knowledge graph embedding improves the coherence of discovered topics. Further, for the top 10 words, the topic coherence of all three datasets are higher than those obtained by KGE-LDA. This shows that topic modeling based on HDP for both entity embedding and words enjoys incomparable advantages over LDA-based modeling. Table 2 shows example topics with their PMI scores learned from the three corpora by KGE-LDA and our TMKGE model. For comparison, we report similar topics to those listed in the KGE-LDA paper. It can be seen that TMKGE finds quite closely related words in a topic. For example, for the second column of 20 Newsgroups, topic words from both TMKGE and KGE-LDA are related to computers. However, it can be noted that words from TMKGE focus more on the core words of computer science. In contrast, words from the same topic in KGE-LDA seems to be closer to the brand, such as windows, mac or apple. In addition, topics found from TMKGE are more diverse than those found in KGE-LDA. For 20 Newsgroups, the three topics we list here refer to theology, computer science and middle east respectively while the three topics from KGE-LDA refer to internet, computer and car respectively. Both TMKGE and KGE-LDA discover probability-related and machine learning topics with different top words from NIPS dataset. Roughly speaking, KGE-LDA discovers gene-related, cancer-related and treatment-related topics from Ohsumed corpus. TMKGE discovers more diverse and more specific topics. For example, one topic TMKGE discovers is about Vietnamese veterans, cancer-related and sexual-disease topics. From the perspective of topic coherence, we can also see that TMKGE obtains higher PMI score in most of those topics. The whole trend is consistent with the average PMI score reported in the last section. Overall, TMKGE performs better than other topic models, including LDA, HDP and KGE-LDA in terms of average PMI and also in qualitative case studies. We evaluate our proposed method through document classification, we follow the approach in (Li and McCallum, 2006) for document classification. We have conducted a five-way classification on the comp subject of 20 Newsgroups dataset and on the top five most frequent labels of Ohsumed dataset (no labels for nips dataset), where each class of documents is divided into 75% training and 25% testing. For each class, the LDA, HDP and TMKGE models are trained on the training documents, and then the predictive likelihood for the test documents is calculated using the E-step in the variational inference procedure of LDA. A document is classified correctly if its corresponding model produces the highest likelihood. Table 3 presents the average classification accuracy for TMKGE, HDP and LDA over five repeated simulations. The table includes the classification accuracy for KGE-LDA, where the learned topic proportions are used as features for SVM classifier. For the majority of document classes, TMKGE has the best classification accuracy, except for the class mac . As shown, the SVM clas-sifier based on KGE-LDA has significantly worst performance. For more complete comparisons, we run experiments on all subjects of 20 Newsgroups and also report experimental results published in Shi et al. (2017) in Table 4. TMKGE achieves the best performance on all models. Firstly, it looks the addition of unnormalized knowledge graph embedding into TMKGE as a proportional vector to the word vector boosts the performance. Secondly, the selection of HDP over LDA plays an essential role. This can be indicated from the poor performance of KGE-LDA (which is even worse than BOW). More impressively, TMKGE achieves even much better performances than STE-Diff, TWE and TMSA, all of which involve the integration of word embedding and topic modeling. Impressively, TMKGE shows its supremacy over the state of the art model, TMSA with high margins. This shows that the knowledge graph structure included into the entity embedding conveys more information than pure word embedding. Meanwhile, this also shows that the two proportional vectors generated with online HDP enables the flexible sharing of information between words and entities. Accordingly, more coherent topics are extracted and the classification result are boosted as well. This paper presents TMKGE, a Bayesian nonparametric model based on hierarchical Dirichlet process for incorporation of entity embeddings from external knowledge graphs into topic modeling. The proposed method allows for flexible sharing of information between documents and knowledge graph. Specifically, TMKGE avoids forcing the words and entities to identical latent factors, thus making it a suitable framework for scenarios where only partial relational information are available. Furthermore, as a Bayesian nonparameteric model, TMKGE learns the number of word topics and entity mixture components automatically from the data. We have derived an efficient and scalable online variational inference for TMKGE. Comprehensive experiments on three different datasets suggest that TMKGE significantly outperforms SOA methods in terms of both topic coherence and document classification accuracy.
abstain objective result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain objective other other other other other other other other other other other other other other other other other other other other objective abstain method abstain abstain other abstain abstain other abstain abstain abstain method abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method other abstain abstain method other abstain abstain other method abstain abstain other abstain method abstain abstain other abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain