Abstract Graph convolutional networks (GCNs) have been applied recently to text classification and produced an excellent performance. However, existing GCN-based methods do not assume an explicit latent semantic structure of documents, making learned representations less effective and difficult to interpret. They are also transductive in nature, thus cannot handle out-of-graph documents. To address these issues, we propose a novel model named inductive Topic Variational Graph Auto-Encoder (T-VGAE), which incorporates a topic model into variational graph-auto-encoder (VGAE) to capture the hidden semantic information between documents and words. T-VGAE inherits the interpretability of the topic model and the efficient information propagation mechanism of VGAE. It learns probabilistic representations of words and documents by jointly encoding and reconstructing the global word-level graph and bipartite graphs of documents, where each document is considered individually and decoupled from the global correlation graph so as to enable inductive learning. Our experiments on several benchmark datasets show that our method outperforms the existing competitive models on supervised and semi-supervised text classification, as well as unsupervised text representation learning. In addition, it has higher interpretability and is able to deal with unseen documents. Recently, graph convolutional networks (GCNs)(Kipf and Welling, 2017; Velickovic et al., 2018) have been successfully applied to text classification tasks (Peng et al., 2018a; Yao et al., 2019; Liu et al., 2020; Wang et al., 2020). In addition to the local information captured by CNN or RNN, GCNs learn word and document representations by taking into account the global correlation information embedded in the corpus-level graph, where words and documents are nodes connected by indexing or citation relations. However, the hidden semantic structures , such as latent topics in documents (Blei et al., 2003; Yan et al., 2013; Peng et al., 2018b), is still ignored by most of these methods (Yao et al., 2019; Huang et al., 2019; Liu et al., 2020; Zhang et al., 2020), which can improve the text representation and provide extra interpretability (in which the probabilistic generative process and topics make more sense to humans compared to neural networks, i.e. topics can be visually represented by top-10 or 20 most probable word clusters). Although few studies such as (Wang et al., 2020) have proposed incorporating a topic structure into GCNs, the topics are extracted in advance from the set of documents, independently from the graph and information propagation among documents and words. We believe that the topics should be determined in accordance with the connections in the graph. For example, the fact that two words are connected provides extra information that these words are on a similar topic(s). Moreover, existing GCN-based methods are limited by their transductive learning nature, i.e. a document can be classified only if it is already seen in the training phase (Wang et al., 2020; Yao et al., 2019; Liu et al., 2020). The lack of inductive learning ability for unseen documents is a critical issue in practical text classification applications, where we have to deal with new documents. It is Table 1: Comparison with related work. intuitive to decouple documents with the global graph and treat each document as an independent graph (Huang et al., 2019; Zhang et al., 2020; Ding et al., 2020; Zhang and Zhang, 2020; Xie et al., 2021). However, no attempt has been made to address both aforementioned issues. To address these issues, we incorporate the topic model into variational graph auto-encoder (VGAE), and propose a novel framework named inductive Topic Variational Graph Auto-Encoder (T-VGAE). T-VGAE first learns to represent the words in a latent topic space by embedding and reconstructing the word correlation graph with the GCN probabilistic encoder and probabilistic decoder. Take the learned word representations as input, a GCN-based message passing probabilistic encoder is adopted to generate document representations via information propagation between words and documents in the bipartite graph. We compare our model with existing related work in Table 1. Different from previous approaches, our method uni-fies topic mining and graph embedding learning with VGAE, thus can fully embed the relations between documents and words into dynamic topics and provide interpretable topic structures into representations. Besides, our model builds a document-independent word correlation graph and a word-document bipartite graph for each document instead of a corpus-level graph to enable inductive learning. 1. We propose a novel model T-VGAE based on topic models and VGAE, which incorporates latent topic structures for inductively document and word representation learning. This makes the model more effective and interpretable. 2. we propose to utilize the auto-encoding variational Bayes (AEVB) method to make efficient black-box inference of our model. 3. Experimental results on benchmark datasets demonstrate that our method outperforms the existing competitive GCN-based methods on supervised and semi-supervised text classification tasks. It also outperforms topic models on unsupervised text representation learning. Recently, GCNs have been applied to various NLP tasks (Zhang et al., 2018; Vashishth et al., 2019). For example, TextGCN (Yao et al., 2019) was proposed for text classification, which enriches the corpus-level graph with the global semantic information to learn word and document embeddings. Inspired by it, Liu et al. (Liu et al., 2020) further considered syntactic and sequential contextual information and proposed TensorGCN. However, none of them utilized the latent semantic structures in the documents to enhance text classification. To address the issue, (Wang et al., 2020) proposed dynamic HTG (DHTG), in an attempt to integrate the topic model into graph construction. DHTG learned latent topics from the document-word correlation information (similar to traditional topic models), which will be used for GCN based document embedding. However, the topics in DHTG were learned independently from the word relation graph and the information propagation process in the graph, in which word relations are ignored. Moreover, the existing GCN-based methods also require a pre-defined graph with all the documents and cannot handle out-of-graph documents, thus limiting their practical applicability. To deal with the inductive learning problem, (Huang et al., 2019; Zhang et al., 2020; Ding et al., 2020; Zhang and Zhang, 2020) proposed to consider each document as an independent graph for text classification. However, the latent semantic structure and interpretability are still ignored in these methods. Different from previous approaches, we aim to deal with both issues of dynamic topic structure and inductive learning. We propose to combine the topic model and graph based information propagation in a unified framework with VGAE to learn interpretable representations for words and documents. There are also studies trying to enhance topic models with efficient message passing in the graph data structure of GCNs. GraphBTM (Zhu et al., 2018) proposed to enrich the biterm topic model (BTM) with the word co-occurrence graph encoded with GCNs. To deal with data streams, (Van Linh et al., 2020) proposed graph convolutional topic model (GCTM), which introduces a knowledge graph modeled with GCNs to the topic model. (Yang et al., 2020) presented Graph Attention TOpic Network (GATON) for correlated topic modeling. It tackles the overfitting issue in topic modeling with a generative stochastic block model (SBM) and GCNs. In contrast with these studies, we focus on integrating the topic model into GCN-based VGAE for supervised learning tasks and derive word-topic and document-topic distributions simultaneously. Variational Graph Auto-encoders (VGAEs) have been widely used in graph representation learning and graph generation. The earliest study (Kipf and Welling, 2016) proposed VGAE method, which extended variational auto-encoder (VAE) on graph structure data for learning graph embedding. Based on VGAE, (Pan et al., 2018) introduced an adversarial training to regularize the latent variables and further proposed adversarially regularized variational graph autoencoder (ARVGA). (Hasanzadeh et al., 2019) incorporated semi-implicit hierarchical variational distribution into VGAE (SIG-VAE) to improve the representation power of node embeddings. (Grover et al., 2019) proposed Graphite model that integrated an iterative graph refinement strategy into VGAE, inspired by low-rank approximations. However, to the best of our knowledge, our model is the first effort to apply VGAE to unify the topic learning and graph embedding for text classification, thus can provide better interpretability and overall performance. Formally, we denote a corpus as C , which contains D documents and the ground truth labels Y 2 c = { 1 , ..., M } of documents, where M is the total number of classes in the corpus. Each document t 2 C is represented by a sequence of words t = { w 1 , ..., w n t } ( w i 2 v ) , where n t is the number of words in document t and v is the vocabulary of size V . From the whole corpus, we build a word correlation graph G = ( v, e ) containing word nodes v and edges e , to capture the word co-occurrence information. Similar to previous work (Yao et al., 2019), we utilize the positive point mutual information (PPMI) to calculate the correlation between two word nodes. Formally, for two words ( w i , w j ) , we have PPMI ( w i , w j ) = max( log p ( w i , w j ) p ( w i ) p ( w j ) , 0) (1) where p ( w i , w j ) is the probability that ( w i , w j ) co-occur in the sliding window and p ( w i ) , p ( w j ) are the probabilities of words w i and w j in the sliding window. They can be empirically estimated as P ( w i , w j ) = n ( w i ,w j ) n and P ( w i ) = n ( w i ) n , where n ( w i , w j ) is the number of co-occurrences of ( w i , w j ) in the sliding windows, n ( w i ) is the number of occurrences of w i in the sliding windows and n the total number of sliding windows. For two word nodes ( w i , w j ) , the weight of the edge between them can be defined as: A vi,j = ( PPMI ( w i , w j ) , i 6 = j 1 , i = j (2) where A v 2 RV  V is the adjacency matrix which represents the word correlation graph structure G . Different from the existing studies (Yao et al., 2019; Liu et al., 2020; Wang et al., 2020) that consider all documents and words in a heterogeneous graph, we propose to build a separate graph for each document to enable inductive learning. Typically, documents can be represented by the document-word matrix A d 2 RD  V , in which the row A di = { x i 1 , ..., x iv } 2 R 1  V represents the document i , and x ij is the TF-IDF weight of the word j in document i . The decoupling of documents from a global pre-defined graph enables our method to handle new documents. Based on A v and A d , we propose the T-VGAE model, as shown in Figure 1. It is a deep generative model with structured latent variables based on GCNs. We consider that the word co-occurrence graph A and the bipartite graph A dt of each document t are generated from the random process with two latent variables z v 2 RV  K and z dt 2 R 1  K , where K denotes the number of latent topics. The generating process for A v , A d and Y are as follows (see Figure 2(a)): v A v z Y d A d z (cid:84) DV 1. For each word i in vocabulary v , draw the latent variable z vi from the prior p  ( z vi ) 2. For each observed edge A vi,j between words i and j , draw A vi,j from conditional distribution p  ( A vi,j | z vi , z vj ) 3. For each document t in corpus C : (a) Draw the latent variable z dt from the prior p  ( z dt ) (b) Draw A dt from the conditional distribution p  ( A dt | z dt , z v ) (c) Draw Y t from the conditional distribution p  ( Y t | z dt ) where  is the set of parameters for all prior distributions. Here, we consider the centered isotropic multivariate Gaussian priors p ( z v ) = Q Vi =1 p ( z vi ) = Q Vi =1 N ( z vi | 0 , I ) and p ( z d ) = Q Dt =1 p ( z dt ) = Q Dt =1 N ( z dt | 0 , I ) . Notice that the priors p ( z v ) and p ( z d ) are parameter free in this case. According to the above generative process, we can maximize the marginal likelihood of observed graph A v , A d and Y to learn parameters  and latent variables as follows: p ( A v , A d , Y | Z v , Z d , X v ) = DY t =1 p  ( Y t | z dt ) p  ( A dt | z dt , z v ) p  ( z dt ) VY i =1 VY j =1 p  ( A vi,j | z vi ( z vj ) T ) p  ( z v ) (3) Because the inference of true posterior of latent variable z v and z d is intractable, we further introduce the variational posterior distribution q  ( z v , z d | A d , A v , X v ) with parameters  to approximate the true posterior p  ( z v , z d ) = p  ( z v ) p  ( z d ) . We make the structured mean-field (SMF) assumption q  ( z v , z d | A d , A v , X v ) = q  ( z v | A v , X v ) q  ( z d | A d , z v ) , where X v 2 RV  M are the feature vectors of words and M is the dimension of the feature vectors (see Figure 2(b)). We can yield the following tractable stochastic evidence lower bound (ELBO): L (  ,  ; A v , A d , X v ) = E q  ( z v | A v ,X v ) [log p  ( A v | z v )] + E q  ( z d | A d ,z v ) [log p  ( A d | z d , z v )] + E q  ( z d | A d ,z v ) [log p  ( Y | z d )]  KL [ q  ( z v | A v , X v ) || p  ( z v )]  KL [ q  ( z d | A d , z v ) || p  ( z d )] (4) where the first three terms are the reconstruction terms, and the latter two terms are the Kullback-Leibler (KL) divergences of variational posterior distributions and true posterior distributions. Using auto-encoding variational Bayes (AVB) approach (Kingma and Welling, 2013), we are able to parametrize the variational posteriors q  and true posteriors p  with the GCN-based probabilistic encoder and decoder, to conduct neural variational inference (NVI). For the latent variable z v , we make the mean-field approximation that: q  ( z v | A v , X v ) = Q Vi =1 q  ( z vi | A v , X v ) . For simplify the model inference, we consider the multivariate normal variational posterior with a diagonal covariance matrix as previous neural topic models (Miao et al., 2016; Bai et al., 2018)that: q  ( z vi | A v , X v ) = N ( z vi |  vi , diag ((  vi ) 2 )) , where  vi , (  vi ) 2 are the mean and diagonal covariance of the multivariate Gaussian distribution. We use the graph convolutional neural network to parametrize the above posterior and inference z v with the input graph A v and feature vectors X v : ( H v ) l +1 =  (  A v ( H v ) l ( W v ) l )  v =  (  A v ( H v ) l +1 ( W v ) l +1 ) log  v =  (  A v ( H v ) l +1 ( W v  ) l +1 ) (5) where  v ,  v are matrices of  vi ,  vi , l is the number of GCN layers, we use one layer in our experiments, { W v , W v  } 2  are weight matrices,  is the ReLU,  A v = ( D v )  12 A v ( D v )  12 is the symmetrically normalized adjacent matrix of the word graph, and D v denotes the corresponding degree matrix. The input of GCN is the feature vectors X v which is initialized as the identity matrix I , i.e., ( H v ) 0 = X v = I , same as in (Yao et al., 2019). Then, z v can be naturally sampled as follows according to the reparameterization trick (Kingma and Welling, 2013): z v =  v +  v   , where  is the element-wise product, and   N (0 , I ) is the noise variable. Through the message propagation of the GCN layer, words that co-occur frequently tend to achieve similar representations in the latent topic space. Similar to z v , we also have: q  ( z d | A d , z v ) = DY t =1 q  ( z dt | A dt , z v ) q  ( z dt | A dt , z v ) = N ( z dt |  dt , diag ((  dt ) 2 )) (6) where  dt , (  dt ) 2 are the mean and diagonal covariance of the multivariate Gaussian distribution. Although there are two types of nodes word and document in the bipartite graph A d , we mainly focus on learning representations of document nodes based on the representations of word nodes learned from A v in this step. Therefore, we propose the unidirectional message passing (UDMP) process on A d , which propagates the information from word nodes to documents: H dt =  ( P Vi =1 A dti z vi W d ) where  is the Relu activation function, W d is the weight matrix. Then, we parametrize the posterior and inference z d based on UDMP:  d = UDMP ( A d , z v , W d ) log  d = UDMP ( A d , z v , W d  ) (7) where  d ,  d are matrices of  dt , (  dt ) 2 , UDMP is the message passing as in Equation 4, W d , W d  are weight matrices. Similarly, we sample z d as follows z d =  d +  d  " , where "  N (0 , I ) is the noise variable. Through the propagation mechanism of UDMP, documents which share similar words tend to yield similar representations in the latent topic space. Although T-VGAE can learn topics z v and document-topic representations z d as in traditional topic models, we do not focus on proposing a novel topic model, but aim to combine the topic model with VGAE, to improve word and document representations with latent topic semantic and provide probabilistic interpretability. Moreover, rather than learning topics and document-topic representations from the document-word feature A d as LDA topic models (Blei et al., 2003), we propose to learn word-topic representations z v from word co-occurrence matrix A v , and then infer document-topic representations z d based on the document-word feature A d and word-topic representations z v , which is similar to the Biterm topic model (Yan et al., 2013). With the learned z v and z d , ideally, the observed graph A v and A d can be reconstructed through a decoding process. For A v , we assume P  ( A v | z v ) conforms to a multivariate Gaussian distribution, whose mean parameters are generated from the inner product of the latent variable z v : P  ( A v | z v ) = VY i =1 p  ( A vi | z v ) p  ( A vi | z v ) = VY i =1 N ( A vi |  ( z vi ( z v ) T ) , I ) (8) where  is the nonlinear activation function. Similarly, the inner product between z v and z d is used to generate A d , which is sampled from the multivariate Gaussian distribution: P  ( A d | z d , z v ) = DY i =1 p  ( A di | z di , z v ) P  ( A di | z di , z v ) = DY i =1 N ( A di |  ( z di ( z v ) T ) , I ) (9) For categorical labels Y , we assume p  ( Y | z d ) follows a multinomial distribution P  ( Y | z d ) = Mul ( Y | f y ( z d )) , whose label probability vectors are generated from z d , where f y is the multi-layer neural network. For each document t , the prediction is given by  y t = argmax y 2 c P  ( y | f y ( z dt )) . We can rewrite Equation 4 to yield the final variational objective function: L (  ,  )  VX i =1 VX j =1 log p  ( A vi,j | z vi , z vj ) + DX t =1  log p  ( A dt | z dt , z v ) + log p  ( Y t | z dt )   KL [ q  ( z v ) || p  ( z v )]  KL [ q  ( z d ) || p  ( z d )] (10) with following reconstruction terms and KL divergences: log p  ( A vi | z v )  || A vi   ( z vi ( z v ) T ) || 2 log p  ( A dt | z dt , z v )  || A dt   ( z dt ( z v ) T ) || 2 log p  ( Y t | z dt )  Y t log  y t + (1  Y t ) log(1   y t ) KL [ q  ( z vi ) || p  ( z vi )]  1 2 VX j =1 ((  vij ) 2 +(  vij ) 2  (1 + log(  vij ) 2 )) KL [ q  ( z dt ) || p  ( z dt )]  1 2 VX j =1 ((  d tj ) 2 +(  d tj ) 2  (1 + log(  d tj ) 2 )) Through maximizing the objective with stochastic gradient descent, we jointly learn the latent word and document representations, which can ef-ficiently reconstruct observed graphs and predict ground truth labels. In this section, to evaluate the effectiveness of our proposed T-VGAE, experiments are conducted on both supervised and semi-supervised text classification tasks, as well as unsupervised topic modeling tasks. We conduct experiments on five commonly used text classification datasets: 20NewsGroups, Ohsumed, R52 and R8, and MR. We use the same data preprocessing as in (Yao et al., 2019). The overview of the five datasets is depicted in Table 2. 4.1.2 Baselines We compare our method with the following two categories of baselines: text classification : 1)TF-IDF+LR: the classical logistic regression method based on TF-IDF features. 2) CNN (Kim, 2014): the convolutional neural network based method with pre-trained word embeddings. 3) LSTM (Liu et al., 2016): the LSTM based method with pre-trained word embeddings. 4) SWEM (Shen et al., 2018): the word embedding model with pooling strategies. 5) fastText (Joulin et al., 2016): the averages word embeddings for text classification. 6) Graph-CNN (Peng et al., 2018a): a graph CNN model based on word embedding similarity graphs 7) LEAM (Wang et al., 2018): the label-embedding attentive models with document embeddings based on word and label descriptions. 8) TextGCN (Yao et al., 2019): a GCN model with a corpus-level graph to learn word and document embeddings. 9) DHTG (Wang et al., 2020): a GCN model with a dynamic hierarchical topic graph based on the topic model. 1 Its code is not released yet, therefore we only report the test micro precision here. (Miao et al., 2016): a deep neural variational document topic model. 3) AVITM (Srivastava and Sutton, 2017): an autoencoding variational Bayes (AEVB) topic model based on LDA. 4) GraphBTM (Zhu et al., 2018): an enriched biterm topic model (BTM) with the word co-occurrence graph encoded by GCN. Following (Yao et al., 2019), we set the hidden size K of latent variables and other neural network layers as 200 and set the window size in PPMI as 20. The dropout is only utilized in the classifier, and is set to 0 . 85 . We train our model for a maximum of 1000 epochs with Adam (Kingma and Ba, 2015) under learning rate 0 . 05 . 10% of the data set is randomly sampled and spared as the validation set for model selection. The parameter settings of all baselines are the same as their original papers or implementations. all the baselines on each dataset, which proves the effectiveness of our proposed methods. Compared with TextGCN, our method yields better performance in both datasets. It demonstrates the importance of integrating the latent semantic structures in text classification. It is also observed from the superior performance of DHTG when compared with TextGCN. However, DHTG only learns from the document-word correlation while our method fully exploits both word-word and document-word correlation information, resulting in a significant improvement over DHTG. This proves the effectiveness of unified topic modeling and graph representation learning in text classification. Moreover, there are no test documents involved during the training of our method, which shows the inductive learning ability of our method, different from TextGCN and DHTG which requires a global graph including all documents and words. encoder, to demonstrate the impact of a different order of word-word correlation information in A v . On datasets R52 and R8, our method achieves the best performance when the layer number is 1 . This is different from TextGCN and DHTG, which generally have the best performance with 2 layer GCN. A possible reason is that our model has already considered one-hop document-word relation information when encoding document-word graph A d . If the layer number is set to 1 when encoding A v , it actually integrates two-hop neighborhood information, thus achieves a similar effect to TextGCN and DHTG. In Table 4, we further present the test accuracy of our method using different layers of GCN encoder, to demonstrate the impact of different orders of word-word correlation information in A v . On datasets R52 and R8, our method achieves the best performance when the layer number is 1 . This is different from TextGCN and DHTG, which generally have the best performance with 2 layer GCN. A possible reason is that our model has already considered one-hop document-word relation information when encoding document-word graph A d . If the layer number is set to 1 when encoding A v , it actually integrates two-hop neighborhood information, thus achieves a similar effect to TextGCN and DHTG. Figure 3 shows the changes of the test accuracy along with different numbers of topics on five datasets. We can see that the test accuracy on five datasets generally improves with the increase of the number of topics and reaches the peak when the topic number is around 200 . The number of topics shows more impact on the Oshumed dataset than on the other four datasets. This does not seem to be related to the number of classes in the dataset. We suspect it has to do with the nature of the text (medical domain vs. other domains). In Figure 4, we further present the semi-supervised classification test accuracy on datasets 20NG and R8 where different proportions ( 1% , 5% , 10% and 20% ) of the original training set are used. We can see that, in cases where labeled samples are limited, our model still consistently outperforms all the baselines. Compared with other methods, TextGCN and our model can preserve good performance with few labeled samples ( 1% , 5% ). This illustrates the effect of label propagation in GCN for semi-supervised learning. When compared with TextGCN, our model yields better performance because of its inductive learning capability and the incorporation of the latent topic semantics. We further evaluate the performance of models on unsupervised topic modeling tasks. We generally assume that the more topics are coherent, the more they are interpretable. Following (Srivastava and Sutton, 2017), We use the average pairwise PMI of the top 10 words in each topic and the perplexity with the ELBO as quality measures of topics. We show in Table 6 the measures under different topic numbers in the 20NG dataset. We remove the supervised loss of our method and the result of GraphBTM is not presented for unable to learn document topic representation for each document. In the table, we can see that our model outperforms the others in terms of topic coherence, which could be attributed to the combination of word co-occurrence graph and message passing in GCN. The message passing leads to similar representations of words that co-occur frequently in the latent topic space, thus improves the semantic coherence of learned topics, as shown in Table 5 that related words tend to belong to the same topic. Our method also benefits from document-word correlation, and yield better performance when compared with GraphBTM which encode bi-term graph via GCN. We utilize t-SNE to visualize the latent test document representations of the 20NG dataset learned by our model, DHTG and TextGCN in Figure 5, in which each dot represents a document and each color represents a category. Our method yields the best clustering results compared with the others, which means the topics are more consistent with pre-defined classes. It shows the superior interpretability of our method for modeling the latent topics along with both word co-occurrence graph and document-word graph when compared with DHTG. In this paper, we proposed a novel deep latent variable model T-VGAE via combining the topic model with VGAE. It can learn more interpretable representations and leverage the latent topic semantic to improve the classification performance. T-VGAE inherits advantages from the topic model and VGAE: probabilistic interpretability and efficient label propagation mechanism. Experimental results demonstrate the effectiveness of our method along with inductive learning. As future work, it would be interesting to explore better-suited prior distribution in the generative process. It is also possible to extend our model to other tasks, such as information recommendation and link prediction. This research is supported by the CSC Scholarship offered by China Scholarship Council. We would like to thank the anonymous reviewers for their constructive comments. We thank MindSpore for the partial support of this work, which is a new deep learning computing framework 1 .
abstain abstain abstain objective abstain abstain result abstain abstain abstain abstain other abstain method abstain abstain objective abstain abstain abstain objective abstain abstain method method objective objective objective objective objective other other other other other other other other other other objective objective other other other other other objective other other other other other objective abstain abstain method method other other abstain abstain abstain objective abstain method abstain abstain abstain abstain abstain abstain abstain method abstain method other abstain other method abstain method method abstain abstain abstain other method abstain other method abstain abstain other abstain method method abstain abstain other abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain result abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result result result abstain abstain abstain result result objective abstain abstain objective abstain objective other other other