The bigger the corpus, the more topics it can potentially support. To truly make full use of massive text corpora, a topic model inference algorithm must therefore scale efficiently in 1) documents and 2) topics, while 3) achieving accurate inference. Previous methods have achieved two out of three of these criteria simultaneously, but never all three at once. In this paper, we develop an online inference algorithm for topic models which leverages stochasticity to scale well in the number of documents, sparsity to scale well in the number of topics, and which operates in the collapsed representation of the topic model for improved accuracy and run-time performance. We use a Monte Carlo inner loop in the online setting to approximate the collapsed variational Bayes updates in a sparse and efficient way, which we accomplish via the Metropolis-Hastings Walker method. We showcase our algorithm on LDA and the recently proposed mixed membership skip-gram topic model. Our method requires only amortized O ( k d ) computation per word token instead of O ( K ) operations, where the number of topics occurring for a particular document k d  the total number of topics in the corpus K , to converge to a high-quality solution. Topic models are powerful tools for analyzing to-day's massive, constantly expanding digital text information by representing high-dimensional data in a low-dimensional subspace. We can recover the main themes of a corpus by using topic models such as latent Dirichlet allocation (LDA) to organize, understand, search, and explore the documents (Blei et al., 2003). Traditional LDA inference techniques such as variational Bayes and collapsed Gibbs sampling do not readily scale to corpora containing millions of documents. To scale up inference, the main approaches are distributed algorithms (New-man et al., 2008) and stochastic algorithms (Hoff-man et al., 2010, 2013). Stochastic algorithms, such as stochastic variational inference (SVI), operate in an online fashion, and hence do not need to see all of the documents before updating the topics, so they can be applied to corpora of any size, without expensive distributed hardware ( Hoffman et al., 2010). The collapsed representation of topic models is also frequently important, as it leads to faster convergence, efficient updates, and lower variance in estimation ( Griffiths and Steyvers , 2004). The stochastic collapsed variational Bayesian inference (SCVB0) algorithm, proposed by (Foulds et al., 2013), combines the benefits of stochastic and collapsed inference. Larger corpora typically support more topics, which brings the additional efficiency challenge of training a larger model ( Mimno et al., 2012). This challenge has been addressed by exploiting sparsity to perform updates in time sublinear in the number of topics. A sparse variant of the SVI algorithm for LDA, SSVI, proposed by (Mimno et al., 2012), is scalable to large numbers of topics, but does not fully exploit the collapsed representation of LDA, which is important for faster convergence and improved inference accuracy, due to a better variational bound (Teh et al., 2007). The Metropolis Hastings Walker (MHW) method (Li et al., 2014) scales well in the number of topics, and uses a collapsed inference algorithm, but it operates in the batch setting, so it is not scalable to large corpora. LightLDA (Yuan et al., 2015) is a distributed approach to the MHW method which adopts a data-and-model-parallel strategy to maximize memory and CPU efficiency. However, it is not an online approach, and furthermore requires multiple expensive computer clusters to converge faster. Tensor methods are another approach to speeding up topic models (Anandku-mar et al., 2014; Arora et al., 2012), which theoretically guarantee the recovery of the true parameters by overcoming the problem of local optima. These techniques use the method of moments instead of maximum likelihood estimation or Bayesian inference, which leads to lower data efficiency, and sometimes unreliable performance. In this work, we propose a highly efficient and scalable inference algorithm for topic models. We develop an online algorithm which leverages stochasticity to scale well in the number of documents, sparsity to scale well in the number of topics, and which operates in the collapsed representation of topic models. We thereby combine the individual benefits of SVI, SSVI, SCVB0, and MHW into a single algorithm. Our approach is to develop a sparse version of SCVB0. Inspired by SSVI, we use a Monte Carlo inner loop to approximate the SCVB0 variational distribution updates in a sparse and efficient way, which we accomplish via MHW method. To show the generality of our algorithm, we explore the benefits of our inference method for LDA and another recently proposed topic model, MMSGTM, with experiments on both small and large-scale datasets. To build the foundation for our proposed method, in this section we provide the necessary background on LDA and MMSGTM topic models and their associated inference algorithms. This is followed by a description of the MHW sampler for reducing topic model sampling complexity. Probabilistic topic models such as LDA (Blei et al., 2003) use latent variables to encode co-occurrence patterns between words in text corpora and other bag-of-words represented data. In LDA, we assume that the D documents in a corpus are each from mixture distributions of K individual topics  k , k 2 f 1 ; :::; K g , each of which are discrete distributions over words. For a document j of length N j , the local (document-level) variables (cid:18) j are a distribution over topics drawn from a Dirichlet prior with parameters (cid:11) k and for each token, global variables (corpus-level)  k are drawn from a Dirichlet prior with parameters (cid:12) w . Due to conjugacy, we can marginalize out topics (cid:2) and distributions over topics (cid:8) , and perform inference only on the topic assignments Z in the collapsed representation of LDA (Griffiths and Steyvers, 2004). For scalable and accurate inference, Foulds et al. (2013) proposed a stochastic collapsed variational inference algorithm, SCVB0. The SCVB0 approach computes a variational discrete distribution (cid:13) ij over the K topic assignment probabilities for each word i in each document j , but does not maintain the (cid:13) variables that increase the memory requirement of original batch CVB0 algorithm (Asuncion et al., 2009). SCVB0 iteratively updates each (cid:13) ij using (cid:13) ijk : / N (cid:8) w ij k + (cid:12) w ij N Zk +  w (cid:12) w ( N (cid:2) jk + (cid:11) k ) (1) for each topic k , with j th document's i th word w ij . The NZ , N (cid:2) , and N (cid:8) are referred to as CVB0 statistics, where NZ is the vector of expected number of words assigned to each topic, each entry j , k of matrix N (cid:2) , and each entry w , k of matrix N (cid:8) are the expected number of times document j , and word w are assigned to topic k , respectively, across the corpus. To do stochastic updates of these variables, one sequence of step-sizes (cid:26) (cid:8) for N (cid:8) and NZ and another sequence (cid:26) (cid:2) for N (cid:2) are maintained. The update of N (cid:2) j for every token i of document j with an online average of the current value and its expected value is N (cid:2) j := (1 (cid:0) (cid:26) (cid:2) t ) N (cid:2) j + (cid:26) (cid:2) t C j (cid:13) ij (2) where C j is the document length. In practice, it is too expensive to update N (cid:8) after every token. This leads to the use of minibatch updates with the average of the M per-token estimates of the form Y ij , which is a W (cid:2) K matrix with the w ij th row being (cid:13) ij and with zeros in the other entries: N (cid:8) := (1 (cid:0) (cid:26) (cid:8) t ) N (cid:8) + (cid:26) (cid:8) t ^ N (cid:8) (3) NZ := (1 (cid:0) (cid:26) (cid:8) t ) NZ + (cid:26) (cid:8) t ^ NZ , (4) where ^ N (cid:8) = C j M j  ij 2 MY ij , ^ NZ = C j M j  ij 2 M (cid:13) ij , and C is the number of words in the corpus. The SCVB0 algorithm outperforms stochastic VB (Hoffman et al., 2010) on large corpora by converging faster and often to a better solution (Foulds et al., 2013). However, the SCVB0 algorithm does not leverage sparsity, and hence requires O ( K ) operations per word token. To show the generality of our approach to topic models other than LDA, we will also apply our method to a recent model called the Mixed Membership Skip-gram Topic Model (MMSGTM) (Foulds, 2018), which combines ideas from topic models and word embeddings (cf. also (Das et al., 2015; Liu et al., 2015)). MMSGTM's generative model for words and their surrounding context is: (cid:15) For each word w i in the corpus  Sample a topic z i (cid:24) Discrete ( (cid:18) w i )  For each word w c 2 context ( i ) (cid:3) Sample a context word w c (cid:24) Discrete (  z i ) . The inferred model can then be used to train embeddings for topics and words, although we do not consider this here. The MMSGTM admits a collapsed Gibbs sampler (CGS) which efficiently resolves the cluster assignments. With Dirichlet priors on the parameters, the CGS update is p ( z i = k j : ) : / ( N ((cid:8)) (cid:0) i w i k + (cid:11) k ) (cid:2) j context ( i ) j  c =1 N ((cid:8)) (cid:0) i w c k + (cid:12) w c + N ( i;c ) w c N ( Z ) (cid:0) i k +  w (cid:12) w + c (cid:0) 1 , (5) where (cid:11) and (cid:12) are parameter vectors for Dirichlet priors over the topic and word distributions, N (cid:8) w i and N (cid:8) w c are input and output word-topic counts (excluding the current word), NZ is the total topic counts in output word-topic counts, and N ( i;c ) w c is the number of occurrences of word w c before the c th word in the i th context. MMSGTM exploits the MHW algorithm, which scales sub-linearly in K, but not in the number of training documents. The MHW method (Li et al., 2014), which is a key component of our approach, uses a data structure called an alias table which allows sampling from a discrete distribution in amortized O (1) time. Assuming initial probabilities p 0 ; p 1 ; :::; p l (cid:0) 1 of a distribution over l outcomes and average of probabilities a = 1 l , the alias table A can be formed as follows (Marsaglia et al., 2004): (cid:15) Initialize: for i from 0 to l (cid:0) 1  A alias [ i ] = i and A prob = ( i + 1) a (cid:15) Do the following steps n (cid:0) 1 times  Find smallest p i and largest p j  Set A alias [ i ] = j and A prob = i (cid:2) a + p i  p j := p j (cid:0) ( a (cid:0) p i ) and p i := a . Then, to sample from p using the alias table: (cid:15) Roll l -sided fair die to choose element i of A (cid:15) If Rand (1) < A prob [ i ] return i , else return A alias [ i ] . Li et al. (2014) cache alias table samples, avoiding the need to store the table. Once the supply of samples is exhausted they compute a new alias table. They draw samples from the Gibbs sampling update, analogous to (cid:13) ij in Equation 1, in amortized O ( k d ) time by decomposing the update into p ( z ij = k j : ) / N (cid:2) jk N (cid:8) w ij k + (cid:12) w ij N Zk +  w (cid:12) w + (cid:11) k N (cid:8) w ij k + (cid:12) w ij N Zk +  w (cid:12) w (6) where the first term, sparse in k d , admits sampling in O ( k d ) time, and the second term is dense but slow changing. A Metropolis-Hastings (M-H) update is used to correct for approximating the CGS update with a proposal distribution q ( k ) based on the stale alias samples. Foulds et al. ( 2018) propose to apply simulated annealing to optimize instead of sample, and which improves mixing for the MMSGTM. This is achieved by raising the model part of the M-H acceptance ratio for a new sample z ( new ) i (cid:24) q ( k ) to the power of 1 T j at iteration j : p ( accept z ( new ) i j : ) = (7) min (1 ; ( p ( z ( new ) i ) p ( z ( old ) i ) ) 1 Tj q ( z ( old ) i ) q ( z ( new ) i )) . In this section, we introduce our approach, a sparse version of SCVB0, which combines the individual benefits of the SVI, SSVI, SCVB0 and MHW algorithms, to scale well not only in the number of documents but also in the number of topics, while gaining the benefits of collapsed inference. We refer to our method as the sparse stochastic collapsed variational Bayesian inference (SparseSCVB0) algorithm. In SparseSCVB0, we obtain sparsity by substituting sparse Monte Carlo approximations (cid:13) ( pseudo ) ij for the original SCVB0 variational distributions (cid:13) ij . The justification for this procedure, also used by (Mimno et al., 2012), is that the expected value of an average over one-hot samples from a distribution is equal to that distribution: E s i (cid:24) p ( s ) [ Si =1 (cid:14) k ( s i ) S ] = 1 SS  i =1 E s i (cid:24) p ( s ) [ (cid:14) k ( s i ) ] = 1 SS  i =1  k  (cid:14) k ( s i = k  ) p ( s i = k  ) = p ( s = k ) . Thus, the overall procedure is still a valid stochastic optimization algorithm. We approximate the inner loop sampler in time sublinear in K , constructing (cid:13) ( pseudo ) by generating S samples from (cid:13) ij using the MHW method. To describe the general form of our algorithm, we introduce SparseSCVB0 statistics: local (e.g. document-level) expected counts NL , global (corpus-level) expected counts NG , and total expected topic counts NZ . We approximate local sufficient statistics NL for each token i in document j via: N Ljk (cid:25) C j E (cid:13) ( pseudo ) ij [ s 2 S (cid:14) z sij = k S ] . Since (cid:13) ( pseudo ) is sparse, we can efficiently update these statistics using only its non-zero entries. This approach allows us to learn high-dimensional topic models efficiently on very large corpora. Before updating global parameters in a similar fashion, it may also be beneficial to perform a small number of burn-in passes to learn the local parameters NL (Foulds et al., 2013). For large-scale datasets (e.g. Wikipedia), SparseSCVB0 operates in a mini-epoch approach where we process a large subset of the corpus (e.g. 5 ; 000 documents) several (e.g. 3 ) times, before discarding and processing the next subset, and so on. This allows a warm start of NL in repeating iterations, with a small memory overhead. Pseudo-code of SparseSCVB0 for a mini-epoch is provided in Algorithm 1, which we discuss more in the next two sections, including model-specific aspects. To deploy SparseSCVB0 for LDA, we use the M-H proposal distribution from (Li et al., 2014) which involves drawing samples exactly from document-specific sparse terms or approximately using cached samples from the alias table: where p L ( k ) and q G ( k ) represent the sparse and dense part, respectively from Equation 6 and PL =  k p L ( k ) , QG =  k q G ( k ) . When PLPL + QG > RandUnif (1) , we sample from the sparse part in O ( k d ) time depending on only the non-zero entries of N Lj (analogous to N (cid:2) j ), as N Lj is sparse in the LDA setting. Otherwise, we sample from the dense part in amortized O (1) time using the alias method. Unfortunately, due to the stochastic update, any entry of N Lj never becomes exactly zero, however it may maintain a very small value. To address this, we apply a sparsification heuristic, where we threshold N Lj after burn-in passes for each document iteration. We parameterize the threshold as (cid:28)(cid:26) Lt Cj C j ; where C j is the length of the j document, (cid:26) Lt Cj is the step size for the last token of this document, and constant 0 < (cid:28) (cid:20) 1 controls the sparsity. In our preliminary experiments, we found that, somewhat counter-intuitively, the Monte Carlo and sparsification approximations actually improve convergence in early iterations. We believe that this is because they help SCVB0 escape the initial high-entropy regime, during which convergence of variational algorithms is poor (Salakhutdinov et al., 2003). This property makes the benefit of annealing insignificant, so we do not use simulated annealing for LDA inference, fixing T j = 1 . An additional optimization of SparseSCVB0 for LDA inference can be performed by clump-ing (Teh et al., 2007; Foulds et al., 2013), where one update of the local parameters is performed for each distinct word type in each document. This is performed by fixing the variational distribution, and scaling the update by number of copies of the distinct word type in the document. If we observe the distinct word type w aj , which occurs m aj times in document j , the update is N Lj := (1 (cid:0) (cid:26) Lt ) m aj N Lj + (1 (cid:0) (1 (cid:0) (cid:26) Lt ) m aj ) C j (cid:13) ( pseudo ) aj . The main contribution of our approach for the MMSGTM algorithm is to scale this algorithm in number of documents with online inference, as MMSGTM already scales sublinearly in K using MHW. Foulds et al. (2018) use an MHW proposal which approximates the CGS update, interpreted as a product of experts (Hinton, 2002) in which each word in the context is an expert which weighs in multiplicatively on the update, with a mixture of experts. In the proposal, they draw a Input word = learning Top words in top 2 topics for the input word NIPS SparseSCVB0 algorithms algorithm reinforcement problem problems j learning gradient descent rate weight machine Original SCVB0 learning networks network algorithm neural j propagation function reinforcement gradient algorithms Wikipedia SparseSCVB0 university school college education students research j public center science schools article information Original SCVB0 students school education university college schools j center year program degree systems information Table 4: Top words in the top 2 topics for an input word using original SCVB0 and SparseSCVB0 for MMSGTM. context word w c uniformly from the context of the current word, w c (cid:24) Uniform ( j context ( w i ) j ) , and then sample a word based on the chosen context word's contribution to the update: q ( k ) : / N Gw c k + (cid:12) w c N Zk +  w (cid:12) w (10) where N Gw c is analogous to the output context word-topic counts N (cid:8) w c of original MMSGTM model. The proposal samples via the alias method in amortized O (1) time, instead of O ( k d ) time, since it does not involve the sparse term. We use this proposal to approximate the CVB0 update for the model, which is a deterministic version of Equation 5, neglecting to exclude the current assignment of z i . We update N Gw i (analo-gous to N (cid:8) w c ) for each current word w i locally, but update N Gw c and N z via minibatch counts ^ N Gw c and ^ NZ , respectively, for each context word w c of current word w i . Unlike for LDA, C j in the local updates represents the total number of input word j in the corpus. As we draw multiple output words from each topic assignment, the effective temperature of the MMSGTM model is much lower than for standard LDA which may cause problems with mixing and leads it to get stuck in the initial regime. Following Foulds et al. (2018), we perform simulated annealing which varies the M-H acceptance ratio to improve mixing. We parameterize the temperature schedule as T j = T 0 + (cid:21)(cid:20) (cid:22)j D , where T 0 is the target final temperature, (cid:20) (cid:20) 1 , constant (cid:22) controls the amount of temperature reduction after each document iteration j and (cid:21) controls the initial temperature. In this section we study the performance of our SparseSCVB0 1 algorithm, on small as well as large corpora to validate the proposed method for We compared SparseSCVB0 to SCVB0 and SVI. For a fair comparison, we implemented all of them in the fast high-level language Julia V0.6.2 ( Bezanson et al., 2017). We conducted all experiments on a computer with 64GB memory and an Intel Xeon E5-2623 V4 processor with 2.60 GHz clock rate, 8 (cid:2) 256KB L2 Cache and 10MB L3 Cache. As we only use one single thread for sampling across all experiments, only one CPU core is active throughout the experiment with only 256KB available L2 Cache. 2 We used NIPS , Reuters-150 , PubMed Central , and Wikipedia as representative very small, small, medium, and large-scale datasets, respectively. The NIPS corpus has 1740 scientific articles from years 1987-1999 with 2 : 3 M tokens, due to Sam Roweis. The newswire corpus Reuters-150 contains 15 ; 500 articles with dictionary size of 8 ; 350 words. PubMed Central has 320 M tokens across 165 ; 000 scientific articles and a vocabulary size of around 38 ; 500 words. The Wikipedia corpus contained 4 : 6 million articles from the online 2 Since SSVI relies on multiple complex implementation details, we were unable to develop a fair implementation, nor were we able to obtain soure code for a previous implementation. We expect that its accuracy would be similar to SVI, with a speed-up at or below that bestowed by a MHW-based inner loop. (Mimno et al., 2012) apply it with only 200 topics for most of their experiments, and at most 1000 topics. network data communication information communicate connection prison prisoners prisoner imprisoned jail escaped detained guards dog dogs shepherd hounds bred coat scent instinct eating companion song sung sing singing sings sang songs recorded melody tune votes vote cast elections voted candidate parties majority election wind winds blowing speed blows direction high low blown chill hour hours noon time daylight minutes midnight morning seconds Table 5: Randomly selected topics from a 10 ; 000 topic model trained using SparseSCVB0 on Wikipedia encyclopedia. We used the dictionary of 7 ; 700 words which was extracted by Hoffman et al. (Hoffman et al., 2013). There were 811 M tokens in the corpus. We implement SparseSCVB0, original SCVB0 and SVI algorithms using the clumping optimization (Teh et al., 2007) technique. In all LDA experiments, each algorithm was trained using mini-batches of size 20 for the NIPS corpus and 100 for other corpora. For PubMed and Wikipedia, we chose mini-epoch subsets of size 5 ; 000 documents and processed for 5 passes. We used a step-size schedule of scale ( (cid:17) + t ) (cid:20) as in original SCVB0 for global parameters, where t is the document iteration with scale = 100 : 0 , (cid:17) = 1000 : 0 and (cid:20) = 0 : 9 . For document-level parameters, we used the scale = 1 : 0 , (cid:17) = 10 : 0 and (cid:20) = 0 : 9 , with t referring here to the word iteration of the current document. In case of PubMed corpus, we found out that original SCVB0 and SVI tend to stuck in the initial regime for document-level step-size parameter (cid:17) = 1 : 0 which we later fixed by setting (cid:17) = 10 : 0 , while SparseSCVB0 didn't suffer from this problem due to the extra randomness from the sparse sampled updates. Finally, we choose hyper-parameters (cid:11) = 0 : 1 and (cid:12) = 0 : 01 and burn-in pass of 5 for each document in all LDA experiments. For SparseSCVB0, we used sample size S = 5 to approximate (cid:13) ( pseudo ) and (cid:28) = 1 =K for the sparsification heuristic on local parameters. To study the acceleration benefits of our approach, we evaluated the runtime performance per Figure 3: Comparison of average log-likelihood vs: Time for LDA on iteration on the number topics and the number of iterations. In Figure 1(a), SparseSCVB0 is compared to original SCVB0 in terms of the average runtime per document iteration as a function of the number topics. We see that original SCVB0 requires average linear runtime due to O ( K ) operations to compute collapsed variational distribution, while the average runtime for SparseSCVB0 grows sublinearly in K , due to O ( k d ) operations instead of O ( K ) operations. SparseSCVB0 starts with approximately O ( K ) operations in its initial stage of iterations, but it starts getting a benefit from sparsification heuristic after burning in, as shown in Figure 1(b) for K = 10 ; 000 . To evaluate the performance in terms of learned topic quality, we start by comparing all of the algorithms in qualitative experiments (see Table 1, Table 2, and Table 3) where we show randomly selected example topics, while all the models were trained on the NIPS, PubMed, and Wikipedia corpus for K = 500 , K = 1 ; 000 , and 1 ; 000 , respectively. To get a quantitative insight we evaluated the topics using the per topic coherence metric, which measures the semantic quality of a topic based on the W most probable words for the topics ( Mimno et al., 2011), thereby approximating the user viewing experience. In Figure 2, we see that SparseSCVB0 generates better quality topics with higher coherence scores than the other two models for K = 1000 with W = 10 after running all the models on each corpus for the same amount of time. The coherence performance of SparseSCVB0 increases substantially in the case of the large-scale corpus (Figure 2(c)), since it gets the opportunity to use its runtime advantage and process more documents than the other algorithms. To investigate model convergence, we measured the held-out log-probability versus wall-clock time for all the algorithms. For each experiment we held-out a set of documents ( 150 documents for NIPS, 3500 documents for Reuters, and 1000 documents for all other corpora) as test data Figure 4: Comparison of runtime per iteration for MMSGTM in terms of: and trained the model on the rest of the corpus. Then, we split each test document in half, estimated local parameters on first half and finally computed the log-likelihood of the remaining half of the document. Figure 3 shows the comparison of average log-likelihood versus wall-clock time for all four corpora. In terms of log-likelihood, SparseSCVB0 provides an approximately similar result to original SCVB0 for the small corpus, but it converged to a better solution than others in the case of large corpora like Wikipedia (see Figure 3(c)), likely due to its processing a larger number of documents. SparseSCVB0 enables the large-scale computation needed to learn high-dimensional topic models that could not feasibly be trained using previous methods due to their runtime complexity in the number of documents and/or topics. We show randomly selected topics from the LDA model with K = 10 ; 000 in Table 5. This big topic model was trained for 36 hours using SparseSCVB0 on Wikipedia. We performed a dense initialization, running original SCVB0 for the first 5 hours, which was found to help avoid local optima. We also conducted experiments to evaluate the performance of SparseSCVB0 for MMSGTM and compare with original SCVB0. In all MMSGTM experiments, we kept the same step size schedule for global parameters as scale = 1 : 0 , (cid:17) = 5 : 0 and (cid:20) = 0 : 9 , but for local parameter updates we maintain a separate step-size schedule of scale ( (cid:17) + t ) (cid:20) for each input word, with t referring to the number of times we processed this input word, while (cid:17) and (cid:20) values remained the same. For simulated annealing of SparseSCVB0, we used T 0 = 0 : 00001 , (cid:20) = 0 : 9 , (cid:22) = 5 and (cid:21) = j context j with a context size of 5 . We kept the same number of document burn-in passes as we did for the LDA experiments. In Figure 4, we show the runtime improvement of SparseSCVB0 over original SCVB0 for MMSGTM in a similar experiment to the one for LDA. For MMSGTM, SparseSCVB0 substantially outperforms original SCVB0 by processing each document in amortized O (1) time. We provide qualitative results in the case of MMSGTM model by showing several top words in the top 2 topics for an input word using original SCVB0 and SparseSCVB0 in Table 4 for K = 500 . As for LDA, SparseSCVB0 allows us to generate topics with higher coherence scores compared to the original SCVB0 after running for the same amount of time (Figure 5) on both small and large corpora. This paper introduced SparseSCVB0, a sparse version of the SCVB0 inference algorithm which performs fast, scalable high-dimensional topic model inference. SparseSCVB0 leverages stochasticity to scale well in both the corpus size and in the number of topics. It operates in the collapsed representation of topic models which leads to fast convergence while providing an improved variational bound. We show that SparseSCVB0 reduces the operational complexity for the variational Bayes update of online topic models from O ( K ) to O ( k d ) time for LDA and amortized O (1) time for MMSGTM. We evaluated and compared the performance of our approach with state-of-the-art models such as original SCVB0 and SVI to demonstrate that SparseSCVB0 converges much more efficiently, while maintaining high quality topics with a better per-topic coherence score.
abstain abstain abstain objective method objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective method objective method objective objective other other method other method other other other other other other other other other other abstain other other method other other other method other other other other other other other other other method abstain abstain abstain other abstain method abstain method method abstain abstain abstain abstain method method abstain method method abstain method method abstain method method abstain abstain abstain abstain abstain other other abstain method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain other abstain abstain result abstain result abstain method abstain abstain result objective