This paper provides a heuristic pretraining for topic models. While we consider latent Dirichlet allocation LDA here, our pretraining can be applied to other topic models. Basically, we use collapsed Gibbs sampling CGS to update the latent variables. However, after every iteration of CGS, we regard the latent variables as observable and construct another LDA over them, which we call LDA over LDA LoL. We then perform the following two types of updates: the update of the latent variables in LoL by CGS and the update of the latent variables in LDA based on the result of the preceding update of the latent variables in LoL. We perform one iteration of CGS for LDA and the above two types of updates alternately only for a small, earlier part of the inference. That is, the proposed method is used as a pretraining. The pretraining stage is followed by the usual iterations of CGS for LDA. The evaluation experiment shows that our pretraining can improve test set perplexity.
['Tomonari Masada', 'Atsuhiro Takasu']
Heuristic Pretraining for Topic Models