Dynamic Rank Factor Model for Text Streams
We propose a semi-parametric and dynamic rank factor model for topic modeling, capable of (i) discovering topic prevalence over time, and (ii) learning contemporary multi-scale dependence structures, providing topic and word correlations as a byproduct. The high-dimensional and time-evolving ordinal/rank observations (such as word counts), after an arbitrary monotone transformation, are well accommodated through an underlying dynamic sparse factor model. The framework naturally admits heavy-tailed innovations, capable of inferring abrupt temporal jumps in the importance of topics. Posterior inference is performed through straightforward Gibbs sampling, based on the forward-filtering backwardsampling algorithm. Moreover, an efficient data subsampling scheme is leveraged to speed up inference on massive datasets. The modeling framework is illustrated on two real datasets: the US State of the Union Address and the JSTOR collection from Science.
This paper proposes the dynamic rank factor models, which can extract topics changes along time, and also discover the topic and word correlations. The two case studies using the proposed model show interesting topic changing patterns, which are aligned well with true events; and the structures between topics and words are also visualized and discussed. In general, this is a good paper. However, I have two main concerns. First, while the introduction is well motivated and the experiment section is clearly written, Section 2 is not well structured and under selling.- The authors start from dynamic factor models (section 2.1), and then talk about general topic models (without time involvement) and then come back to dynamic models again in section 2.3. This structure can be improved to avoid jump back and forth.- Section 2.2 is not related with dynamic changing, but the authors use the subscript "t" to denote the t^th document, while using "t" to denote the time t in Section 2.1 and Section 2.3, and "n_t" to denote the document at time t. This is very confusing. - Probably because the math part is dense, the proposed model in general is mixed with existing work and is not well emphasized. Even the dynamic part (Section 2.3), an important proposed work, only gives the latent factor model, but does not say anything about dynamic changing but refer back to Section 2.1- It would be stronger if the nice properties of the proposed model are discussed more, instead of just listing.Second, while the authors indicates that the proposed model is an alternative to dynamic topic models and correlated topic models, there is no comparison at all against these models in the experiment section. - The experiment shows the changing of topics posterior mean along time and the detected topics, however, there is no evaluation of topic quality, like topic coherence, etc.- This paper would be stronger with comparison against dynamic topic models and correlated topic models. - For some interesting topics, it would be nice to see the actual topic changes (the top words) along time.Some minor comments:- \alpha and \beta are used as the parameters for gamma distribution and also the parameters for Dirichlet distributions.- Figure 1, the displayed topics are the final topics after observing all documents? This should be clear.- Figure 2, it would be nice to display a topic name rather than topic index, it would be easier to interprete the meanings of the topics.- Figure 2, it's hard to read the word dendrogram, too small. It would be great if there is more discussion about the results.