Robust Spectral Inference for Joint Stochastic Matrix Factorization
Spectral inference provides fast algorithms and provable optimality for latent topic analysis. But for real data these algorithms require additional ad-hoc heuristics, and even then often produce unusable results. We explain this poor performance by casting the problem of topic inference in the framework of Joint Stochastic Matrix Factorization (JSMF) and showing that previous methods violate the theoretical conditions necessary for a good solution to exist. We then propose a novel rectification method that learns high quality topics and their interactions even on small, noisy data. This method achieves results comparable to probabilistic techniques in several domains while maintaining scalability and provable optimality.
Summary of the paper: This paper improves Anchor Word topic models [1,2] with rectifying operations in order to satisfy a newly introduced double non-negativity constraint, which is proved to be necessary by authors. Experimental results present the rectification indeed improves various aspects of topic modeling performance.Quality: Although most contents of the paper feel solid, experimental part can be further improved. Figure 4, in particular, have a lot of missing data points and too small to read it easily. I also want to encourage authors to provide training time comparisons because one of the largest benefit of using [1,2] was computation even though I feel the rectification will not require extreme computation.Clarity: Mostly clear.Originality: I believe authors are the first to employ rectification in Anchor Word topic models. Even their method appears to be a straight-forward extension, authors provided enough discussions and theories that justify their idea in a broad spectrum.Significance: Although their idea will not begin an entirely new topic modeling methodology, I feel their algorithm can be easily employed various NMF models and can be directly utilized in large scale topic modeling as authors suggested in Section 6.[1] Arora, S., Ge, R., & Moitra, A. (2012, October). Learning topic models--going beyond SVD. In Foundations of Computer Science (FOCS), 2012 IEEE 53rd Annual Symposium on (pp. 1-10). IEEE.[2] Arora, S., Ge, R., Halpern, Y., Mimno, D., Moitra, A., Sontag, D., & Zhu, M. (2012). A practical algorithm for topic modeling with provable guarantees. arXiv preprint arXiv:1212.4777.