Spectral Methods for Supervised Topic Models
Supervised topic models simultaneously model the latent topic structure of large collections of documents and a response variable associated with each document. Existing inference methods are based on either variational approximation or Monte Carlo sampling. This paper presents a novel spectral decomposition algorithm to recover the parameters of supervised latent Dirichlet allocation (sLDA) models. The Spectral-sLDA algorithm is provably correct and computationally efficient. We prove a sample complexity bound and subsequently derive a sufficient condition for the identifiability of sLDA. Thorough experiments on a diverse range of synthetic and real-world datasets verify the theory and demonstrate the practical effectiveness of the algorithm.
This paper addresses the classical problem of unsupervised learning of latent topic model, with an extra variable called response, which can be a score. The main issue is that the topic model (and the embeddings deduced from it) may not help in learning this extra variable, as the response can be induced by phenomena that are orthogonal to the topics. The goal of the so-called supervised topic model learning is to drive the topic learning into a direction which makes it useful w.r.t. the prediction of this extra variable (by a regression).The basic model considered is the Latent Dirichlet allocation (LDA) model. The parameters of such a model can be estimated with Max-Likelihood, sampling, or recently spectral methods (methods of moments). The spectral method is based on the consistent estimate of a finite number of observable statistical moments, from which it is possible to recover the parameters thanks to simple linear algebra operations (in the original paper) or more complex ones (like tensor decomposition). Starting from previous work done on spectral learning of LDA, the paper shows how the same method  can be used to learn the response variable, by adding an extra family of observable moments, and by showing how it is possible to recover the parameters of the regression via tensor decomposition.Some experiments show that the algorithm is good w.r.t. other methods in parameter recover  on a synthetic dataset. Conversely, the method does not perform well in real data experiments, with a max. likelihood objective. This is not surprising, as the non-spectral method usually focus on direct maximization of likelihood, whereas spectral does not. Also, the spectral method does not behave well in general with small samples (compared to the information needed for a good estimate)The paper is well written, the mathematical part is solid. The algorithm described in the paper, to my opinion, is more an extension of the spectral algorithm for LDA than a real new method. Its main interest is to show that the spectral method can be also applied to the sLDA model. More work needs to be done to make this method efficient in real data problems (as it is for spectral method in general).(I think that when the authors say that « Theorem 1 is that it provides a necessary condition for a supervised LDA model to be identifiable », they mean in fact « sufficient », right?)Pros:- good adaptation of the spectral LDA learning algorithm- good theoretical work- theoretical boundsCons: - more an extension of known methods than a new contribution- not convincing in real data learning