Sparse Additive Text Models with Low Rank Background
The sparse additive model for text modeling involves the sum-of-exp computing, whose cost is consuming for large scales. Moreover, the assumption of equal background across all classes/topics may be too strong. This paper extends to propose sparse additive model with low rank background (SAM-LRB) and obtains simple yet efficient estimation. Particularly, employing a double majorization bound, we approximate log-likelihood into a quadratic lower-bound without the log-sumexp terms. The constraints of low rank and sparsity are then simply embodied by nuclear norm and ℓ 1 -norm regularizers. Interestingly, we find that the optimization task of SAM-LRB can be transformed into the same form as in Robust PCA. Consequently, parameters of supervised SAM-LRB can be efficiently learned using an existing algorithm for Robust PCA based on accelerated proximal gradient. Besides the supervised case, we extend SAM-LRB to favor unsupervised and multifaceted scenarios. Experiments on three real data demonstrate the effectiveness and efficiency of SAM-LRB, compared with a few state-of-the-art models.
This paper presents a model inspired by the SAGE 
      (Sparse Additive GEnerative) model of Eisenstein et al. The authors use a 
      different approach for modeling the "background" component of the model. 
      SAGE uses the same background model for all; the authors allow different 
      backgrounds for different topics/classification labels/etc., but try to 
      keep the background matrix low rank. To make inference faster when using 
      this low rank constraint, they use a bound on the likelihood function that 
      avoids the log-sum-exp calculations from SAGE. Experimental results are 
      positive for a few different tasks. Quality: The paper is 
      solid. The idea of using a low-rank background matrix makes a lot of sense 
      for SAGE, and the use of the double majorization bound and robust PCA 
      learning algorithm seem like good ideas. It works in practice in terms of 
      classification accuracy and perplexity, doing at least as well as SAGE but 
      is simultaneously faster. All in all, I think it's a nice contribution. 
      The paper is clear and the approach is sound. I had some questions, which 
      are listed below, but in general they are minor things. I would 
      have liked to have seen a little bit more analysis/perspective on why it's 
      working better than SAGE, especially if the reason happens to relate to 
      the use of multiple background vectors with the low-rank constraint. One 
      could also try SAGE with multiple background vectors, right? Presumably 
      this would not work as well, because something like the low-rank 
      constraint would be needed to help bias the learning, but I would prefer 
      more discussion on what exactly the multiple background vectors buys you 
      in practice. Clarity: For the most part, the paper is 
      clearly-written. It does draw upon lots of other work which is not fully 
      described for space reasons, but the core ideas are mostly self-contained. 
      I did have some questions here and there, which perhaps the authors can 
      clarify. I deduce from Algorithm 1 that \nu is set by hand to be 
      0.05. But is \mu also set by hand? Were the same values used for all 
      experiments or were they tuned for each experiment separately? How 
      sensitive is the performance to the particular values used? Giving some 
      more perspective on this point with more experimental results would be 
      very helpful. In Figure 2, what is lb-quad-loc? I couldn't find it 
      in the text. I suspect it's lb-quad-j, which is in the text. I guess 
      lb-quad-j should be changed to lb-quad-loc. The figures are very difficult 
      to understand in black-and-white; please make them easier to read, both by 
      increasing the size of the fonts used and using a representation that can 
      be understood in black and white. Around line 140, "bound proposed 
      by Jaakkola and Jordan [6]" -- I think you meant [14] there instead of 
      [6]. Originality: This paper is not extremely original, 
      but it's certainly not trivial either. I think that several members of the 
      NIPS community could have come up with something like this if they took 
      the time to think about it, but some points in here are rather neat, like 
      the improvement through optimization of the variational parameters in the 
      bound and the connection to robust PCA. Significance: SAGE 
      has been used by a few research groups and seems to be generally known by 
      researchers, so having an improvement to SAGE has the potential for 
      significant impact. I like also that there are some simple ways to extend 
      the approach due to its clean formulation (e.g., different norms could be 
      considered). At the end of the day, it's incremental work, but it's done 
      well and should get published at some point.