Contrastive Learning Using Spectral Methods
In many natural settings, the analysis goal is not to characterize a single data set in isolation, but rather to understand the difference between one set of observations and another. For example, given a background corpus of news articles together with writings of a particular author, one may want a topic model that explains word patterns and themes specific to the author. Another example comes from genomics, in which biological signals may be collected from different regions of a genome, and one wants a model that captures the differential statistics observed in these regions. This paper formalizes this notion of contrastive learning for mixture models, and develops spectral algorithms for inferring mixture components specific to a foreground data set when contrasted with a background data set. The method builds on recent moment-based estimators and tensor decompositions for latent variable models, and has the intuitive feature of using background data statistics to appropriately modify moments estimated from foreground data. A key advantage of the method is that the background data need only be coarsely modeled, which is important when the background is too complex, noisy, or not of interest. The method is demonstrated on applications in contrastive topic modeling and genomic sequence analysis.
This paper tackles the problem of learning a 
      mixture model to capture properties of "foreground data" which are not 
      present in background data. The proposed technique extends recent work 
      on method of moments and tensor factorization. The novelty in the 
      paper is (i) the observation that one can subtract moments before 
      factorization and (ii) developing a variant of the robust power method 
      from Anandkumar et al. 2013 that doesn't require positive 
      semidefiniteness. The idea presented in the paper is simple, 
      natural, and novel. The main weakness is that the paper doesn't really 
      compare the proposed method with several natural baselines either 
      theoretically or empirically. 073: the paper says that the 
      stated goal is not to discriminate between foreground and background, 
      but isn't this exactly the intuition of being contrastive? 
      383: In the case where the foreground words are disjoint from the 
      rest (B and C), the intuition that you only need enough background 
      data to get the signs of the topics in B and C correct seems right. 
      However, the paragraph then talks about the accuracy of the foreground 
      topics, which doesn't relevant. I would have expected to see some 
      statement characterizing the number of samples you need to get the 
      sign correct. Also, I normally think of the foreground as being 
      captured by a modest number of topics and that the background is 
      enormous and requires a lot more, so $K$ would have to be pretty 
      large, close to $D$. So footnote 1, which talks about one topic, seems 
      pretty unmotivated. The fact that power iteration requires 
      multiple passes over the training data ($K$ times the number of power 
      iteration updates times the number of restarts of the power tensor 
      method) is somewhat discouraging, since one of the benefits of these 
      method of moments approaches is that you can take one pass over the 
      data. Could you randomly project the data onto $O(K)$ dimensions, 
      building up a much smaller dense tensor that you can store in memory? 
      Baseline 1: estimate the foreground model and background models 
      separately. Under the assumption that the model is well-specified, 
      then the normal spectral method of moments will provide a consistent 
      estimator of both models, from which it is easy to detect which are 
      the common topics. What can you say about this method theoretically 
      (it is consistent after all) and empirically? Baseline 2: 
      simply use EM to do parameter estimation in the full generative model. 
      Is local optima is actually a problem in this setting? My experience 
      is that EM works pretty well anyway, and any sort of empirical 
      statement about this approach must compare with EM. Even if EM 
      outperforms spectral, you can always use the latter as initialization 
      to the former. Theoretically, I think it's interesting to think 
      about the behavior of the maximum likelihood estimator in this case, 
      which in some sense tries to spread its modeling efforts evenly, 
      whereas the contrastive estimator seems more targeted towards what you 
      want. So there might be a regime that the contrastive estimator is 
      better, even theoretically? One piece of work which is not 
      directly relevant but has a similar high-level flavor is contrastive 
      estimation: 
      http://www.cs.cmu.edu/~nasmith/papers/smith+eisner.acl05.pdf 
      Actually, now that I think about it, the intuition is somewhat 
      different - instead of using all the background data, we are using a 
      subset which is "close" to the foreground data.