Learning Feature Selection Dependencies in Multi-task Learning
A probabilistic model based on the horseshoe prior is proposed for learning dependencies in the process of identifying relevant features for prediction. Exact inference is intractable in this model. However, expectation propagation offers an approximate alternative. Because the process of estimating feature selection dependencies may suffer from over-fitting in the model proposed, additional data from a multi-task learning scenario are considered for induction. The same model can be used in this setting with few modifications. Furthermore, the assumptions made are less restrictive than in other multi-task methods: The different tasks must share feature selection dependencies, but can have different relevant features and model coefficients. Experiments with real and synthetic data show that this model performs better than other multi-task alternatives from the literature. The experiments also show that the model is able to induce suitable feature selection dependencies for the problems considered, only from the training data.
Paper Summary: The paper handles a problem 
      that has garnered growing research interest: Learning sparse solutions for 
      problems with relatively small amount of samples and many features (i.e.; 
      mandate feature selection), where we try to leverage data from several 
      learning tasks over the same set of features. The assumption is that while 
      the learning tasks are different they may share some characteristics that 
      would enable us to generalize better. Specifically, the authors 
      concentrate on the case where the learning tasks may share a dependency 
      structure between the variables, though whether they are actually "active" 
      and the magnitude of their coefficients (w) in each learning task may 
      differ. The coefficient for each feature in each learning task is 
      controlled via a horseshoe prior to insure sparseness. However, instead of 
      assuming these are independent priors for each feature, the authors 
      introduce a correlation matrix (C) over the hyper parameters (u,v) that 
      control the features coefficients. They assume that matrix (C) is shared 
      between the learning tasks. In this learning setting, we are generally 
      interested in two things: (1) prediction for new data (2) detecting what 
      are the "active" features for each learning task, given by z = (w,u,v) for 
      each task. Since the integrals do not have a closed form solution they 
      resort to approximate inference using EP. For testing their algorithm the 
      authors use two datasets: A synthetic dataset which has a clear regularity 
      in the selected active features in each task (active iff mod(i/16) = j 
      where j in 1...16 and i in 1....128) and a 10x10 digitized handwritten 
      digits. In both cases they show improved results compared to a host of 
      other algorithms that do not utilize mulit-task learning or use multi-task 
      learning but assume the active features are shared or at least have no 
      dependency structure. Pros: 1. The paper includes comparison 
      to several other methods, and demonstrate improved performance for those 
      tasks. 2. The method developed is new making good use of a whole host 
      of ML related methods. These methods have been gaining interest: Sparse 
      Bayesian priors such as the horseshoe with alternative representations of 
      it, EP with various tricks to make it more efficient. Thus, the paper has 
      the side benefit of educating the readers while the authors claim 
      significant speedup for EP. 3. The intro to the horseshoe prior usage 
      is clearly written. Cons: 1. While the method is novel and 
      non-trivial the overall relevance of it for real life applications is not 
      clear, and the authors make no attempt at clarifying this. Several things 
      stand out: First, the authors make a general comment that their approach 
      is useful since the different learning tasks "share only the dependency 
      structure" (p.2, top). True, sharing exactly the same active features and 
      coefficients is a stronger assumptions, but other methods relax these 
      assumptions. Sharing (exactly) the same dependency structure actually 
      seems quite a strong assumption, and not necessarily a realistic one if 
      you think of applications such as networks in the bio-medical domain. The 
      authors claim this to be "a very flexible assumption" (p. 8 conclusions) 
      but do not support this in any way. Specifically, they only evaluate their 
      method on two datasets with very clear and unique regularities in their 
      feature dependency structure. Comparison on cases where some features are 
      shared but with no underlying dependency structure exists is warranted. 
      Finally, the authors mention only theoretical computational costs. The 
      experiments involve only small toy data. It is not clear how such a method 
      would perform compared to alternatives when number of samples (n), number 
      of tasks (k) and number of features (f) grows. The latter is especially 
      problematic as the method aimes to reconstruct the correlation matrix 
      between all features. 2. The writeup is missing crucial details 
      for demonstrating mathematical soundness required for publication. The 
      authors constantly use arguments such as "The gradient of .... can be used 
      for this task" (p. 4), "the gradient of.... with respect to .... can be 
      easily computed in terms of the ...." p. 5 Then sate "The derivations are 
      long and tedious and hence omitted here" (p. 6). Similarly, all the actual 
      implementation of the EP in Sec. 3 is only reviewed. The details should be 
      given in a matching supplementary for completeness. 3. The write 
      up is not well balanced/structured. While the main novel element is the 
      estimation of the joint dependency structure the authors spend very little 
      on it. Instead they give a clear but maybe too long intro to the horseshoe 
      prior. Specifically, Figure 1 is taken straight out of Caravlho et al 2009 
      (Fig 1,2) and the authors should at least point that out clearly and/or 
      retain only say Fig1c. Sec 3 should be extended and and requires supp (see 
      above). In the experiments the author spend almost a full page (p. 7) on a 
      synthetic data which can only be described as a sanity check. The 3 panels 
      in Fig 3 are too much. The set up of the digits experiments is not clear 
      and should be revised (e.g. where does n = 75 comes from? "100" appears as 
      the value of too many different params not defined, which is confusing). 
      Other: 1. On p. 6 the authors state they define P (which 
      is at the heart of the correlation matrix C) to have m x d params, and set 
      m = n where n is #samples. But in the multi class learning framework 
      they claim each learning task can have a different number of samples n_k, 
      though P is presumably the same. Can you explain? 2. Fig 2 the 
      annotation "f" for the top factor is missing