Large-Scale Bayesian Multi-Label Learning via Topic-Based Label Embeddings
We present a scalable Bayesian multi-label learning model based on learning lowdimensional label embeddings. Our model assumes that each label vector is generated as a weighted combination of a set of topics (each topic being a distribution over labels), where the combination weights (i.e., the embeddings) for each label vector are conditioned on the observed feature vector. This construction, coupled with a Bernoulli-Poisson link function for each label of the binary label vector, leads to a model with a computational cost that scales in the number of positive labels in the label matrix. This makes the model particularly appealing for real-world multi-label learning problems where the label matrix is usually very massive but highly sparse. Using a data-augmentation strategy leads to full local conjugacy in our model, facilitating simple and very efficient Gibbs sampling, as well as an Expectation Maximization algorithm for inference. Also, predicting the label vector at test time does not require doing an inference for the label embeddings and can be done in closed form. We report results on several benchmark data sets, comparing our model with various state-of-the art methods.
This paper is about a new Bayesian method for multi label learning. The goal is to classify accurately in settings where there are many potential labels but only a few of them apply to each data point. The basis of the new results is a new generative model for the label vector of each example. Specifically the label vector y_n of the n-th example is generated as  y_n = f(V(\sigma(Wx_n)),where Wx_n is a lower dimensional projection of the n-th instance x_n, followed by an element-wise sigmoid activation \sigma. The final operation f corresponds to drawing Poisson random variables with rates given by V(\sigma(Wx_n)) and thresholding these so-called latent counts by taking the minimum with 1. The inference problem boils down to (heuristically) estimating various parameters of the model using Gibbs sampling and EM. In doing so, the authors observe that the latent counts are sufficient statistics of the model parameters. Evaluation: I found the generative model interesting and natural as it combines low-rank approximations with non-linearities in an interesting way. It can also be interpreted as a 3 layer neural net. I'm not sure however that inference in this model is really feasible on larger problems. The authors evaluate the approach on four tiny data sets. The largest one has fewer than 20k samples and one has as few as 40 test samples. It's not clear to me that these benchmarks are representative of real problems. The authors have yet to demonstrate that this approach will be competitive with, say, WSABIE on even medium size data sets (doesn't fit in memory but on a single machine). In particular, I don't follow the narrative in this paper which repeatedly emphasizes the "large-scale" or "massive data" aspect when there isn't even a medium data benchmark.At any rate, the multi label problem is also interesting for small data and in this regime the approach appears to be competitive with or better than state of the art. Complementing stronger experimental results, it would also be interesting to have more theoretical support for the inference method. Currently the approach is heuristic. Small remark:- The g() function in (10) appears to be a typo. If it's supposed to be there, please define it.