β-risk: a New Surrogate Risk for Learning from Weakly Labeled Data
During the past few years, the machine learning community has paid attention to developing new methods for learning from weakly labeled data. This field covers different settings like semi-supervised learning, learning with label proportions, multi-instance learning, noise-tolerant learning, etc. This paper presents a generic framework to deal with these weakly labeled scenarios. We introduce the β-risk as a generalized formulation of the standard empirical risk based on surrogate marginbased loss functions. This risk allows us to express the reliability on the labels and to derive different kinds of learning algorithms. We specifically focus on SVMs and propose a soft margin β-SVM algorithm which behaves better that the state of the art.
The paper proposes a surrogate risk for weakly supervised learning. The risk allows for a per example weighting which can be adapted to different learning scenarios. A condition is determined under which the proposed risk is a tight bound to the 01 risk. This is used to derive an iterative optimisation algorithm. Experiments are presented for semi-supervised and noisy label learning.       
UPDATE AFTER REBUTTAL:  I still feel the value of the framework isn't fully convincing. My basic issue is that for the weakly supervised scenarios that don't already have principled algorithms, the precise reason the provided formulation is superior is unclear. For example in SSL the proposed method has a very similar flavour to self-training. I like however that there is an attempt at an unified approach to a range of such problems, and it's possible that one could do something interesting with this framework in future work.   On the technical side, I still don't quite get the optimisation proposed for beta (Line 144). As I understand, Eqn 4 implies   R_\phi^\beta(X, h) = R_\phi(S, h) + (1/m) sum_{i = 1}^m beta_i y_i h(x_i)   so it seems like optimising R_\phi^\beta(X, h) is equivalently optimising the weighted mean term. But as that's constrained to be zero in the optimisation, we're just finding any feasible beta? The authors comment that there is a constraint on beta that is needed for the above equality, but this constraint seems indeed imposed in the optimisation.  The basic idea of constructing new risks that can adapt to different learning scenarios is interesting, and in keeping with some recent work for special cases e.g. Patrini et al. 2013 for learning from label proportions. The paper begins with a decomposition of an arbitrary supervised risk into a weighted unsupervised risk, plus an additional term which is a certain beta-weighted mean. The idea is then that if one can somehow omit the second term, one can learn using the first term alone, which is purely unsupervised.  The precise decomposition given in Lemma 1 is certainly interesting, and appears novel, but seems a simple extension of Thm 3 of Patrini et al. 2013 (specifically, the result is implicit in the proof of said theorem), or Lemma 1 of   Patrini et al. Loss factorization, weakly supervised learning and label noise robustness. ICML 2016.  Thus, I think the key issue is whether the result is compellingly used for new weakly supervised learning problems. On this front, I think the paper perhaps falls a bit short. It starts with a nice observation, which perhaps is intended to be the key insight of the paper: to make the new beta risk and the original as close as possible, one must constraint the beta-weighted mean of the classifier to be zero. This appears a departure from previous work, and gives a concrete criterion to learn from weakly supervised data. However, this term depends on the unobserved labels for the instances, and so the proposed method unavoidably runs into the issue that purely learning from unlabelled data is impossible without further assumptions.  The chosen approach to get around this is, unsurprisingly, a heuristic: one guesses the labels for the instances based on the current model (reminiscent somewhat of self-training for semi-supervised learning). The framework would have been more compelling were there something cleverer that could be done here. Given that a heuristic is employed, I think the paper could have explained in more detail why the proposed scheme is more attractive than existing proposals (unified framework?).  With this, the paper seeks to iteratively optimise for a good classifier, and the set of per instance weights beta. The optimisation of the classifier is standard; for the beta weights, one seeks to minimise the new risk subject to the constraint that the weighted mean is zero. while the latter is intuitive, I was unsure as to the well posedness of this optimisation. In the second stage of the iterative algorithm of section 3, the object that is optimised is R(h, beta; X). As I understand, section 2.2 establishes that this is equivalent to R(h, beta; S) + mu(beta), where mu(beta) is the weighted mean that is constrained to 0. But isn't R(h, beta; S) further just R(h; S) by Lemma 2.1, in which case the objective is effectively mu(beta), as the other term is independent of the precise choice of beta? In which case we would simply pick any admissible beta which satisfied the constraint? (This is potentially ok, as it doesn't affect the final risk? But the issue seems unclear.)  The experiments presented are promising, showing the potential to improve upon the WellSVM in certain cases.  Overall, the paper has an interesting perspective on weakly supervised learning, albeit one that builds upon similar proposals for special cases in Patrini et al. The proposed method is simple, and seems to perform well in practice. However, the method perhaps falls short of being a theoretically justified, unified solution to weakly supervised learning: it unavoidably faces the problem of having to make assumptions to get around the lack of labels, and does so in an acceptable if not particularly insightful way.  Other comments:    - it might be more standard to use \ell instead of \mathcal{L} for the loss function    - consider using e^-x or exp(-x) instead of exp^{-x}    - explicitly show dot product or transpose in definition of Bregman divergence    - the discussion about seeking an upper bound to the 01 risk could perhaps comment on the further need for a regret bound between the 01 and the newly proposed risk. Otherwise, the optimisation would have no guarantee of actually converging to the correct solution.    - the notation ^t\beta for the iterates seems awkward to me.    - It is stated that one estimates labels based on the current set of weights beta. in the case of semi-supervised learning, the weights are initialised to 0.5. How does one pick weights in this case? Is it a random flip?    - kernel should be on X x X, not X^2?   - the proposed method requires that one learns a separate wait for every instance. When the number of instances is large, this might be problematic. Can you comment on possible ways of alleviating the burden?    - consider having markers for the points displayed in the plots.   - for citation [13], capitalize SVMs.