(Almost) No Label No Cry
In Learning with Label Proportions (LLP), the objective is to learn a supervised classifier when, instead of labels, only label proportions for bags of observations are known. This setting has broad practical relevance, in particular for privacy preserving data processing. We first show that the mean operator, a statistic which aggregates all labels, is minimally sufficient for the minimization of many proper scoring losses with linear (or kernelized) classifiers without using labels. We provide a fast learning algorithm that estimates the mean operator via a manifold regularizer with guaranteed approximation bounds. Then, we present an iterative learning algorithm that uses this as initialization. We ground this algorithm in Rademacher-style generalization bounds that fit the LLP setting, introducing a generalization of Rademacher complexity and a Label Proportion Complexity measure. This latter algorithm optimizes tractable bounds for the corresponding bag-empirical risk. Experiments are provided on fourteen domains, whose size ranges up to ≈300K observations. They display that our algorithms are scalable and tend to consistently outperform the state of the art in LLP. Moreover, in many cases, our algorithms compete with or are just percents of AUC away from the Oracle that learns knowing all labels. On the largest domains, half a dozen proportions can suffice, i.e. roughly 40K times less than the total number of labels.
Summary: This paper presents new algorithms for the task of learning with label proportions (LLP). One of the proposed algorithm, the Laplacian Mean Map algorithm, extends previously proposed algorithm called Mean Map algorithm. It doesn’t pose homogeneity assumption but uses the Laplacian of the bag similarities. The Alternating Mean Map algorithm, an iterative variant of the LMM that uses LMM for initialization, is also investigated. The paper presents generalization bounds for LMM and AMM. Experimental results using different datasets and domains show that the proposed methods achieve the state-of-the-art performance in many cases, yet both of the algorithms scale to a large dataset.Quality: Detailed proof and thorough experiments clearly support the soundness and effectiveness of the proposed algorithms. Clarity: The paper is written very clearly. The supplemental material well complement the proofs and detailed experimental results.  Originality: Although application of manifold regularization and iterative procedure themselves is a natural extension to the Mean Map algorithm, it is worth publishing. Significance: Since the algorithms are easy to implement and scale to a large dataset, they will soon become strong baselines in LLP settings. Privacy-preserving technologies may also benefit from these algorithms.