Max-Margin Majority Voting for Learning from Crowds
Learning-from-crowds aims to design proper aggregation strategies to infer the unknown true labels from the noisy labels provided by ordinary web workers. This paper presents max-margin majority voting (M 3 V) to improve the discriminative ability of majority voting and further presents a Bayesian generalization to incorporate the flexibility of generative methods on modeling noisy observations with worker confusion matrices. We formulate the joint learning as a regularized Bayesian inference problem, where the posterior regularization is derived by maximizing the margin between the aggregated score of a potential true label and that of any alternative label. Our Bayesian model naturally covers the Dawid-Skene estimator and M 3 V. Empirical results demonstrate that our methods are competitive, often achieving better results than state-of-the-art estimators.
This paper formulates learning from crowds as a max-margin learning problem. To further consider the uncertainty of the model, it provides two Bayesian solutions: a Bayesian mean estimator and a Gibbs estimator to the posterior regularization term induced from the hinge-loss term in the original max-margin learning problem. Overall, the paper demonstrates a new successful application within the "Regularized Bayesian" framework. The discriminative function for each task and each label was defined and the resulting margin was characterized. The resulting CrowdSVM problem differs from multi-task SVM, in the aspect that multi-class SVM handles individual features, while CrowdSVM represent a single task as dependent points lies on the axises. To consider generative perspective, the regularized Bayesian framework was introduced. The inference techniques (both variational inference and Gibbs sampling based on data augmentation for SVMs) were borrowed from previous RegBayes papers. I have some concerns in the experiments. Since the entropy-based methods are the most competitive ones with max-margin methods, it worth further comparing these two class of methods. The online learning results seem weird at the first sight. To my understanding, the workers were fixed and the tasks comes in a stream. Assuming the tasks are distributed in some regular patterns, then the online result should recover the batch result. However, in the experiments small batch sizes perform significantly worse. Why? The paper did not give a clear answer. Figure 5 in the supplementary material shows some fluctuation in performance, which can be smoothed by repeating experiments.Although having some concern in the experiments, the paper provides a novel max-margin learning strategy for learning from crowd. It is another empirically successful application for Bayesian discriminative learning.