Double or Nothing: Multiplicative Incentive Mechanisms for Crowdsourcing
Crowdsourcing has gained immense popularity in machine learning applications for obtaining large amounts of labeled data. Crowdsourcing is cheap and fast, but suffers from the problem of low-quality data. To address this fundamental challenge in crowdsourcing, we propose a simple payment mechanism to incentivize workers to answer only the questions that they are sure of and skip the rest. We show that surprisingly, under a mild and natural "no-free-lunch" requirement, this mechanism is the one and only incentive-compatible payment mechanism possible. We also show that among all possible incentive-compatible mechanisms (that may or may not satisfy no-free-lunch), our mechanism makes the smallest possible payment to spammers. Interestingly, this unique mechanism takes a "multiplicative" form. The simplicity of the mechanism is an added benefit. In preliminary experiments involving over several hundred workers, we observe a significant reduction in the error rates under our unique mechanism for the same or lower monetary expenditure.
Authors consider a setting in which workers can either choose to answer or to "skip" the question. They define the "no free-lunch condition", which states that a payment should give reward zero if all answers (in the ground truth subset) are wrong.The payment rule is very simple: either zero reward if one or more answers are wrong or a reward that grows (decreases for each skipped answer) multiplicatively. Theoretical results regarding the correctness and uniqueness of the proposed method are derived. The results are generalized to the case of different degrees of confidence and evaluated in a real experiment.It is indeed surprising that the payment rule that has to give zero reward as soon as the worker answers incorrectly one question.The key idea is very simple and the authors did a good job on the theoretical side. I have doubts, however, that NIPS should be the conference for this type of work. Nevertheless, I have checked in depth until page 14 in the supplementary material, which includes the proof of uniqueness (the most important). I would like to hear the opinion of the authors on the following points:1- I can replace the factor $T^{G-C}$ by $T^{k(G-C)}$, for arbitrary $k$. This is a more general scoring rule that also satisfies the no-free lunch condition. For k=1, we recover the proposed rule. For k > 1 we penalize more the skips and or k < 1 (and here comes a problem) we penalize less the skips. I say this is a problem because in the limit of k->0 we do not penalize skipping, and the rule would give full reward for a all-skipped answer, something clearly not desired.2- Concerning the empirical evaluation, there is significant selection bias in the experiments: only the good workers "having at least 95% of their prior work approved by the requesters" are selected for the experiment (lines 1115-1116 suppl). This seems to be a critical point because the main motivation is to defer spammers, and this is not really evaluated. Authors should mention that as an (additional) limitation in the beginning of section 5.Related to this, if the main goal is to compare with the traditional method, wouldn't be worthy to make a simulation where the p's are generated randomly without the confounders of a real experiment?3a- As authors note in lines 234-238, this payment rule is indeed restrictive. For example, given N questions, performing three different tasks is different than performing one task only, since failing one single question will lead to zero reward in the later and will not in the former. It is mentioned that in reality, "workers play a repeated game", but this is not considered at all in the analysis/evaluation.3b- Finally, and related to the previous point, note that the proposed rule is more informative than the standard and can leak information, in the sense that zero reward means that one question of the gold standard was answered incorrectly. This could be used by a group of cheaters to learn the subset of gold standard questions.I have other minor remarks:- In Axiom 1, why not replacing the inequality and equality just with the condition $\sum_{i=1}^G\one\{x_i=-1\} = G$? - line 510 suppl: $kappa$ should be $\kappa$ - line 577 suppl: $j\in[\eta+\gamma]$ should be$j\in[1,\eta+\gamma]$. This should also be corrected many times later on. - line 577 suppl: $j\in\{\eta+1,\eta+\gamma\}$ should be $j\in\{\eta+1,\hdots,\eta+\gamma\}$ - It is not clear why the wrong answers are changed (eg, lines 580-588 suppl). - line 738 suppl: Algorithm ?? - (lines 1199, 1209) suppl: the user is informed that the bonus is not altered if a question is skipped, but that is not true, since skipping decreases reward exponentially.