Inference Aided Reinforcement Learning for Incentive Mechanism Design in Crowdsourcing
Incentive mechanisms for crowdsourcing are designed to incentivize financially self-interested workers to generate and report high-quality labels. Existing mechanisms are often developed as one-shot static solutions, assuming a certain level of knowledge about worker models (expertise levels, costs of exerting efforts, etc.). In this paper, we propose a novel inference aided reinforcement mechanism that learns to incentivize high-quality data sequentially and requires no such prior assumptions. Specifically, we first design a Gibbs sampling augmented Bayesian inference algorithm to estimate workers' labeling strategies from the collected labels at each step. Then we propose a reinforcement incentive learning (RIL) method, building on top of the above estimates, to uncover how workers respond to different payments. RIL dynamically determines the payment without accessing any ground-truth labels. We theoretically prove that RIL is able to incentivize rational workers to provide high-quality labels. Empirical results show that our mechanism performs consistently well under both rational and non-fully rational (adaptive learning) worker models. Besides, the payments offered by RIL are more robust and have lower variances compared to the existing one-shot mechanisms.
Summary:  In this paper, the authors explore the problem of data collecting using crowdsourcing. In the setting of the paper, each task is a labeling task with binary labels, and workers are strategic in choosing effort levels and reporting strategies that maximize their utility. The true label for each task and workers’ parameters are all unknown to the requester. The requester’s goal is to learn how to decide the payment and how to aggregate the collected labels by learning from workers’ past answers. The authors’ proposed approach is a combination of incentive design, Bayesian inference, and reinforcement learning. At each time step, the authors use Gibbs sampling to help infer the true labels. The estimates are then fed into the reinforcement framework to decide how to choose the payments. Simulated experiments are conducted to evaluate the performance of the proposed approach.  Comments:  On the positive side, while the techniques used in the paper seem pretty standard individually, I like the idea of using Bayesian inference to obtain estimated ground truth for deciding how to set the payments. The overall approach also seems sound and works reasonably well in simulations. On the negative side, I have some concerns about the analysis (more comments below). Using only simulated experiments for evaluations is also not satisfactory (especially a big component of the paper is on learning worker strategies).  Overall, I would still slightly lean towards accepting the paper but won’t argue for it.  I have a few questions about the analysis. In Theorem 1, the authors show that the estimate of P_i (PoBC) will converge. However, the proof seems to treat P_i as a constant while it is actually a strategic choice of agent i (as described in Equation 1).  This is fine if we can show that agents always exert high effort and gives truthful reports, as stated in Theorem 2. However, the proof of Theorem 2 is based on the condition that P_i already converges.  This creates the chicken-and-egg problem (Theorem 1 relies on Theorem 2 to be true, and Theorem 2 relies on Theorem 1 to be true). I might have missed something. It would be helpful if the authors can clarify this.  In Theorem 2 and 3, it is assumed that a_t is larger than some threshold (which is a function of workers’ cost and accuracy). However, in Algorithm 2, there is a probability of \epsilon that a_t will be randomly chosen from A. To satisfy the condition, it seems we need to make all choices in A to be larger than the threshold. Does this mean that the requester needs to know workers’ cost/accuracy in advance?   Overall, the analysis on the three major components seem relatively isolated, while they influence each other. For example, when there is not enough data in the beginning, the Bayesian inference won’t be accurate, could this change workers’ strategies in the beginning and latter influence the follow-up learning (I lean towards believing this won’t happen given the simulation results, however, it is not clear from the current analysis)?  In Figure 2-(a), I am surprised to see the aggregation accuracy of the proposed approach significantly outperforms variational inference even when no noise is added (variational inference has been reported to perform consistently well in the literature). If this result is consistent across many datasets, it could be worth more discussion.  I think the paper would be much stronger if there are some real-world experiments, since real workers are usually not rational (or QR/MWU) and don’t just have two effort levels.