Many machine learning systems today are trained on large amounts of humanannotated data. Annotation tasks that require a high level of competency make data acquisition expensive, while the resulting labels are often subjective, inconsistent, and may contain a variety of human biases. To improve data quality, practitioners often need to collect multiple annotations per example and aggregate them before training models. Such a multi-stage approach results in redundant annotations and may often produce imperfect "ground truth" labels that limit the potential of training supervised machine learning models. We propose a new end-to-end framework that enables us to: (i) merge the aggregation step with model training, thus allowing deep learning systems to learn to predict ground truth estimates directly from the available data, and (ii) model difficulties of examples and learn representations of the annotators that allow us to estimate and take into account their competencies. Our approach is general and has many applications, including training more accurate models on crowdsourced data, ensemble learning, as well as classifier accuracy estimation from unlabeled data. We conduct an extensive experimental evaluation of our method on 5 crowdsourcing datasets of varied difficulty and show accuracy gains of up to 25% over the current state-of-the-art approaches for aggregating annotations, as well as significant reductions in the required annotation redundancy. We further conduct an ablation study to evaluate the effect of both end-to-end learning and instance features and show that both contribute to the performance gains achieved by the proposed method.
Update after author response: 
I would like to thank the authors for their thoughtful response, and for addressing some of the concerns raised by the reviewers. One of my main complaints arose from a misunderstanding that none of the baselines model worker competencies and task difficulty. The authors clarified that MMCE does model those, making it a competitive baseline. Given that and the other additions to the draft, I am changing my assessment from 3 to 6.
--------------------------- 
In this paper, the authors extend the classical probabilistic model of Dawid-Skene (DS) for predicting the final label of crowdsourced tasks. The extensions include explicit modeling of image difficulty and worker competence as a function of image and worker features respectively, as well as a more expressive formulation of learned functions in terms of neural networks (NNs). The authors show that the proposed approach outperforms several baselines on 5 different datasets.
The problem studied in the paper is relevant since more and more data is needed to train ever more complex models, and often crowdsourcing is the way to generate such data. The paper is clearly written and conveys its central idea concisely. I liked the paper but I believe the contribution to be incremental in its current state. Here are the main issues in my opinion:
1. The presentation suggests that somehow the proposed approach is novel in its end-to-end framework. However, the central idea is very similar to Dawid-Skene (1979), and its subsequent augmentations like the ones in Carpenter (2008) that model both image difficulty and worker competence. These previously proposed models are also end-to-end approaches so that they can infer worker competence and image difficulties while also outputting a final label. Therefore, I believe the novel contributions are then slightly different formulation of these variables, and using NNs to learn the function parameters. I see that the proposed model can be somewhat more general since it uses semantic image features (as opposed to indicator features) but then it learns worker embeddings starting from random initializations which will not generalize to new workers, necessitating a re-run of the whole EM loop.
2. If Q is a stochastic matrix, its rows should sum to 1. I don’t understand the summation on Q in eq. 4. I am also a little confused by eq. 5 since the joint likelihood should just be the prior times the conditional likelihood of the observed annotations. At least trying to do it in my head doesn’t lead to what’s presented in eq. 5. It may just be an issue of notations which can be simplified by using conditional distributions instead of expectations of indicator random variables (same quantity). 
3. As far as I understand, none of the baselines considered explicitly model the task difficulty and worker competence. Therefore, the proposed model enjoys this extra level of expressiveness making its superior performance relatively unsurprising. I skimmed Zhou et al. (2015) and simple DS was already competitive on some of the datasets so I would think inclusion of its successors that model image difficulty and worker competence could perform quite well. 
3. Minor issue but eq. 6 is not majority vote as stated just above. It’s the maximum likelihood estimate P(y_i = k | D) assuming each worker is an unbiased estimator of the true label. 
4. Minor formatting issues: “learn a models” (sec 3.2), “lantent variables” (sec 4) “and by merging” (sec 5). 
In summary, even though I like that the paper is clearly written and tackles an important problem, I am not convinced that the contribution is substantial or the experiments very insightful. I would recommend addressing these issues and resubmitting the paper.
Carpenter, B. (2008). Multilevel bayesian models of categorical data annotation. Unpublished manuscript, 17(122), 45-50.