  Relying on crowdsourced workers, data crowdsourcing platforms are able to
efficiently provide vast amounts of labeled data. Due to the variability in the
annotation quality of crowd workers, modern techniques resort to redundant
annotations and subsequent label aggregation to infer true labels. However,
these methods require model updating during the inference, posing challenges in
real-world implementation. Meanwhile, in recent years, many data labeling tasks
have begun to require skilled and experienced annotators, leading to an
increasing demand for long-term annotators. These annotators could leave
substantial historical annotation records on the crowdsourcing platforms, which
can benefit label aggregation, but are ignored by previous works. Hereby, in
this paper, we propose a novel label aggregation technique, which does not need
any model updating during inference and can extensively explore the historical
annotation records. We call it SuperLA, a Supervised Label Aggregation method.
Inside this model, we design three types of input features and a
straightforward neural network structure to merge all the information together
and subsequently produce aggregated labels. Based on comparison experiments
conducted on 22 public datasets and 11 baseline methods, we find that SuperLA
not only outperforms all those baselines in inference performance but also
offers significant advantages in terms of efficiency.

Human-Computer Interaction;Machine Learning