Annotator-Centric Active Learning for Subjective NLP Tasks
<|reference_start|>Annotator-Centric Active Learning for Subjective NLP Tasks: Active Learning (AL) addresses the high costs of collecting human annotations by strategically annotating the most informative samples. However, for subjective NLP tasks, incorporating a wide range of perspectives in the annotation process is crucial to capture the variability in human judgments. We introduce Annotator-Centric Active Learning (ACAL), which incorporates an annotator selection strategy following data sampling. Our objective is two-fold: (1) to efficiently approximate the full diversity of human judgments, and (2) to assess model performance using annotator-centric metrics, which emphasize minority perspectives over a majority. We experiment with multiple annotator selection strategies across seven subjective NLP tasks, employing both traditional and novel, human-centered evaluation metrics. Our findings indicate that ACAL improves data efficiency and excels in annotator-centric performance evaluations. However, its success depends on the availability of a sufficiently large and diverse pool of annotators to sample from.<|reference_end|>
@article{van der meer2024annotator-centric,
    title={Annotator-Centric Active Learning for Subjective NLP Tasks},
    author={Michiel van der Meer, Neele Falk, Pradeep K. Murukannaiah, Enrico
  Liscio},
    journal={arXiv preprint arXiv:2404.15720},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.15720},
    primaryClass={cs.CL}
}