Can Large Language Models Capture Human Annotator Disagreements?
Human annotation variation (i.e., annotation disagreements) is common in NLP
and often reflects important information such as task subjectivity and sample
ambiguity. While Large Language Models (LLMs) are increasingly used for
automatic annotation to reduce human effort, their evaluation often focuses on
predicting the majority-voted "ground truth" labels. It is still unclear,
however, whether these models also capture informative human annotation
variation. Our work addresses this gap by extensively evaluating LLMs' ability
to predict annotation disagreements without access to repeated human labels.
Our results show that LLMs struggle with modeling disagreements, which can be
overlooked by majority label-based evaluations. Notably, while RLVR-style
(Reinforcement learning with verifiable rewards) reasoning generally boosts LLM
performance, it degrades performance in disagreement prediction. Our findings
highlight the critical need for evaluating and improving LLM annotators in
disagreement modeling. Code and data at
https://github.com/EdisonNi-hku/Disagreement_Prediction.
685bbdb2aafa002dfd2bdeb4
Jingwei Ni
claimed_verified
2025-06-25T11:04:06.025Z
64493579b6ac93fe65152f4f
/avatars/4047cf8549f162994f44101b4e7c27fa.svg
Jingwei Ni
user
JingweiNi
685bbdb2aafa002dfd2bdeb5
Yu Fan
685bbdb2aafa002dfd2bdeb6
Vilém Zouhar
685bbdb2aafa002dfd2bdeb7
Donya Rooein
685bbdb2aafa002dfd2bdeb8
Alexander Hoyle
685bbdb2aafa002dfd2bdeb9
Mrinmaya Sachan
685bbdb2aafa002dfd2bdeba
Markus Leippold
685bbdb2aafa002dfd2bdebb
Dirk Hovy
685bbdb2aafa002dfd2bdebc
Elliott Ash
Can Large Language Models Capture Human Annotator Disagreements?
Human annotation variation (i.e., annotation disagreements) is common in NLP
and often reflects important information such as task subjectivity and sample
ambiguity. While Large Language Models (LLMs) are increasingly used for
automatic annotation to reduce human effort, their evaluation often focuses on
predicting the majority-voted "ground truth" labels. It is still unclear,
however, whether these models also capture informative human annotation
variation. Our work addresses this gap by extensively evaluating LLMs' ability
to predict annotation disagreements without access to repeated human labels.
Our results show that LLMs struggle with modeling disagreements, which can be
overlooked by majority label-based evaluations. Notably, while RLVR-style
(Reinforcement learning with verifiable rewards) reasoning generally boosts LLM
performance, it degrades performance in disagreement prediction. Our findings
highlight the critical need for evaluating and improving LLM annotators in
disagreement modeling. Code and data at
https://github.com/EdisonNi-hku/Disagreement_Prediction.
LLMs struggle to predict human annotation disagreements, contrary to their performance in predicting majority labels, and RLVR-style reasoning exacerbates this issue.
Large Language Models RLVR-style Reinforcement learning with verifiable rewards disagreement modeling