LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks
<|reference_start|>LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks: There is an increasing trend towards evaluating NLP models with LLM-generated judgments instead of human judgments. In the absence of a comparison against human data, this raises concerns about the validity of these evaluations; in case they are conducted with proprietary models, this also raises concerns over reproducibility. We provide JUDGE-BENCH, a collection of 20 NLP datasets with human annotations, and comprehensively evaluate 11 current LLMs, covering both open-weight and proprietary models, for their ability to replicate the annotations. Our evaluations show that each LLM exhibits a large variance across datasets in its correlation to human judgments. We conclude that LLMs are not yet ready to systematically replace human judges in NLP.<|reference_end|>
@article{bavaresco2024llms,
    title={LLMs instead of Human Judges? A Large Scale Empirical Study across 20
  NLP Evaluation Tasks},
    author={Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond
  Elliott, Raquel Fern'andez, Albert Gatt, Esam Ghaleb, Mario Giulianelli,
  Michael Hanna, Alexander Koller, Andr'e F. T. Martins, Philipp Mondorf, Vera
  Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro
  Suglia, Aditya K Surikuchi, Ece Takmaz, Alberto Testoni},
    journal={arXiv preprint arXiv:2406.18403},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.18403},
    primaryClass={cs.CL}
}