Large Language Models as Evaluators for Scientific Synthesis
<|reference_start|>Large Language Models as Evaluators for Scientific Synthesis: Our study explores how well the state-of-the-art Large Language Models (LLMs), like GPT-4 and Mistral, can assess the quality of scientific summaries or, more fittingly, scientific syntheses, comparing their evaluations to those of human annotators. We used a dataset of 100 research questions and their syntheses made by GPT-4 from abstracts of five related papers, checked against human quality ratings. The study evaluates both the closed-source GPT-4 and the open-source Mistral model's ability to rate these summaries and provide reasons for their judgments. Preliminary results show that LLMs can offer logical explanations that somewhat match the quality ratings, yet a deeper statistical analysis shows a weak correlation between LLM and human ratings, suggesting the potential and current limitations of LLMs in scientific synthesis evaluation.<|reference_end|>
@article{evans2024large,
    title={Large Language Models as Evaluators for Scientific Synthesis},
    author={Julia Evans, Jennifer D'Souza and S"oren Auer},
    journal={arXiv preprint arXiv:2407.02977},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.02977},
    primaryClass={cs.CL cs.AI cs.IT math.IT}
}