Is GPT-4 Alone Sufficient for Automated Essay Scoring?: A Comparative Judgment Approach Based on Rater Cognition
<|reference_start|>Is GPT-4 Alone Sufficient for Automated Essay Scoring?: A Comparative Judgment Approach Based on Rater Cognition: Large Language Models (LLMs) have shown promise in Automated Essay Scoring (AES), but their zero-shot and few-shot performance often falls short compared to state-of-the-art models and human raters. However, fine-tuning LLMs for each specific task is impractical due to the variety of essay prompts and rubrics used in real-world educational contexts. This study proposes a novel approach combining LLMs and Comparative Judgment (CJ) for AES, using zero-shot prompting to choose between two essays. We demonstrate that a CJ method surpasses traditional rubric-based scoring in essay scoring using LLMs.<|reference_end|>
@article{kim2024is,
    title={Is GPT-4 Alone Sufficient for Automated Essay Scoring?: A Comparative
  Judgment Approach Based on Rater Cognition},
    author={Seungju Kim and Meounggun Jo},
    journal={arXiv preprint arXiv:2407.05733},
    year={2024},
    doi={10.1145/3657604.3664703},
    archivePrefix={arXiv},
    eprint={2407.05733},
    primaryClass={cs.CL cs.CY}
}