Evaluating the Quality of Code Comments Generated by Large Language Models for Novice Programmers
<|reference_start|>Evaluating the Quality of Code Comments Generated by Large Language Models for Novice Programmers: Large Language Models (LLMs) show promise in generating code comments for novice programmers, but their educational effectiveness remains under-evaluated. This study assesses the instructional quality of code comments produced by GPT-4, GPT-3.5-Turbo, and Llama2, compared to expert-developed comments, focusing on their suitability for novices. Analyzing a dataset of ``easy'' level Java solutions from LeetCode, we find that GPT-4 exhibits comparable quality to expert comments in aspects critical for beginners, such as clarity, beginner-friendliness, concept elucidation, and step-by-step guidance. GPT-4 outperforms Llama2 in discussing complexity (chi-square = 11.40, p = 0.001) and is perceived as significantly more supportive for beginners than GPT-3.5 and Llama2 with Mann-Whitney U-statistics = 300.5 and 322.5, p = 0.0017 and 0.0003). This study highlights the potential of LLMs for generating code comments tailored to novice programmers.<|reference_end|>
@article{fan2024evaluating,
    title={Evaluating the Quality of Code Comments Generated by Large Language
  Models for Novice Programmers},
    author={Aysa Xuemo Fan, Arun Balajiee Lekshmi Narayanan, Mohammad Hassany,
  Jiaze Ke},
    journal={arXiv preprint arXiv:2409.14368},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.14368},
    primaryClass={cs.SE cs.AI cs.HC}
}