CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding?
<|reference_start|>CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding?: Recent advancements in large language models (LLMs) have showcased impressive code generation capabilities, primarily evaluated through language-to-code benchmarks. However, these benchmarks may not fully capture a model's code understanding abilities. We introduce CodeJudge-Eval (CJ-Eval), a novel benchmark designed to assess LLMs' code understanding abilities from the perspective of code judging rather than code generation. CJ-Eval challenges models to determine the correctness of provided code solutions, encompassing various error types and compilation issues. By leveraging a diverse set of problems and a fine-grained judging system, CJ-Eval addresses the limitations of traditional benchmarks, including the potential memorization of solutions. Evaluation of 12 well-known LLMs on CJ-Eval reveals that even state-of-the-art models struggle, highlighting the benchmark's ability to probe deeper into models' code understanding abilities. Our codes and benchmark are available at \url{https://github.com/CodeLLM-Research/CodeJudge-Eval}.<|reference_end|>
@article{zhao2024codejudge-eval:,
    title={CodeJudge-Eval: Can Large Language Models be Good Judges in Code
  Understanding?},
    author={Yuwei Zhao, Ziyang Luo, Yuchen Tian, Hongzhan Lin, Weixiang Yan, Annan
  Li, Jing Ma},
    journal={arXiv preprint arXiv:2408.10718},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.10718},
    primaryClass={cs.SE cs.CL}
}