CodeCriticBench: A Holistic Code Critique Benchmark for Large Language
  Models
The critique capacity of Large Language Models (LLMs) is essential for
reasoning abilities, which can provide necessary suggestions (e.g., detailed
analysis and constructive feedback). Therefore, how to evaluate the critique
capacity of LLMs has drawn great attention and several critique benchmarks have
been proposed. However, existing critique benchmarks usually have the following
limitations: (1). Focusing on diverse reasoning tasks in general domains and
insufficient evaluation on code tasks (e.g., only covering code generation
task), where the difficulty of queries is relatively easy (e.g., the code
queries of CriticBench are from Humaneval and MBPP). (2). Lacking comprehensive
evaluation from different dimensions. To address these limitations, we
introduce a holistic code critique benchmark for LLMs called CodeCriticBench.
Specifically, our CodeCriticBench includes two mainstream code tasks (i.e.,
code generation and code QA) with different difficulties. Besides, the
evaluation protocols include basic critique evaluation and advanced critique
evaluation for different characteristics, where fine-grained evaluation
checklists are well-designed for advanced settings. Finally, we conduct
extensive experimental results of existing LLMs, which show the effectiveness
of CodeCriticBench.
67bd36334a9a04b9ca9bbb68
Alexander Zhang
67bd36334a9a04b9ca9bbb69
Marcus Dong
67bd36334a9a04b9ca9bbb6a
Jiaheng Liu
admin_assigned
2025-02-25T16:09:48.690Z
65377c30e48353201e6fdda0
/avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
Jiaheng Liu
user
CheeryLJH
67bd36334a9a04b9ca9bbb6b
Wei Zhang
67bd36334a9a04b9ca9bbb6c
Yejie Wang
admin_assigned
2025-02-25T16:10:04.798Z
6342dc0ee2647466b42918ab
/avatars/a80a3df2f67410662bf9681ed8834b17.svg
Yejie Wang
user
banksy235
67bd36334a9a04b9ca9bbb6d
Jian Yang
67bd36334a9a04b9ca9bbb6e
Ge Zhang
admin_assigned
2025-02-25T16:19:39.556Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
67bd36334a9a04b9ca9bbb6f
Tianyu Liu
67bd36334a9a04b9ca9bbb70
Zhongyuan Peng
admin_assigned
2025-02-25T16:20:18.644Z
63299f93688ad82b783aaf20
/avatars/7c11e60e551ef1c62aa2862529e357f5.svg
zhongyuan peng
user
happzy2633
67bd36334a9a04b9ca9bbb71
Yingshui Tan
admin_assigned
2025-02-25T16:20:25.822Z
6732f05d6d413742b5547249
/avatars/c77b9fc579b0353e9c271d985b410342.svg
Yingshui Tan
user
YingshuiTan1996
67bd36334a9a04b9ca9bbb72
Yuanxing Zhang
admin_assigned
2025-02-25T16:20:33.271Z
64241749a05235e2f8d34cb0
/avatars/e88967d77588f7205fbb110a51125e5b.svg
Yuanxing Zhang
user
LongoXC
67bd36334a9a04b9ca9bbb73
Zhexu Wang
claimed_verified
2025-07-11T13:13:23.837Z
666aab1be0ecb2edac62e914
/avatars/8eda456501bff02c4f625a568c7f2350.svg
yugongwzx
user
yugongwzx
67bd36334a9a04b9ca9bbb74
Weixun Wang
67bd36334a9a04b9ca9bbb75
Yancheng He
67bd36334a9a04b9ca9bbb76
Ken Deng
67bd36334a9a04b9ca9bbb77
Wangchunshu Zhou
admin_assigned
2025-02-25T16:21:18.710Z
628c8598ef14f971b698107f
/avatars/3a4ad87e6b5f9e836a1160d869df1447.svg
Zhou
user
Wangchunshu
67bd36334a9a04b9ca9bbb78
Wenhao Huang
admin_assigned
2025-02-25T16:21:27.459Z
641e5bf65f274a0a92c2f6a2
/avatars/c15a54c51998c0e6367685e8e1737ec9.svg
Wenhao Huang
user
EZ-hwh
67bd36334a9a04b9ca9bbb79
Zhaoxiang Zhang
CodeCriticBench: A Holistic Code Critique Benchmark for Large Language
  Models
The critique capacity of Large Language Models (LLMs) is essential for
reasoning abilities, which can provide necessary suggestions (e.g., detailed
analysis and constructive feedback). Therefore, how to evaluate the critique
capacity of LLMs has drawn great attention and several critique benchmarks have
been proposed. However, existing critique benchmarks usually have the following
limitations: (1). Focusing on diverse reasoning tasks in general domains and
insufficient evaluation on code tasks (e.g., only covering code generation
task), where the difficulty of queries is relatively easy (e.g., the code
queries of CriticBench are from Humaneval and MBPP). (2). Lacking comprehensive
evaluation from different dimensions. To address these limitations, we
introduce a holistic code critique benchmark for LLMs called CodeCriticBench.
Specifically, our CodeCriticBench includes two mainstream code tasks (i.e.,
code generation and code QA) with different difficulties. Besides, the
evaluation protocols include basic critique evaluation and advanced critique
evaluation for different characteristics, where fine-grained evaluation
checklists are well-designed for advanced settings. Finally, we conduct
extensive experimental results of existing LLMs, which show the effectiveness
of CodeCriticBench.
A new benchmark, CodeCriticBench, is introduced to evaluate code critique capacity of LLMs by addressing limitations in existing benchmarks, such as narrow focus and insufficient task diversity.
Large Language Models (LLMs) critique capacity reasoning abilities critique benchmarks code tasks code generation code QA evaluation protocols basic critique evaluation advanced critique evaluation fine-grained evaluation checklists