title: CodeCriticBench: A Holistic Code Critique Benchmark for Large Language
    Models
summary: The critique capacity of Large Language Models (LLMs) is essential for
  reasoning abilities, which can provide necessary suggestions (e.g., detailed
  analysis and constructive feedback). Therefore, how to evaluate the critique
  capacity of LLMs has drawn great attention and several critique benchmarks have
  been proposed. However, existing critique benchmarks usually have the following
  limitations: (1). Focusing on diverse reasoning tasks in general domains and
  insufficient evaluation on code tasks (e.g., only covering code generation
  task), where the difficulty of queries is relatively easy (e.g., the code
  queries of CriticBench are from Humaneval and MBPP). (2). Lacking comprehensive
  evaluation from different dimensions. To address these limitations, we
  introduce a holistic code critique benchmark for LLMs called CodeCriticBench.
  Specifically, our CodeCriticBench includes two mainstream code tasks (i.e.,
  code generation and code QA) with different difficulties. Besides, the
  evaluation protocols include basic critique evaluation and advanced critique
  evaluation for different characteristics, where fine-grained evaluation
  checklists are well-designed for advanced settings. Finally, we conduct
  extensive experimental results of existing LLMs, which show the effectiveness
  of CodeCriticBench.
paper_authors._id: 67bd36334a9a04b9ca9bbb68
paper_authors.name: Alexander Zhang
paper_authors._id: 67bd36334a9a04b9ca9bbb69
paper_authors.name: Marcus Dong
paper_authors._id: 67bd36334a9a04b9ca9bbb6a
paper_authors.name: Jiaheng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-25T16:09:48.690Z
paper_authors._id: 65377c30e48353201e6fdda0
paper_authors.avatarUrl: /avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
paper_authors.fullname: Jiaheng Liu
paper_authors.type: user
paper_authors.user: CheeryLJH
paper_authors._id: 67bd36334a9a04b9ca9bbb6b
paper_authors.name: Wei Zhang
paper_authors._id: 67bd36334a9a04b9ca9bbb6c
paper_authors.name: Yejie Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-25T16:10:04.798Z
paper_authors._id: 6342dc0ee2647466b42918ab
paper_authors.avatarUrl: /avatars/a80a3df2f67410662bf9681ed8834b17.svg
paper_authors.fullname: Yejie Wang
paper_authors.type: user
paper_authors.user: banksy235
paper_authors._id: 67bd36334a9a04b9ca9bbb6d
paper_authors.name: Jian Yang
paper_authors._id: 67bd36334a9a04b9ca9bbb6e
paper_authors.name: Ge Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-25T16:19:39.556Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 67bd36334a9a04b9ca9bbb6f
paper_authors.name: Tianyu Liu
paper_authors._id: 67bd36334a9a04b9ca9bbb70
paper_authors.name: Zhongyuan Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-25T16:20:18.644Z
paper_authors._id: 63299f93688ad82b783aaf20
paper_authors.avatarUrl: /avatars/e68e6f5add62edddfbdd3795f3a72347.svg
paper_authors.fullname: zhongyuan peng
paper_authors.type: user
paper_authors.user: happzy2633
paper_authors._id: 67bd36334a9a04b9ca9bbb71
paper_authors.name: Yingshui Tan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-25T16:20:25.822Z
paper_authors._id: 6732f05d6d413742b5547249
paper_authors.avatarUrl: /avatars/c77b9fc579b0353e9c271d985b410342.svg
paper_authors.fullname: Yingshui Tan
paper_authors.type: user
paper_authors.user: YingshuiTan1996
paper_authors._id: 67bd36334a9a04b9ca9bbb72
paper_authors.name: Yuanxing Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-25T16:20:33.271Z
paper_authors._id: 64241749a05235e2f8d34cb0
paper_authors.avatarUrl: /avatars/e88967d77588f7205fbb110a51125e5b.svg
paper_authors.fullname: Yuanxing Zhang
paper_authors.type: user
paper_authors.user: LongoXC
paper_authors._id: 67bd36334a9a04b9ca9bbb73
paper_authors.name: Zhexu Wang
paper_authors._id: 67bd36334a9a04b9ca9bbb74
paper_authors.name: Weixun Wang
paper_authors._id: 67bd36334a9a04b9ca9bbb75
paper_authors.name: Yancheng He
paper_authors._id: 67bd36334a9a04b9ca9bbb76
paper_authors.name: Ken Deng
paper_authors._id: 67bd36334a9a04b9ca9bbb77
paper_authors.name: Wangchunshu Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-25T16:21:18.710Z
paper_authors._id: 628c8598ef14f971b698107f
paper_authors.avatarUrl: /avatars/3a4ad87e6b5f9e836a1160d869df1447.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: Wangchunshu
paper_authors._id: 67bd36334a9a04b9ca9bbb78
paper_authors.name: Wenhao Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-25T16:21:27.459Z
paper_authors._id: 641e5bf65f274a0a92c2f6a2
paper_authors.avatarUrl: /avatars/c15a54c51998c0e6367685e8e1737ec9.svg
paper_authors.fullname: Wenhao Huang
paper_authors.type: user
paper_authors.user: EZ-hwh
paper_authors._id: 67bd36334a9a04b9ca9bbb79
paper_authors.name: Zhaoxiang Zhang
paper_title: CodeCriticBench: A Holistic Code Critique Benchmark for Large Language
    Models
paper_summary: The critique capacity of Large Language Models (LLMs) is essential for
  reasoning abilities, which can provide necessary suggestions (e.g., detailed
  analysis and constructive feedback). Therefore, how to evaluate the critique
  capacity of LLMs has drawn great attention and several critique benchmarks have
  been proposed. However, existing critique benchmarks usually have the following
  limitations: (1). Focusing on diverse reasoning tasks in general domains and
  insufficient evaluation on code tasks (e.g., only covering code generation
  task), where the difficulty of queries is relatively easy (e.g., the code
  queries of CriticBench are from Humaneval and MBPP). (2). Lacking comprehensive
  evaluation from different dimensions. To address these limitations, we
  introduce a holistic code critique benchmark for LLMs called CodeCriticBench.
  Specifically, our CodeCriticBench includes two mainstream code tasks (i.e.,
  code generation and code QA) with different difficulties. Besides, the
  evaluation protocols include basic critique evaluation and advanced critique
  evaluation for different characteristics, where fine-grained evaluation
  checklists are well-designed for advanced settings. Finally, we conduct
  extensive experimental results of existing LLMs, which show the effectiveness
  of CodeCriticBench.
paper_ai_keywords: Large Language Models (LLMs) critique capacity reasoning abilities constructive feedback critique benchmarks code tasks code generation code QA (Quality Assurance) CriticBench Humaneval MBPP basic critique evaluation advanced critique evaluation fine-grained evaluation checklists