title: HumanEval Pro and MBPP Pro: Evaluating Large Language Models on
    Self-invoking Code Generation
summary: We introduce self-invoking code generation, a new task designed to evaluate
  the progressive reasoning and problem-solving capabilities of LLMs. In this
  task, models are presented with a base problem and a related, more complex
  problem. They must solve the base problem and then utilize its solution to
  address the more complex one. This work features three key contributions.
  First, we propose a general recipe for generating more challenging versions of
  existing benchmarks, resulting in three new benchmarks: HumanEval Pro, MBPP
  Pro, and BigCodeBench-Lite Pro, specifically designed to assess LLMs on
  self-invoking code generation. Second, from the analysis of experimental
  results over twenty LLMs on our benchmarks, we have two important observations:
  (i) Most LLMs excel in traditional code generation benchmarks like HumanEval
  and MBPP, but their performance declines on self-invoking tasks. For example,
  o1-mini achieves 96.2% pass@1 on HumanEval but only 76.2% on HumanEval Pro.
  (ii) On self-invoking code generation task, the instruction-tuned models
  demonstrate only marginal improvements compared to the base models. Third, we
  disclose the types of failure modes that exist in our evaluation results. All
  these results underscore the need for further advancements in self-invoking
  code generation tasks and provide a new direction for future research on
  enhancing LLMs' code reasoning capabilities.
paper_authors._id: 67738e22b8df35b9c86ace86
paper_authors.name: Zhaojian Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-01T20:12:35.965Z
paper_authors._id: 646f3443c261dc413383b8a4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646f3443c261dc413383b8a4/hEJd8wLyR5HTdMzApaloN.png
paper_authors.fullname: Zhaojian Yu
paper_authors.type: user
paper_authors.user: zjy2001
paper_authors._id: 67738e22b8df35b9c86ace87
paper_authors.name: Yilun Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-22T20:49:57.088Z
paper_authors._id: 62f662bcc58915315c4eccea
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62f662bcc58915315c4eccea/zOAQLONfMP88zr70sxHK-.jpeg
paper_authors.fullname: Yilun
paper_authors.type: user
paper_authors.user: yilunzhao
paper_authors._id: 67738e22b8df35b9c86ace88
paper_authors.name: Arman Cohan
paper_authors._id: 67738e22b8df35b9c86ace89
paper_authors.name: Xiao-Ping Zhang
paper_title: HumanEval Pro and MBPP Pro: Evaluating Large Language Models on
    Self-invoking Code Generation
paper_summary: We introduce self-invoking code generation, a new task designed to evaluate
  the progressive reasoning and problem-solving capabilities of LLMs. In this
  task, models are presented with a base problem and a related, more complex
  problem. They must solve the base problem and then utilize its solution to
  address the more complex one. This work features three key contributions.
  First, we propose a general recipe for generating more challenging versions of
  existing benchmarks, resulting in three new benchmarks: HumanEval Pro, MBPP
  Pro, and BigCodeBench-Lite Pro, specifically designed to assess LLMs on
  self-invoking code generation. Second, from the analysis of experimental
  results over twenty LLMs on our benchmarks, we have two important observations:
  (i) Most LLMs excel in traditional code generation benchmarks like HumanEval
  and MBPP, but their performance declines on self-invoking tasks. For example,
  o1-mini achieves 96.2% pass@1 on HumanEval but only 76.2% on HumanEval Pro.
  (ii) On self-invoking code generation task, the instruction-tuned models
  demonstrate only marginal improvements compared to the base models. Third, we
  disclose the types of failure modes that exist in our evaluation results. All
  these results underscore the need for further advancements in self-invoking
  code generation tasks and provide a new direction for future research on
  enhancing LLMs' code reasoning capabilities.