title: CodeEditorBench: Evaluating Code Editing Capability of Large Language
    Models
summary: Large Language Models (LLMs) for code are rapidly evolving, with code editing
  emerging as a critical capability. We introduce CodeEditorBench, an evaluation
  framework designed to rigorously assess the performance of LLMs in code editing
  tasks, including debugging, translating, polishing, and requirement switching.
  Unlike existing benchmarks focusing solely on code generation, CodeEditorBench
  emphasizes real-world scenarios and practical aspects of software development.
  We curate diverse coding challenges and scenarios from five sources, covering
  various programming languages, complexity levels, and editing tasks. Evaluation
  of 19 LLMs reveals that closed-source models (particularly Gemini-Ultra and
  GPT-4), outperform open-source models in CodeEditorBench, highlighting
  differences in model performance based on problem types and prompt
  sensitivities. CodeEditorBench aims to catalyze advancements in LLMs by
  providing a robust platform for assessing code editing capabilities. We will
  release all prompts and datasets to enable the community to expand the dataset
  and benchmark emerging LLMs. By introducing CodeEditorBench, we contribute to
  the advancement of LLMs in code editing and provide a valuable resource for
  researchers and practitioners.
paper_authors._id: 660f55529bd915fe6b095cc2
paper_authors.name: Jiawei Guo
paper_authors._id: 660f55529bd915fe6b095cc3
paper_authors.name: Ziming Li
paper_authors._id: 660f55529bd915fe6b095cc4
paper_authors.name: Xueling Liu
paper_authors._id: 660f55529bd915fe6b095cc5
paper_authors.name: Kaijing Ma
paper_authors._id: 660f55529bd915fe6b095cc6
paper_authors.name: Tianyu Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-21T07:54:28.404Z
paper_authors._id: 64ab99dcb76bfd863eba64c1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ab99dcb76bfd863eba64c1/UBXwDPx17X-gl-SzBPvrc.jpeg
paper_authors.fullname: TY.Zheng
paper_authors.type: user
paper_authors.user: aaabiao
paper_authors._id: 660f55529bd915fe6b095cc7
paper_authors.name: Zhouliang Yu
paper_authors._id: 660f55529bd915fe6b095cc8
paper_authors.name: Ding Pan
paper_authors._id: 660f55529bd915fe6b095cc9
paper_authors.name: Yizhi LI
paper_authors._id: 660f55529bd915fe6b095cca
paper_authors.name: Ruibo Liu
paper_authors._id: 660f55529bd915fe6b095ccb
paper_authors.name: Yue Wang
paper_authors._id: 660f55529bd915fe6b095ccc
paper_authors.name: Shuyue Guo
paper_authors._id: 660f55529bd915fe6b095ccd
paper_authors.name: Xingwei Qu
paper_authors._id: 660f55529bd915fe6b095cce
paper_authors.name: Xiang Yue
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-27T07:21:10.097Z
paper_authors._id: 6230d750d93e84e233882dbc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
paper_authors.fullname: Xiang Yue
paper_authors.type: user
paper_authors.user: yuexiang96
paper_authors._id: 660f55529bd915fe6b095ccf
paper_authors.name: Ge Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-05T10:13:12.793Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 660f55529bd915fe6b095cd0
paper_authors.name: Wenhu Chen
paper_authors._id: 660f55529bd915fe6b095cd1
paper_authors.name: Jie Fu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-06T15:19:38.155Z
paper_authors._id: 641a6895fb5ffff5ac79d593
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/dFR_ofjbqCrcqGa9R3MMq.jpeg
paper_authors.fullname: Jie Fu
paper_authors.type: user
paper_authors.user: bigaidream
paper_title: CodeEditorBench: Evaluating Code Editing Capability of Large Language
    Models
paper_summary: Large Language Models (LLMs) for code are rapidly evolving, with code editing
  emerging as a critical capability. We introduce CodeEditorBench, an evaluation
  framework designed to rigorously assess the performance of LLMs in code editing
  tasks, including debugging, translating, polishing, and requirement switching.
  Unlike existing benchmarks focusing solely on code generation, CodeEditorBench
  emphasizes real-world scenarios and practical aspects of software development.
  We curate diverse coding challenges and scenarios from five sources, covering
  various programming languages, complexity levels, and editing tasks. Evaluation
  of 19 LLMs reveals that closed-source models (particularly Gemini-Ultra and
  GPT-4), outperform open-source models in CodeEditorBench, highlighting
  differences in model performance based on problem types and prompt
  sensitivities. CodeEditorBench aims to catalyze advancements in LLMs by
  providing a robust platform for assessing code editing capabilities. We will
  release all prompts and datasets to enable the community to expand the dataset
  and benchmark emerging LLMs. By introducing CodeEditorBench, we contribute to
  the advancement of LLMs in code editing and provide a valuable resource for
  researchers and practitioners.
paper_ai_keywords: CodeEditorBench Large Language Models (LLMs) code editing debugging translating polishing requirement switching code generation programming languages closed-source models Gemini-Ultra GPT-4 open-source models problem types prompt sensitivities software development