title: Evaluating and Aligning CodeLLMs on Human Preference
summary: Code large language models (codeLLMs) have made significant strides in code
  generation. Most previous code-related benchmarks, which consist of various
  programming exercises along with the corresponding test cases, are used as a
  common measure to evaluate the performance and capabilities of code LLMs.
  However, the current code LLMs focus on synthesizing the correct code snippet,
  ignoring the alignment with human preferences, where the query should be
  sampled from the practical application scenarios and the model-generated
  responses should satisfy the human preference. To bridge the gap between the
  model-generated response and human preference, we present a rigorous
  human-curated benchmark CodeArena to emulate the complexity and diversity of
  real-world coding tasks, where 397 high-quality samples spanning 40 categories
  and 44 programming languages, carefully curated from user queries. Further, we
  propose a diverse synthetic instruction corpus SynCode-Instruct (nearly 20B
  tokens) by scaling instructions from the website to verify the effectiveness of
  the large-scale synthetic instruction fine-tuning, where Qwen2.5-SynCoder
  totally trained on synthetic instruction data can achieve top-tier performance
  of open-source code LLMs. The results find performance differences between
  execution-based benchmarks and CodeArena. Our systematic experiments of
  CodeArena on 40+ LLMs reveal a notable performance gap between open SOTA code
  LLMs (e.g. Qwen2.5-Coder) and proprietary LLMs (e.g., OpenAI o1), underscoring
  the importance of the human preference
  alignment.\url{https://codearenaeval.github.io/ }
paper_authors._id: 67591839a7adc43651b0d085
paper_authors.name: Jian Yang
paper_authors._id: 67591839a7adc43651b0d086
paper_authors.name: Jiaxi Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T10:10:29.983Z
paper_authors._id: 646df403ad20c6fa4f30b7ec
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646df403ad20c6fa4f30b7ec/Q64-XMghOcBoo3itZDGYA.jpeg
paper_authors.fullname: Jiaxi Yang
paper_authors.type: user
paper_authors.user: jx-yang
paper_authors._id: 67591839a7adc43651b0d087
paper_authors.name: Ke Jin
paper_authors._id: 67591839a7adc43651b0d088
paper_authors.name: Yibo Miao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T10:10:18.768Z
paper_authors._id: 64a139c098fad0c8a5a627a4
paper_authors.avatarUrl: /avatars/6eb508abd827d4a4f5abb6b24155b22d.svg
paper_authors.fullname: Yibo Miao
paper_authors.type: user
paper_authors.user: instro
paper_authors._id: 67591839a7adc43651b0d089
paper_authors.name: Lei Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-01T20:14:07.740Z
paper_authors._id: 64c38871f9cd765462fa1a17
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c38871f9cd765462fa1a17/yuIlVcqeDlQVKsUF8uEl3.jpeg
paper_authors.fullname: Lei Zhang
paper_authors.type: user
paper_authors.user: Lemoncoke
paper_authors._id: 67591839a7adc43651b0d08a
paper_authors.name: Liqun Yang
paper_authors._id: 67591839a7adc43651b0d08b
paper_authors.name: Zeyu Cui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T10:09:40.232Z
paper_authors._id: 672c25ca8cfb61188128eb6f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/FJWy9Tt7UQmu9KcTOx3Rt.png
paper_authors.fullname: Zeyu Cui
paper_authors.type: user
paper_authors.user: misakamage
paper_authors._id: 67591839a7adc43651b0d08c
paper_authors.name: Yichang Zhang
paper_authors._id: 67591839a7adc43651b0d08d
paper_authors.name: Binyuan Hui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T10:09:25.313Z
paper_authors._id: 61e4c4ca1ab24785ac11ba69
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61e4c4ca1ab24785ac11ba69/1Q1zhhyGSJ9RJG9MzwxVv.jpeg
paper_authors.fullname: Binyuan Hui
paper_authors.type: user
paper_authors.user: huybery
paper_authors._id: 67591839a7adc43651b0d08e
paper_authors.name: Junyang Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T10:09:17.833Z
paper_authors._id: 620760a26e3b7210c2ff1943
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
paper_authors.fullname: Junyang Lin
paper_authors.type: user
paper_authors.user: JustinLin610
paper_title: Evaluating and Aligning CodeLLMs on Human Preference
paper_summary: Code large language models (codeLLMs) have made significant strides in code
  generation. Most previous code-related benchmarks, which consist of various
  programming exercises along with the corresponding test cases, are used as a
  common measure to evaluate the performance and capabilities of code LLMs.
  However, the current code LLMs focus on synthesizing the correct code snippet,
  ignoring the alignment with human preferences, where the query should be
  sampled from the practical application scenarios and the model-generated
  responses should satisfy the human preference. To bridge the gap between the
  model-generated response and human preference, we present a rigorous
  human-curated benchmark CodeArena to emulate the complexity and diversity of
  real-world coding tasks, where 397 high-quality samples spanning 40 categories
  and 44 programming languages, carefully curated from user queries. Further, we
  propose a diverse synthetic instruction corpus SynCode-Instruct (nearly 20B
  tokens) by scaling instructions from the website to verify the effectiveness of
  the large-scale synthetic instruction fine-tuning, where Qwen2.5-SynCoder
  totally trained on synthetic instruction data can achieve top-tier performance
  of open-source code LLMs. The results find performance differences between
  execution-based benchmarks and CodeArena. Our systematic experiments of
  CodeArena on 40+ LLMs reveal a notable performance gap between open SOTA code
  LLMs (e.g. Qwen2.5-Coder) and proprietary LLMs (e.g., OpenAI o1), underscoring
  the importance of the human preference
  alignment.\url{https://codearenaeval.github.io/ }
paper_ai_keywords: code large language models (codeLLMs) benchmarks programming exercises test cases synthesizing human preferences practical application scenarios CodeArena high-quality samples programming languages SynCode-Instruct synthetic instruction corpus fine-tuning Qwen2.5-SynCoder top-tier performance open-source code LLMs execution-based benchmarks LLMs open SOTA code LLMs proprietary LLMs Qwen2.5-Coder OpenAI o1