Evaluating and Aligning CodeLLMs on Human Preference
Code large language models (codeLLMs) have made significant strides in code
generation. Most previous code-related benchmarks, which consist of various
programming exercises along with the corresponding test cases, are used as a
common measure to evaluate the performance and capabilities of code LLMs.
However, the current code LLMs focus on synthesizing the correct code snippet,
ignoring the alignment with human preferences, where the query should be
sampled from the practical application scenarios and the model-generated
responses should satisfy the human preference. To bridge the gap between the
model-generated response and human preference, we present a rigorous
human-curated benchmark CodeArena to emulate the complexity and diversity of
real-world coding tasks, where 397 high-quality samples spanning 40 categories
and 44 programming languages, carefully curated from user queries. Further, we
propose a diverse synthetic instruction corpus SynCode-Instruct (nearly 20B
tokens) by scaling instructions from the website to verify the effectiveness of
the large-scale synthetic instruction fine-tuning, where Qwen2.5-SynCoder
totally trained on synthetic instruction data can achieve top-tier performance
of open-source code LLMs. The results find performance differences between
execution-based benchmarks and CodeArena. Our systematic experiments of
CodeArena on 40+ LLMs reveal a notable performance gap between open SOTA code
LLMs (e.g. Qwen2.5-Coder) and proprietary LLMs (e.g., OpenAI o1), underscoring
the importance of the human preference
alignment.\url{https://codearenaeval.github.io/ }
67591839a7adc43651b0d085
Jian Yang
67591839a7adc43651b0d086
Jiaxi Yang
admin_assigned
2024-12-11T10:10:29.983Z
646df403ad20c6fa4f30b7ec
https://cdn-avatars.huggingface.co/v1/production/uploads/646df403ad20c6fa4f30b7ec/Q64-XMghOcBoo3itZDGYA.jpeg
Jiaxi Yang
user
jx-yang
67591839a7adc43651b0d087
Ke Jin
67591839a7adc43651b0d088
Yibo Miao
admin_assigned
2024-12-11T10:10:18.768Z
64a139c098fad0c8a5a627a4
https://cdn-avatars.huggingface.co/v1/production/uploads/64a139c098fad0c8a5a627a4/MgWortQS64cJOZ3pL4WyH.jpeg
Yibo Miao
user
instro
67591839a7adc43651b0d089
Lei Zhang
claimed_verified
2025-01-01T20:14:07.740Z
64c38871f9cd765462fa1a17
https://cdn-avatars.huggingface.co/v1/production/uploads/64c38871f9cd765462fa1a17/yuIlVcqeDlQVKsUF8uEl3.jpeg
Lei Zhang
user
Lemoncoke
67591839a7adc43651b0d08a
Liqun Yang
67591839a7adc43651b0d08b
Zeyu Cui
admin_assigned
2024-12-11T10:09:40.232Z
672c25ca8cfb61188128eb6f
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/FJWy9Tt7UQmu9KcTOx3Rt.png
Zeyu Cui
user
misakamage
67591839a7adc43651b0d08c
Yichang Zhang
67591839a7adc43651b0d08d
Binyuan Hui
admin_assigned
2024-12-11T10:09:25.313Z
61e4c4ca1ab24785ac11ba69
https://cdn-avatars.huggingface.co/v1/production/uploads/61e4c4ca1ab24785ac11ba69/1Q1zhhyGSJ9RJG9MzwxVv.jpeg
Binyuan Hui
user
huybery
67591839a7adc43651b0d08e
Junyang Lin
admin_assigned
2024-12-11T10:09:17.833Z
620760a26e3b7210c2ff1943
https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
Junyang Lin
user
JustinLin610
Evaluating and Aligning CodeLLMs on Human Preference
Code large language models (codeLLMs) have made significant strides in code
generation. Most previous code-related benchmarks, which consist of various
programming exercises along with the corresponding test cases, are used as a
common measure to evaluate the performance and capabilities of code LLMs.
However, the current code LLMs focus on synthesizing the correct code snippet,
ignoring the alignment with human preferences, where the query should be
sampled from the practical application scenarios and the model-generated
responses should satisfy the human preference. To bridge the gap between the
model-generated response and human preference, we present a rigorous
human-curated benchmark CodeArena to emulate the complexity and diversity of
real-world coding tasks, where 397 high-quality samples spanning 40 categories
and 44 programming languages, carefully curated from user queries. Further, we
propose a diverse synthetic instruction corpus SynCode-Instruct (nearly 20B
tokens) by scaling instructions from the website to verify the effectiveness of
the large-scale synthetic instruction fine-tuning, where Qwen2.5-SynCoder
totally trained on synthetic instruction data can achieve top-tier performance
of open-source code LLMs. The results find performance differences between
execution-based benchmarks and CodeArena. Our systematic experiments of
CodeArena on 40+ LLMs reveal a notable performance gap between open SOTA code
LLMs (e.g. Qwen2.5-Coder) and proprietary LLMs (e.g., OpenAI o1), underscoring
the importance of the human preference
alignment.\url{https://codearenaeval.github.io/ }
A human-curated benchmark (CodeArena) and a large synthetic instruction corpus (SynCode-Instruct) are introduced to evaluate code LLMs based on human preference alignment, revealing performance differences between open-source and proprietary models.
code large language models code generation benchmarks programming exercises human preferences CodeArena high-quality samples programming languages SynCode-Instruct synthetic instruction fine-tuning Qwen2.5-SynCoder execution-based benchmarks open-source code LLMs proprietary LLMs human preference alignment