How Do Your Code LLMs Perform? Empowering Code Instruction Tuning with
  High-Quality Data
Recently, there has been a growing interest in studying how to construct
better code instruction tuning data. However, we observe Code models trained
with these datasets exhibit high performance on HumanEval but perform worse on
other benchmarks such as LiveCodeBench. Upon further investigation, we find
that many datasets suffer from severe data leakage. After cleaning up most of
the leaked data, some well-known high-quality datasets perform poorly. This
discovery reveals a new challenge: identifying which dataset genuinely qualify
as high-quality code instruction data. To address this, we propose an efficient
code data pruning strategy for selecting good samples. Our approach is based on
three dimensions: instruction complexity, response quality, and instruction
diversity. Based on our selected data, we present XCoder, a family of models
finetuned from LLaMA3. Our experiments show XCoder achieves new
state-of-the-art performance using fewer training data, which verify the
effectiveness of our data strategy. Moreover, we perform a comprehensive
analysis on the data composition and find existing code datasets have different
characteristics according to their construction methods, which provide new
insights for future code LLMs. Our models and dataset are released in
https://github.com/banksy23/XCoder
66de6a5713e3a3853608938e
Yejie Wang
claimed_verified
2024-09-09T10:58:02.998Z
6342dc0ee2647466b42918ab
/avatars/a80a3df2f67410662bf9681ed8834b17.svg
Yejie Wang
user
banksy235
66de6a5713e3a3853608938f
Keqing He
claimed_verified
2024-09-10T12:39:53.674Z
64bf71792915a87970c07446
/avatars/b24403f9fa699e0143e441b56528e6af.svg
Keqing He
user
HelicHe
66de6a5713e3a38536089390
Dayuan Fu
admin_assigned
2024-09-09T07:31:16.055Z
65572fd2bcdc315c0a1395d7
/avatars/36135ca35bbd4baca37a9092f3e2d82c.svg
Fu-Dayuan
user
fudayuan
66de6a5713e3a38536089391
Zhuoma Gongque
66de6a5713e3a38536089392
Heyang Xu
66de6a5713e3a38536089393
Yanxu Chen
66de6a5713e3a38536089394
Zhexu Wang
claimed_verified
2025-07-11T13:13:29.065Z
666aab1be0ecb2edac62e914
/avatars/8eda456501bff02c4f625a568c7f2350.svg
yugongwzx
user
yugongwzx
66de6a5713e3a38536089395
Yujia Fu
66de6a5713e3a38536089396
Guanting Dong
admin_assigned
2024-09-09T07:00:28.698Z
61cd4b833dd34ba1985e0753
https://cdn-avatars.huggingface.co/v1/production/uploads/61cd4b833dd34ba1985e0753/BfHfrwotoMESpXZOHiIe4.png
KABI
user
dongguanting
66de6a5713e3a38536089397
Muxi Diao
claimed_verified
2024-12-30T19:38:53.722Z
6768c97367e4b4606a3c9cec
/avatars/5ddafe7a05828366f66c79072556f370.svg
diaomuxi
user
diaomuxi
66de6a5713e3a38536089398
Jingang Wang
claimed_verified
2024-09-09T07:53:06.442Z
647097cbcfd57849518e656b
/avatars/c66fe0add29c1bde9e3a98bf4a8793b9.svg
Jingang Wang
user
bitwjg
66de6a5713e3a38536089399
Mengdi Zhang
admin_assigned
2024-09-09T07:29:23.706Z
654d8d98d61e57476c2b6a18
/avatars/f4a6be7f1642060542e5ec9b693313ed.svg
Mengdi Zhang
user
mdizhang
66de6a5713e3a3853608939a
Xunliang Cai
66de6a5713e3a3853608939b
Weiran Xu
How Do Your Code LLMs Perform? Empowering Code Instruction Tuning with
  High-Quality Data
Recently, there has been a growing interest in studying how to construct
better code instruction tuning data. However, we observe Code models trained
with these datasets exhibit high performance on HumanEval but perform worse on
other benchmarks such as LiveCodeBench. Upon further investigation, we find
that many datasets suffer from severe data leakage. After cleaning up most of
the leaked data, some well-known high-quality datasets perform poorly. This
discovery reveals a new challenge: identifying which dataset genuinely qualify
as high-quality code instruction data. To address this, we propose an efficient
code data pruning strategy for selecting good samples. Our approach is based on
three dimensions: instruction complexity, response quality, and instruction
diversity. Based on our selected data, we present XCoder, a family of models
finetuned from LLaMA3. Our experiments show XCoder achieves new
state-of-the-art performance using fewer training data, which verify the
effectiveness of our data strategy. Moreover, we perform a comprehensive
analysis on the data composition and find existing code datasets have different
characteristics according to their construction methods, which provide new
insights for future code LLMs. Our models and dataset are released in
https://github.com/banksy23/XCoder
XCoder, a family of models fine-tuned from LLaMA3, achieves state-of-the-art performance on code instruction tasks using a novel data pruning strategy that addresses data leakage and quality in training datasets.
code instruction tuning HumanEval LiveCodeBench data leakage code data pruning instruction complexity response quality instruction diversity XCoder LLaMA3