title: How Do Your Code LLMs Perform? Empowering Code Instruction Tuning with
    High-Quality Data
summary: Recently, there has been a growing interest in studying how to construct
  better code instruction tuning data. However, we observe Code models trained
  with these datasets exhibit high performance on HumanEval but perform worse on
  other benchmarks such as LiveCodeBench. Upon further investigation, we find
  that many datasets suffer from severe data leakage. After cleaning up most of
  the leaked data, some well-known high-quality datasets perform poorly. This
  discovery reveals a new challenge: identifying which dataset genuinely qualify
  as high-quality code instruction data. To address this, we propose an efficient
  code data pruning strategy for selecting good samples. Our approach is based on
  three dimensions: instruction complexity, response quality, and instruction
  diversity. Based on our selected data, we present XCoder, a family of models
  finetuned from LLaMA3. Our experiments show XCoder achieves new
  state-of-the-art performance using fewer training data, which verify the
  effectiveness of our data strategy. Moreover, we perform a comprehensive
  analysis on the data composition and find existing code datasets have different
  characteristics according to their construction methods, which provide new
  insights for future code LLMs. Our models and dataset are released in
  https://github.com/banksy23/XCoder
paper_authors._id: 66de6a5713e3a3853608938e
paper_authors.name: Yejie Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-09T10:58:02.998Z
paper_authors._id: 6342dc0ee2647466b42918ab
paper_authors.avatarUrl: /avatars/a80a3df2f67410662bf9681ed8834b17.svg
paper_authors.fullname: Yejie Wang
paper_authors.type: user
paper_authors.user: banksy235
paper_authors._id: 66de6a5713e3a3853608938f
paper_authors.name: Keqing He
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-10T12:39:53.674Z
paper_authors._id: 64bf71792915a87970c07446
paper_authors.avatarUrl: /avatars/b24403f9fa699e0143e441b56528e6af.svg
paper_authors.fullname: Keqing He
paper_authors.type: user
paper_authors.user: HelicHe
paper_authors._id: 66de6a5713e3a38536089390
paper_authors.name: Dayuan Fu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-09T07:31:16.055Z
paper_authors._id: 65572fd2bcdc315c0a1395d7
paper_authors.avatarUrl: /avatars/36135ca35bbd4baca37a9092f3e2d82c.svg
paper_authors.fullname: Fu-Dayuan
paper_authors.type: user
paper_authors.user: fudayuan
paper_authors._id: 66de6a5713e3a38536089391
paper_authors.name: Zhuoma Gongque
paper_authors._id: 66de6a5713e3a38536089392
paper_authors.name: Heyang Xu
paper_authors._id: 66de6a5713e3a38536089393
paper_authors.name: Yanxu Chen
paper_authors._id: 66de6a5713e3a38536089394
paper_authors.name: Zhexu Wang
paper_authors._id: 66de6a5713e3a38536089395
paper_authors.name: Yujia Fu
paper_authors._id: 66de6a5713e3a38536089396
paper_authors.name: Guanting Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-09T07:00:28.698Z
paper_authors._id: 61cd4b833dd34ba1985e0753
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61cd4b833dd34ba1985e0753/BfHfrwotoMESpXZOHiIe4.png
paper_authors.fullname: KABI
paper_authors.type: user
paper_authors.user: dongguanting
paper_authors._id: 66de6a5713e3a38536089397
paper_authors.name: Muxi Diao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:38:53.722Z
paper_authors._id: 6768c97367e4b4606a3c9cec
paper_authors.avatarUrl: /avatars/5ddafe7a05828366f66c79072556f370.svg
paper_authors.fullname: diaomuxi
paper_authors.type: user
paper_authors.user: diaomuxi
paper_authors._id: 66de6a5713e3a38536089398
paper_authors.name: Jingang Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-09T07:53:06.442Z
paper_authors._id: 647097cbcfd57849518e656b
paper_authors.avatarUrl: /avatars/c66fe0add29c1bde9e3a98bf4a8793b9.svg
paper_authors.fullname: Jingang Wang
paper_authors.type: user
paper_authors.user: bitwjg
paper_authors._id: 66de6a5713e3a38536089399
paper_authors.name: Mengdi Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-09T07:29:23.706Z
paper_authors._id: 654d8d98d61e57476c2b6a18
paper_authors.avatarUrl: /avatars/f4a6be7f1642060542e5ec9b693313ed.svg
paper_authors.fullname: Mengdi Zhang
paper_authors.type: user
paper_authors.user: mdizhang
paper_authors._id: 66de6a5713e3a3853608939a
paper_authors.name: Xunliang Cai
paper_authors._id: 66de6a5713e3a3853608939b
paper_authors.name: Weiran Xu
paper_title: How Do Your Code LLMs Perform? Empowering Code Instruction Tuning with
    High-Quality Data
paper_summary: Recently, there has been a growing interest in studying how to construct
  better code instruction tuning data. However, we observe Code models trained
  with these datasets exhibit high performance on HumanEval but perform worse on
  other benchmarks such as LiveCodeBench. Upon further investigation, we find
  that many datasets suffer from severe data leakage. After cleaning up most of
  the leaked data, some well-known high-quality datasets perform poorly. This
  discovery reveals a new challenge: identifying which dataset genuinely qualify
  as high-quality code instruction data. To address this, we propose an efficient
  code data pruning strategy for selecting good samples. Our approach is based on
  three dimensions: instruction complexity, response quality, and instruction
  diversity. Based on our selected data, we present XCoder, a family of models
  finetuned from LLaMA3. Our experiments show XCoder achieves new
  state-of-the-art performance using fewer training data, which verify the
  effectiveness of our data strategy. Moreover, we perform a comprehensive
  analysis on the data composition and find existing code datasets have different
  characteristics according to their construction methods, which provide new
  insights for future code LLMs. Our models and dataset are released in
  https://github.com/banksy23/XCoder
paper_ai_keywords: code instruction tuning HumanEval LiveCodeBench data leakage code data classification strategy instruction complexity response quality instruction diversity XCoder LLaMA3 parameter-efficient fine-tuning