title: WaveCoder: Widespread And Versatile Enhanced Instruction Tuning with
    Refined Data Generation
summary: Recent work demonstrates that, after being fine-tuned on a high-quality
  instruction dataset, the resulting model can obtain impressive capabilities to
  address a wide range of tasks. However, existing methods for instruction data
  generation often produce duplicate data and are not controllable enough on data
  quality. In this paper, we extend the generalization of instruction tuning by
  classifying the instruction data to 4 code-related tasks and propose a
  LLM-based Generator-Discriminator data process framework to generate diverse,
  high-quality instruction data from open source code. Hence, we introduce
  CodeOcean, a dataset comprising 20,000 instruction instances across 4 universal
  code-related tasks,which is aimed at augmenting the effectiveness of
  instruction tuning and improving the generalization ability of fine-tuned
  model. Subsequently, we present WaveCoder, a fine-tuned Code LLM with
  Widespread And Versatile Enhanced instruction tuning. This model is
  specifically designed for enhancing instruction tuning of Code Language Models
  (LLMs). Our experiments demonstrate that Wavecoder models outperform other
  open-source models in terms of generalization ability across different
  code-related tasks at the same level of fine-tuning scale. Moreover, Wavecoder
  exhibits high efficiency in previous code generation tasks. This paper thus
  offers a significant contribution to the field of instruction data generation
  and fine-tuning models, providing new insights and tools for enhancing
  performance in code-related tasks.
paper_authors._id: 658a4687eac1c5dac0efee86
paper_authors.name: Zhaojian Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-26T10:37:01.996Z
paper_authors._id: 646f3443c261dc413383b8a4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646f3443c261dc413383b8a4/hEJd8wLyR5HTdMzApaloN.png
paper_authors.fullname: Zhaojian Yu
paper_authors.type: user
paper_authors.user: zjy2001
paper_authors._id: 658a4687eac1c5dac0efee87
paper_authors.name: Xin Zhang
paper_authors._id: 658a4687eac1c5dac0efee88
paper_authors.name: Ning Shang
paper_authors._id: 658a4687eac1c5dac0efee89
paper_authors.name: Yangyu Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-13T13:03:19.388Z
paper_authors._id: 64c66647725ffa04b2fd6c94
paper_authors.avatarUrl: /avatars/620f63f27fa1e90423b0dc22aa8e5809.svg
paper_authors.fullname: yangyu huang
paper_authors.type: user
paper_authors.user: yangyu90
paper_authors._id: 658a4687eac1c5dac0efee8a
paper_authors.name: Can Xu
paper_authors._id: 658a4687eac1c5dac0efee8b
paper_authors.name: Yishujie Zhao
paper_authors._id: 658a4687eac1c5dac0efee8c
paper_authors.name: Wenxiang Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-03T12:36:18.126Z
paper_authors._id: 6327cafc46ffcfc27cfa4c9a
paper_authors.avatarUrl: /avatars/4daea27b432ff428c41d590669f7330c.svg
paper_authors.fullname: Wenxiang Hu
paper_authors.type: user
paper_authors.user: wenxcs
paper_authors._id: 658a4687eac1c5dac0efee8d
paper_authors.name: Qiufeng Yin
paper_title: WaveCoder: Widespread And Versatile Enhanced Instruction Tuning with
    Refined Data Generation
paper_summary: Recent work demonstrates that, after being fine-tuned on a high-quality
  instruction dataset, the resulting model can obtain impressive capabilities to
  address a wide range of tasks. However, existing methods for instruction data
  generation often produce duplicate data and are not controllable enough on data
  quality. In this paper, we extend the generalization of instruction tuning by
  classifying the instruction data to 4 code-related tasks and propose a
  LLM-based Generator-Discriminator data process framework to generate diverse,
  high-quality instruction data from open source code. Hence, we introduce
  CodeOcean, a dataset comprising 20,000 instruction instances across 4 universal
  code-related tasks,which is aimed at augmenting the effectiveness of
  instruction tuning and improving the generalization ability of fine-tuned
  model. Subsequently, we present WaveCoder, a fine-tuned Code LLM with
  Widespread And Versatile Enhanced instruction tuning. This model is
  specifically designed for enhancing instruction tuning of Code Language Models
  (LLMs). Our experiments demonstrate that Wavecoder models outperform other
  open-source models in terms of generalization ability across different
  code-related tasks at the same level of fine-tuning scale. Moreover, Wavecoder
  exhibits high efficiency in previous code generation tasks. This paper thus
  offers a significant contribution to the field of instruction data generation
  and fine-tuning models, providing new insights and tools for enhancing
  performance in code-related tasks.
paper_ai_keywords: LLM-based-Generator-Discriminator CodeOcean fine-tuned Code LLM WaveCoder instruction tuning generalization ability