title: InverseCoder: Unleashing the Power of Instruction-Tuned Code LLMs with
    Inverse-Instruct
summary: Recent advancements in open-source code large language models (LLMs) have
  demonstrated remarkable coding abilities by fine-tuning on the data generated
  from powerful closed-source LLMs such as GPT-3.5 and GPT-4 for instruction
  tuning. This paper explores how to further improve an instruction-tuned code
  LLM by generating data from itself rather than querying closed-source LLMs. Our
  key observation is the misalignment between the translation of formal and
  informal languages: translating formal language (i.e., code) to informal
  language (i.e., natural language) is more straightforward than the reverse.
  Based on this observation, we propose INVERSE-INSTRUCT, which summarizes
  instructions from code snippets instead of the reverse. Specifically, given an
  instruction tuning corpus for code and the resulting instruction-tuned code
  LLM, we ask the code LLM to generate additional high-quality instructions for
  the original corpus through code summarization and self-evaluation. Then, we
  fine-tune the base LLM on the combination of the original corpus and the
  self-generated one, which yields a stronger instruction-tuned LLM. We present a
  series of code LLMs named InverseCoder, which surpasses the performance of the
  original code LLMs on a wide range of benchmarks, including Python text-to-code
  generation, multilingual coding, and data-science code generation.
paper_authors._id: 668c9c719f4aead59cf00fef
paper_authors.name: Yutong Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-09T07:42:06.880Z
paper_authors._id: 64e42be971139393aa3dd90a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64e42be971139393aa3dd90a/N7CwgOiU6rx5FTIfLO1Yo.png
paper_authors.fullname: Yutong Wu
paper_authors.type: user
paper_authors.user: wyt2000
paper_authors._id: 668c9c719f4aead59cf00ff0
paper_authors.name: Di Huang
paper_authors._id: 668c9c719f4aead59cf00ff1
paper_authors.name: Wenxuan Shi
paper_authors._id: 668c9c719f4aead59cf00ff2
paper_authors.name: Wei Wang
paper_authors._id: 668c9c719f4aead59cf00ff3
paper_authors.name: Lingzhe Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-09T08:39:58.699Z
paper_authors._id: 64bbe24a8e051085bada61e8
paper_authors.avatarUrl: /avatars/ff9131485c3797671a38ed45af9de807.svg
paper_authors.fullname: Lingzhe Gao
paper_authors.type: user
paper_authors.user: youlitaiglz
paper_authors._id: 668c9c719f4aead59cf00ff4
paper_authors.name: Shihao Liu
paper_authors._id: 668c9c719f4aead59cf00ff5
paper_authors.name: Ziyuan Nan
paper_authors._id: 668c9c719f4aead59cf00ff6
paper_authors.name: Kaizhao Yuan
paper_authors._id: 668c9c719f4aead59cf00ff7
paper_authors.name: Rui Zhang
paper_authors._id: 668c9c719f4aead59cf00ff8
paper_authors.name: Xishan Zhang
paper_authors._id: 668c9c719f4aead59cf00ff9
paper_authors.name: Zidong Du
paper_authors._id: 668c9c719f4aead59cf00ffa
paper_authors.name: Qi Guo
paper_authors._id: 668c9c719f4aead59cf00ffb
paper_authors.name: Yewen Pu
paper_authors._id: 668c9c719f4aead59cf00ffc
paper_authors.name: Dawei Yin
paper_authors._id: 668c9c719f4aead59cf00ffd
paper_authors.name: Xing Hu
paper_authors._id: 668c9c719f4aead59cf00ffe
paper_authors.name: Yunji Chen
paper_title: InverseCoder: Unleashing the Power of Instruction-Tuned Code LLMs with
    Inverse-Instruct
paper_summary: Recent advancements in open-source code large language models (LLMs) have
  demonstrated remarkable coding abilities by fine-tuning on the data generated
  from powerful closed-source LLMs such as GPT-3.5 and GPT-4 for instruction
  tuning. This paper explores how to further improve an instruction-tuned code
  LLM by generating data from itself rather than querying closed-source LLMs. Our
  key observation is the misalignment between the translation of formal and
  informal languages: translating formal language (i.e., code) to informal
  language (i.e., natural language) is more straightforward than the reverse.
  Based on this observation, we propose INVERSE-INSTRUCT, which summarizes
  instructions from code snippets instead of the reverse. Specifically, given an
  instruction tuning corpus for code and the resulting instruction-tuned code
  LLM, we ask the code LLM to generate additional high-quality instructions for
  the original corpus through code summarization and self-evaluation. Then, we
  fine-tune the base LLM on the combination of the original corpus and the
  self-generated one, which yields a stronger instruction-tuned LLM. We present a
  series of code LLMs named InverseCoder, which surpasses the performance of the
  original code LLMs on a wide range of benchmarks, including Python text-to-code
  generation, multilingual coding, and data-science code generation.
paper_ai_keywords: open-source code large language models (LLMs) fine-tuning instruction-tuning code LLM closed-source LLMs GPT-3.5 GPT-4 INVERSE-INSTRUCT code summarization self-evaluation summary instructions multilingual coding data-science code generation