title: MFTCoder: Boosting Code LLMs with Multitask Fine-Tuning
summary: Code LLMs have emerged as a specialized research field, with remarkable
  studies dedicated to enhancing model's coding capabilities through fine-tuning
  on pre-trained models. Previous fine-tuning approaches were typically tailored
  to specific downstream tasks or scenarios, which meant separate fine-tuning for
  each task, requiring extensive training resources and posing challenges in
  terms of deployment and maintenance. Furthermore, these approaches failed to
  leverage the inherent interconnectedness among different code-related tasks. To
  overcome these limitations, we present a multi-task fine-tuning framework,
  MFTcoder, that enables simultaneous and parallel fine-tuning on multiple tasks.
  By incorporating various loss functions, we effectively address common
  challenges in multi-task learning, such as data imbalance, varying difficulty
  levels, and inconsistent convergence speeds. Extensive experiments have
  conclusively demonstrated that our multi-task fine-tuning approach outperforms
  both individual fine-tuning on single tasks and fine-tuning on a mixed ensemble
  of tasks. Moreover, MFTcoder offers efficient training capabilities, including
  efficient data tokenization modes and PEFT fine-tuning, resulting in
  significantly improved speed compared to traditional fine-tuning methods.
  MFTcoder seamlessly integrates with several mainstream open-source LLMs, such
  as CodeLLama and Qwen. Leveraging the CodeLLama foundation, our MFTcoder
  fine-tuned model, CodeFuse-CodeLLama-34B, achieves an impressive
  pass@1 score of 74.4\% on the HumaneEval benchmark, surpassing GPT-4
  performance (67\%, zero-shot). MFTCoder is open-sourced at
  https://github.com/codefuse-ai/MFTCOder
paper_authors._id: 6549aa2a899653d1d13299f4
paper_authors.name: Bingchang Liu
paper_authors._id: 6549aa2a899653d1d13299f5
paper_authors.name: Chaoyu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T15:33:38.184Z
paper_authors._id: 63f4860dcfd7ba6e26f3c694
paper_authors.avatarUrl: /avatars/de1ac926d4395688e1da75568ac50c76.svg
paper_authors.fullname: Chaoyu Chen
paper_authors.type: user
paper_authors.user: chencyudel
paper_authors._id: 6549aa2a899653d1d13299f6
paper_authors.name: Cong Liao
paper_authors._id: 6549aa2a899653d1d13299f7
paper_authors.name: Zi Gong
paper_authors._id: 6549aa2a899653d1d13299f8
paper_authors.name: Huan Wang
paper_authors._id: 6549aa2a899653d1d13299f9
paper_authors.name: Zhichao Lei
paper_authors._id: 6549aa2a899653d1d13299fa
paper_authors.name: Ming Liang
paper_authors._id: 6549aa2a899653d1d13299fb
paper_authors.name: Dajun Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T15:37:51.451Z
paper_authors._id: 61d41037443d3eaec85f1d4c
paper_authors.avatarUrl: /avatars/9ab12d5c485bd2e3b2f351f0419d2164.svg
paper_authors.fullname: ChenDajun
paper_authors.type: user
paper_authors.user: cdj0311
paper_authors._id: 6549aa2a899653d1d13299fc
paper_authors.name: Min Shen
paper_authors._id: 6549aa2a899653d1d13299fd
paper_authors.name: Hailian Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T15:37:27.292Z
paper_authors._id: 6475ea4ef60449361e569677
paper_authors.avatarUrl: /avatars/ff00c019480dabccf86818e441426dd1.svg
paper_authors.fullname: hailian zhou
paper_authors.type: user
paper_authors.user: hailian
paper_authors._id: 6549aa2a899653d1d13299fe
paper_authors.name: Hang Yu
paper_authors._id: 6549aa2a899653d1d13299ff
paper_authors.name: Jianguo Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T15:36:06.534Z
paper_authors._id: 6539fc3fe279d2fc80c88902
paper_authors.avatarUrl: /avatars/432ac2287a7f7d04c2c7e505070a3331.svg
paper_authors.fullname: Jianguo Li
paper_authors.type: user
paper_authors.user: jglee2046
paper_title: MFTCoder: Boosting Code LLMs with Multitask Fine-Tuning
paper_summary: Code LLMs have emerged as a specialized research field, with remarkable
  studies dedicated to enhancing model's coding capabilities through fine-tuning
  on pre-trained models. Previous fine-tuning approaches were typically tailored
  to specific downstream tasks or scenarios, which meant separate fine-tuning for
  each task, requiring extensive training resources and posing challenges in
  terms of deployment and maintenance. Furthermore, these approaches failed to
  leverage the inherent interconnectedness among different code-related tasks. To
  overcome these limitations, we present a multi-task fine-tuning framework,
  MFTcoder, that enables simultaneous and parallel fine-tuning on multiple tasks.
  By incorporating various loss functions, we effectively address common
  challenges in multi-task learning, such as data imbalance, varying difficulty
  levels, and inconsistent convergence speeds. Extensive experiments have
  conclusively demonstrated that our multi-task fine-tuning approach outperforms
  both individual fine-tuning on single tasks and fine-tuning on a mixed ensemble
  of tasks. Moreover, MFTcoder offers efficient training capabilities, including
  efficient data tokenization modes and PEFT fine-tuning, resulting in
  significantly improved speed compared to traditional fine-tuning methods.
  MFTcoder seamlessly integrates with several mainstream open-source LLMs, such
  as CodeLLama and Qwen. Leveraging the CodeLLama foundation, our MFTcoder
  fine-tuned model, CodeFuse-CodeLLama-34B, achieves an impressive
  pass@1 score of 74.4\% on the HumaneEval benchmark, surpassing GPT-4
  performance (67\%, zero-shot). MFTCoder is open-sourced at
  https://github.com/codefuse-ai/MFTCOder
paper_ai_keywords: multi-task fine-tuning fine-tuning data imbalance varying difficulty levels inconsistent convergence speeds PEFT fine-tuning data tokenization CodeLLama Qwen CodeFuse-CodeLLama-34B pass@1 score HumaneEval benchmark GPT-4