MFTCoder: Boosting Code LLMs with Multitask Fine-Tuning
Code LLMs have emerged as a specialized research field, with remarkable
studies dedicated to enhancing model's coding capabilities through fine-tuning
on pre-trained models. Previous fine-tuning approaches were typically tailored
to specific downstream tasks or scenarios, which meant separate fine-tuning for
each task, requiring extensive training resources and posing challenges in
terms of deployment and maintenance. Furthermore, these approaches failed to
leverage the inherent interconnectedness among different code-related tasks. To
overcome these limitations, we present a multi-task fine-tuning framework,
MFTcoder, that enables simultaneous and parallel fine-tuning on multiple tasks.
By incorporating various loss functions, we effectively address common
challenges in multi-task learning, such as data imbalance, varying difficulty
levels, and inconsistent convergence speeds. Extensive experiments have
conclusively demonstrated that our multi-task fine-tuning approach outperforms
both individual fine-tuning on single tasks and fine-tuning on a mixed ensemble
of tasks. Moreover, MFTcoder offers efficient training capabilities, including
efficient data tokenization modes and PEFT fine-tuning, resulting in
significantly improved speed compared to traditional fine-tuning methods.
MFTcoder seamlessly integrates with several mainstream open-source LLMs, such
as CodeLLama and Qwen. Leveraging the CodeLLama foundation, our MFTcoder
fine-tuned model, CodeFuse-CodeLLama-34B, achieves an impressive
pass@1 score of 74.4\% on the HumaneEval benchmark, surpassing GPT-4
performance (67\%, zero-shot). MFTCoder is open-sourced at
https://github.com/codefuse-ai/MFTCOder
6549aa2a899653d1d13299f4
Bingchang Liu
claimed_verified
2025-04-08T08:47:06.077Z
640ec9fc8512ec51d7f64409
https://cdn-avatars.huggingface.co/v1/production/uploads/640ec9fc8512ec51d7f64409/LSq6prKfiWCsvAPcBI4Rm.jpeg
L BC
user
twelveand0
6549aa2a899653d1d13299f5
Chaoyu Chen
admin_assigned
2023-11-07T15:33:38.184Z
63f4860dcfd7ba6e26f3c694
/avatars/de1ac926d4395688e1da75568ac50c76.svg
Chaoyu Chen
user
chencyudel
6549aa2a899653d1d13299f6
Cong Liao
6549aa2a899653d1d13299f7
Zi Gong
6549aa2a899653d1d13299f8
Huan Wang
6549aa2a899653d1d13299f9
Zhichao Lei
6549aa2a899653d1d13299fa
Ming Liang
6549aa2a899653d1d13299fb
Dajun Chen
admin_assigned
2023-11-07T15:37:51.451Z
61d41037443d3eaec85f1d4c
/avatars/9ab12d5c485bd2e3b2f351f0419d2164.svg
ChenDajun
user
cdj0311
6549aa2a899653d1d13299fc
Min Shen
6549aa2a899653d1d13299fd
Hailian Zhou
admin_assigned
2023-11-07T15:37:27.292Z
6475ea4ef60449361e569677
/avatars/ff00c019480dabccf86818e441426dd1.svg
hailian zhou
user
hailian
6549aa2a899653d1d13299fe
Hang Yu
claimed_verified
2025-05-28T19:15:48.710Z
6837243353739ee0d588d04c
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/CYU8xxEHxxGkwNVCpua6X.png
Hang Yu
user
fhlyhv
6549aa2a899653d1d13299ff
Jianguo Li
admin_assigned
2023-11-07T15:36:06.534Z
6539fc3fe279d2fc80c88902
/avatars/432ac2287a7f7d04c2c7e505070a3331.svg
Jianguo Li
user
jglee2046
MFTCoder: Boosting Code LLMs with Multitask Fine-Tuning
Code LLMs have emerged as a specialized research field, with remarkable
studies dedicated to enhancing model's coding capabilities through fine-tuning
on pre-trained models. Previous fine-tuning approaches were typically tailored
to specific downstream tasks or scenarios, which meant separate fine-tuning for
each task, requiring extensive training resources and posing challenges in
terms of deployment and maintenance. Furthermore, these approaches failed to
leverage the inherent interconnectedness among different code-related tasks. To
overcome these limitations, we present a multi-task fine-tuning framework,
MFTcoder, that enables simultaneous and parallel fine-tuning on multiple tasks.
By incorporating various loss functions, we effectively address common
challenges in multi-task learning, such as data imbalance, varying difficulty
levels, and inconsistent convergence speeds. Extensive experiments have
conclusively demonstrated that our multi-task fine-tuning approach outperforms
both individual fine-tuning on single tasks and fine-tuning on a mixed ensemble
of tasks. Moreover, MFTcoder offers efficient training capabilities, including
efficient data tokenization modes and PEFT fine-tuning, resulting in
significantly improved speed compared to traditional fine-tuning methods.
MFTcoder seamlessly integrates with several mainstream open-source LLMs, such
as CodeLLama and Qwen. Leveraging the CodeLLama foundation, our MFTcoder
fine-tuned model, CodeFuse-CodeLLama-34B, achieves an impressive
pass@1 score of 74.4\% on the HumaneEval benchmark, surpassing GPT-4
performance (67\%, zero-shot). MFTCoder is open-sourced at
https://github.com/codefuse-ai/MFTCOder
MFTcoder, a multi-task fine-tuning framework, improves the coding capabilities of LLMs through parallel fine-tuning on multiple tasks, achieving better performance than single-task fine-tuning and GPT-4 in zero-shot evaluation.
multi-task fine-tuning MFTcoder CodeLLama Qwen PEFT fine-tuning HumaneEval pass@1 score zero-shot performance