SFTMix: Elevating Language Model Instruction Tuning with Mixup Recipe
To induce desired behaviors in large language models (LLMs) for
interaction-driven tasks, the instruction-tuning stage typically trains LLMs on
instruction-response pairs using the next-token prediction (NTP) loss. Previous
work aiming to improve instruction-tuning performance often emphasizes the need
for higher-quality supervised fine-tuning (SFT) datasets, which typically
involves expensive data filtering with proprietary LLMs or labor-intensive data
generation by human annotators. However, these approaches do not fully leverage
the datasets' intrinsic properties, resulting in high computational and labor
costs, thereby limiting scalability and performance gains. In this paper, we
propose SFTMix, a novel recipe that elevates instruction-tuning performance
beyond the conventional NTP paradigm, without the need for well-curated
datasets. Observing that LLMs exhibit uneven confidence across the semantic
representation space, we argue that examples with different confidence levels
should play distinct roles during the instruction-tuning process. Based on this
insight, SFTMix leverages training dynamics to identify examples with varying
confidence levels, then applies a Mixup-based regularization to mitigate
overfitting on confident examples while propagating supervision signals to
improve learning on relatively unconfident ones. This approach enables SFTMix
to significantly outperform NTP across a wide range of instruction-following
and healthcare domain-specific SFT tasks, demonstrating its adaptability to
diverse LLM families and scalability to datasets of any size. Comprehensive
ablation studies further verify the robustness of SFTMix's design choices,
underscoring its versatility in consistently enhancing performance across
different LLMs and datasets in broader natural language processing
applications.
6704ad5b5788bbf517e602e2
Yuxin Xiao
claimed_verified
2024-10-08T07:19:13.853Z
64bf072bae436c8813494ba3
/avatars/afb96d2bbf90411f4b1a030ebebff300.svg
Yuxin Xiao
user
YuxinXiao
6704ad5b5788bbf517e602e3
Shujian Zhang
admin_assigned
2024-10-11T10:44:29.851Z
5e614b26f0c9e93097d0d4a6
/avatars/b85e7714205b573d03412f975b7e4348.svg
Shujian Zhang
user
szhang42
6704ad5b5788bbf517e602e4
Wenxuan Zhou
admin_assigned
2024-10-11T10:44:48.186Z
619414c7979eb76ec3cc5bab
/avatars/c1a7055db6150b1bba92e7b13266b489.svg
Wenxuan Zhou
user
wzhouad
6704ad5b5788bbf517e602e5
Marzyeh Ghassemi
6704ad5b5788bbf517e602e6
Sanqiang Zhao
SFTMix: Elevating Language Model Instruction Tuning with Mixup Recipe
To induce desired behaviors in large language models (LLMs) for
interaction-driven tasks, the instruction-tuning stage typically trains LLMs on
instruction-response pairs using the next-token prediction (NTP) loss. Previous
work aiming to improve instruction-tuning performance often emphasizes the need
for higher-quality supervised fine-tuning (SFT) datasets, which typically
involves expensive data filtering with proprietary LLMs or labor-intensive data
generation by human annotators. However, these approaches do not fully leverage
the datasets' intrinsic properties, resulting in high computational and labor
costs, thereby limiting scalability and performance gains. In this paper, we
propose SFTMix, a novel recipe that elevates instruction-tuning performance
beyond the conventional NTP paradigm, without the need for well-curated
datasets. Observing that LLMs exhibit uneven confidence across the semantic
representation space, we argue that examples with different confidence levels
should play distinct roles during the instruction-tuning process. Based on this
insight, SFTMix leverages training dynamics to identify examples with varying
confidence levels, then applies a Mixup-based regularization to mitigate
overfitting on confident examples while propagating supervision signals to
improve learning on relatively unconfident ones. This approach enables SFTMix
to significantly outperform NTP across a wide range of instruction-following
and healthcare domain-specific SFT tasks, demonstrating its adaptability to
diverse LLM families and scalability to datasets of any size. Comprehensive
ablation studies further verify the robustness of SFTMix's design choices,
underscoring its versatility in consistently enhancing performance across
different LLMs and datasets in broader natural language processing
applications.
SFTMix enhances instruction-tuning performance using Mixup-based regularization to manage examples with varying confidence, outperforming NTP in diverse instruction-following and healthcare tasks.
instruction-tuning next-token prediction supervised fine-tuning SFT Mixup confidence levels overfitting instruction-following healthcare domain SFTMix large language models LLMs