title: SFTMix: Elevating Language Model Instruction Tuning with Mixup Recipe
summary: To induce desired behaviors in large language models (LLMs) for
  interaction-driven tasks, the instruction-tuning stage typically trains LLMs on
  instruction-response pairs using the next-token prediction (NTP) loss. Previous
  work aiming to improve instruction-tuning performance often emphasizes the need
  for higher-quality supervised fine-tuning (SFT) datasets, which typically
  involves expensive data filtering with proprietary LLMs or labor-intensive data
  generation by human annotators. However, these approaches do not fully leverage
  the datasets' intrinsic properties, resulting in high computational and labor
  costs, thereby limiting scalability and performance gains. In this paper, we
  propose SFTMix, a novel recipe that elevates instruction-tuning performance
  beyond the conventional NTP paradigm, without the need for well-curated
  datasets. Observing that LLMs exhibit uneven confidence across the semantic
  representation space, we argue that examples with different confidence levels
  should play distinct roles during the instruction-tuning process. Based on this
  insight, SFTMix leverages training dynamics to identify examples with varying
  confidence levels, then applies a Mixup-based regularization to mitigate
  overfitting on confident examples while propagating supervision signals to
  improve learning on relatively unconfident ones. This approach enables SFTMix
  to significantly outperform NTP across a wide range of instruction-following
  and healthcare domain-specific SFT tasks, demonstrating its adaptability to
  diverse LLM families and scalability to datasets of any size. Comprehensive
  ablation studies further verify the robustness of SFTMix's design choices,
  underscoring its versatility in consistently enhancing performance across
  different LLMs and datasets in broader natural language processing
  applications.
paper_authors._id: 6704ad5b5788bbf517e602e2
paper_authors.name: Yuxin Xiao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T07:19:13.853Z
paper_authors._id: 64bf072bae436c8813494ba3
paper_authors.avatarUrl: /avatars/afb96d2bbf90411f4b1a030ebebff300.svg
paper_authors.fullname: Yuxin Xiao
paper_authors.type: user
paper_authors.user: YuxinXiao
paper_authors._id: 6704ad5b5788bbf517e602e3
paper_authors.name: Shujian Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T10:44:29.851Z
paper_authors._id: 5e614b26f0c9e93097d0d4a6
paper_authors.avatarUrl: /avatars/b85e7714205b573d03412f975b7e4348.svg
paper_authors.fullname: Shujian Zhang
paper_authors.type: user
paper_authors.user: szhang42
paper_authors._id: 6704ad5b5788bbf517e602e4
paper_authors.name: Wenxuan Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-11T10:44:48.186Z
paper_authors._id: 619414c7979eb76ec3cc5bab
paper_authors.avatarUrl: /avatars/c1a7055db6150b1bba92e7b13266b489.svg
paper_authors.fullname: Wenxuan Zhou
paper_authors.type: user
paper_authors.user: wzhouad
paper_authors._id: 6704ad5b5788bbf517e602e5
paper_authors.name: Marzyeh Ghassemi
paper_authors._id: 6704ad5b5788bbf517e602e6
paper_authors.name: Sanqiang Zhao
paper_title: SFTMix: Elevating Language Model Instruction Tuning with Mixup Recipe
paper_summary: To induce desired behaviors in large language models (LLMs) for
  interaction-driven tasks, the instruction-tuning stage typically trains LLMs on
  instruction-response pairs using the next-token prediction (NTP) loss. Previous
  work aiming to improve instruction-tuning performance often emphasizes the need
  for higher-quality supervised fine-tuning (SFT) datasets, which typically
  involves expensive data filtering with proprietary LLMs or labor-intensive data
  generation by human annotators. However, these approaches do not fully leverage
  the datasets' intrinsic properties, resulting in high computational and labor
  costs, thereby limiting scalability and performance gains. In this paper, we
  propose SFTMix, a novel recipe that elevates instruction-tuning performance
  beyond the conventional NTP paradigm, without the need for well-curated
  datasets. Observing that LLMs exhibit uneven confidence across the semantic
  representation space, we argue that examples with different confidence levels
  should play distinct roles during the instruction-tuning process. Based on this
  insight, SFTMix leverages training dynamics to identify examples with varying
  confidence levels, then applies a Mixup-based regularization to mitigate
  overfitting on confident examples while propagating supervision signals to
  improve learning on relatively unconfident ones. This approach enables SFTMix
  to significantly outperform NTP across a wide range of instruction-following
  and healthcare domain-specific SFT tasks, demonstrating its adaptability to
  diverse LLM families and scalability to datasets of any size. Comprehensive
  ablation studies further verify the robustness of SFTMix's design choices,
  underscoring its versatility in consistently enhancing performance across
  different LLMs and datasets in broader natural language processing
  applications.
paper_ai_keywords: instruction-tuning next-token prediction (NTP) supervised fine-tuning (S) semantic representation space Mixup-based instruction-following healthcare domain-specific SFT tasks LLM families natural language processing (NLP)