title: Selective Self-to-Supervised Fine-Tuning for Generalization in Large
    Language Models
summary: Fine-tuning Large Language Models (LLMs) on specific datasets is a common
  practice to improve performance on target tasks. However, this performance gain
  often leads to overfitting, where the model becomes too specialized in either
  the task or the characteristics of the training data, resulting in a loss of
  generalization. This paper introduces Selective Self-to-Supervised Fine-Tuning
  (S3FT), a fine-tuning approach that achieves better performance than the
  standard supervised fine-tuning (SFT) while improving generalization. S3FT
  leverages the existence of multiple valid responses to a query. By utilizing
  the model's correct responses, S3FT reduces model specialization during the
  fine-tuning stage. S3FT first identifies the correct model responses from the
  training set by deploying an appropriate judge. Then, it fine-tunes the model
  using the correct model responses and the gold response (or its paraphrase) for
  the remaining samples. The effectiveness of S3FT is demonstrated through
  experiments on mathematical reasoning, Python programming and reading
  comprehension tasks. The results show that standard SFT can lead to an average
  performance drop of up to 4.4 on multiple benchmarks, such as MMLU and
  TruthfulQA. In contrast, S3FT reduces this drop by half, i.e. 2.5, indicating
  better generalization capabilities than SFT while performing significantly
  better on the fine-tuning tasks.
paper_authors._id: 67b3716bab1b992c7f4599da
paper_authors.name: Sonam Gupta
paper_authors._id: 67b3716bab1b992c7f4599db
paper_authors.name: Yatin Nandwani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T15:52:54.491Z
paper_authors._id: 6626284411772517e546d799
paper_authors.avatarUrl: /avatars/a9f2f1958c9f17ccc911fa323d9af2f3.svg
paper_authors.fullname: Yatin Nandwani
paper_authors.type: user
paper_authors.user: ynandwan
paper_authors._id: 67b3716bab1b992c7f4599dc
paper_authors.name: Asaf Yehudai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T15:53:01.485Z
paper_authors._id: 638324f862badff43269e588
paper_authors.avatarUrl: /avatars/907a39a9b44fc8b7f3fad35858b01fb7.svg
paper_authors.fullname: Asaf Yehudai
paper_authors.type: user
paper_authors.user: Asaf-Yehudai
paper_authors._id: 67b3716bab1b992c7f4599dd
paper_authors.name: Dinesh Khandelwal
paper_authors._id: 67b3716bab1b992c7f4599de
paper_authors.name: Dinesh Raghu
paper_authors._id: 67b3716bab1b992c7f4599df
paper_authors.name: Sachindra Joshi
paper_title: Selective Self-to-Supervised Fine-Tuning for Generalization in Large
    Language Models
paper_summary: Fine-tuning Large Language Models (LLMs) on specific datasets is a common
  practice to improve performance on target tasks. However, this performance gain
  often leads to overfitting, where the model becomes too specialized in either
  the task or the characteristics of the training data, resulting in a loss of
  generalization. This paper introduces Selective Self-to-Supervised Fine-Tuning
  (S3FT), a fine-tuning approach that achieves better performance than the
  standard supervised fine-tuning (SFT) while improving generalization. S3FT
  leverages the existence of multiple valid responses to a query. By utilizing
  the model's correct responses, S3FT reduces model specialization during the
  fine-tuning stage. S3FT first identifies the correct model responses from the
  training set by deploying an appropriate judge. Then, it fine-tunes the model
  using the correct model responses and the gold response (or its paraphrase) for
  the remaining samples. The effectiveness of S3FT is demonstrated through
  experiments on mathematical reasoning, Python programming and reading
  comprehension tasks. The results show that standard SFT can lead to an average
  performance drop of up to 4.4 on multiple benchmarks, such as MMLU and
  TruthfulQA. In contrast, S3FT reduces this drop by half, i.e. 2.5, indicating
  better generalization capabilities than SFT while performing significantly
  better on the fine-tuning tasks.
paper_ai_keywords: Selective Self-to-Supervised Fine-Tuning (S3FT) standard supervised fine-tuning (SFT) overfitting generalization valid responses correct model responses judge gold response paraphrase mathematical reasoning Python programming reading comprehension MMLU TruthfulQA