title: Step-On-Feet Tuning: Scaling Self-Alignment of LLMs via Bootstrapping
summary: Self-alignment is an effective way to reduce the cost of human annotation
  while ensuring promising model capability. However, most current methods
  complete the data collection and training steps in a single round, which may
  overlook the continuously improving ability of self-aligned models. This gives
  rise to a key query: What if we do multi-time bootstrapping self-alignment?
  Does this strategy enhance model performance or lead to rapid degradation? In
  this paper, our pioneering exploration delves into the impact of bootstrapping
  self-alignment on large language models. Our findings reveal that bootstrapping
  self-alignment markedly surpasses the single-round approach, by guaranteeing
  data diversity from in-context learning. To further exploit the capabilities of
  bootstrapping, we investigate and adjust the training order of data, which
  yields improved performance of the model. Drawing on these findings, we propose
  Step-On-Feet Tuning (SOFT) which leverages model's continuously enhanced
  few-shot ability to boost zero or one-shot performance. Based on easy-to-hard
  training recipe, we propose SOFT+ which further boost self-alignment's
  performance. Our experiments demonstrate the efficiency of SOFT (SOFT+) across
  various classification and generation tasks, highlighting the potential of
  bootstrapping self-alignment on continually enhancing model alignment
  performance.
paper_authors._id: 65caddae6427380bc2e5d1ad
paper_authors.name: Haoyu Wang
paper_authors._id: 65caddae6427380bc2e5d1ae
paper_authors.name: Guozheng Ma
paper_authors._id: 65caddae6427380bc2e5d1af
paper_authors.name: Ziqiao Meng
paper_authors._id: 65caddae6427380bc2e5d1b0
paper_authors.name: Zeyu Qin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T14:00:08.794Z
paper_authors._id: 641a7a8ff9dd6391a2457bbe
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/641a7a8ff9dd6391a2457bbe/_qdCnlrkpTfqZjXY4DFq_.jpeg
paper_authors.fullname: Zeyu Qin
paper_authors.type: user
paper_authors.user: qqqzzzyyy
paper_authors._id: 65caddae6427380bc2e5d1b1
paper_authors.name: Li Shen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-26T11:40:55.463Z
paper_authors._id: 62de356ad89af8c07209e7d4
paper_authors.avatarUrl: /avatars/610629958726b270418368b8b7f61469.svg
paper_authors.fullname: Li Shen
paper_authors.type: user
paper_authors.user: mathshenli
paper_authors._id: 65caddae6427380bc2e5d1b2
paper_authors.name: Zhong Zhang
paper_authors._id: 65caddae6427380bc2e5d1b3
paper_authors.name: Bingzhe Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T14:01:39.393Z
paper_authors._id: 635e7cf8d1c91c2ee1e8d421
paper_authors.avatarUrl: /avatars/4ce7699d79cb1f005f95e58763cdf81f.svg
paper_authors.fullname: Wu
paper_authors.type: user
paper_authors.user: Bingzhe
paper_authors._id: 65caddae6427380bc2e5d1b4
paper_authors.name: Liu Liu
paper_authors._id: 65caddae6427380bc2e5d1b5
paper_authors.name: Yatao Bian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T14:01:59.345Z
paper_authors._id: 6357c85c00f138b8ca550c0e
paper_authors.avatarUrl: /avatars/95dad6bb2fe1d31a4e9da0d6281be17b.svg
paper_authors.fullname: BlueWhaleLab
paper_authors.type: user
paper_authors.user: BlueWhaleLab
paper_authors._id: 65caddae6427380bc2e5d1b6
paper_authors.name: Tingyang Xu
paper_authors._id: 65caddae6427380bc2e5d1b7
paper_authors.name: Xueqian Wang
paper_authors._id: 65caddae6427380bc2e5d1b8
paper_authors.name: Peilin Zhao
paper_title: Step-On-Feet Tuning: Scaling Self-Alignment of LLMs via Bootstrapping
paper_summary: Self-alignment is an effective way to reduce the cost of human annotation
  while ensuring promising model capability. However, most current methods
  complete the data collection and training steps in a single round, which may
  overlook the continuously improving ability of self-aligned models. This gives
  rise to a key query: What if we do multi-time bootstrapping self-alignment?
  Does this strategy enhance model performance or lead to rapid degradation? In
  this paper, our pioneering exploration delves into the impact of bootstrapping
  self-alignment on large language models. Our findings reveal that bootstrapping
  self-alignment markedly surpasses the single-round approach, by guaranteeing
  data diversity from in-context learning. To further exploit the capabilities of
  bootstrapping, we investigate and adjust the training order of data, which
  yields improved performance of the model. Drawing on these findings, we propose
  Step-On-Feet Tuning (SOFT) which leverages model's continuously enhanced
  few-shot ability to boost zero or one-shot performance. Based on easy-to-hard
  training recipe, we propose SOFT+ which further boost self-alignment's
  performance. Our experiments demonstrate the efficiency of SOFT (SOFT+) across
  various classification and generation tasks, highlighting the potential of
  bootstrapping self-alignment on continually enhancing model alignment
  performance.
paper_ai_keywords: self-aligned models bootstrapping self-alignment in-context learning Step-On-Feet Tuning (SOFT) zero or one-shot performance easy-to-hard training recipe continually enhancing model alignment