title: Aligning Large Language Models via Self-Steering Optimization
summary: Automated alignment develops alignment systems with minimal human
  intervention. The key to automated alignment lies in providing learnable and
  accurate preference signals for preference learning without human annotation.
  In this paper, we introduce Self-Steering Optimization (SSO), an algorithm
  that autonomously generates high-quality preference signals based on predefined
  principles during iterative training, eliminating the need for manual
  annotation. SSO maintains the accuracy of signals by ensuring a consistent
  gap between chosen and rejected responses while keeping them both on-policy to
  suit the current policy model's learning capacity. SSO can benefit the online
  and offline training of the policy model, as well as enhance the training of
  reward models. We validate the effectiveness of SSO with two foundation
  models, Qwen2 and Llama3.1, indicating that it provides accurate, on-policy
  preference signals throughout iterative training. Without any manual annotation
  or external models, SSO leads to significant performance improvements across
  six subjective or objective benchmarks. Besides, the preference data generated
  by SSO significantly enhanced the performance of the reward model on
  Rewardbench. Our work presents a scalable approach to preference optimization,
  paving the way for more efficient and effective automated alignment.
paper_authors._id: 6718a271354948dfca7a1d1d
paper_authors.name: Hao Xiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-23T15:06:49.659Z
paper_authors._id: 63f33d500be81bdc5d902356
paper_authors.avatarUrl: /avatars/125812b9a86c3379b34ebfa8026f1a7f.svg
paper_authors.fullname: xianghao
paper_authors.type: user
paper_authors.user: xiangh
paper_authors._id: 6718a271354948dfca7a1d1e
paper_authors.name: Bowen Yu
paper_authors._id: 6718a271354948dfca7a1d1f
paper_authors.name: Hongyu Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-23T08:00:36.142Z
paper_authors._id: 6711c702f858a456b4b9f3a4
paper_authors.avatarUrl: /avatars/178e9567c3111ab22717c3c0dd003a6a.svg
paper_authors.fullname: Hongyu  Lin
paper_authors.type: user
paper_authors.user: sanmusunrise
paper_authors._id: 6718a271354948dfca7a1d20
paper_authors.name: Keming Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-23T08:00:30.026Z
paper_authors._id: 6453fa96ed6d7fede94408e0
paper_authors.avatarUrl: /avatars/e8c9025ef24cec958c87a1008bb54fd7.svg
paper_authors.fullname: Keming Lu
paper_authors.type: user
paper_authors.user: keminglu
paper_authors._id: 6718a271354948dfca7a1d21
paper_authors.name: Yaojie Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-23T08:00:24.045Z
paper_authors._id: 6216496a9b34d2fb49144599
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6216496a9b34d2fb49144599/41CKA_h1Ffj3RzVabSAkm.jpeg
paper_authors.fullname: Yaojie Lu
paper_authors.type: user
paper_authors.user: luyaojie
paper_authors._id: 6718a271354948dfca7a1d22
paper_authors.name: Xianpei Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-23T08:00:18.135Z
paper_authors._id: 65e99a77e71555ed193609cf
paper_authors.avatarUrl: /avatars/38ceb127883944677665da967d17dd18.svg
paper_authors.fullname: Xianpei Han
paper_authors.type: user
paper_authors.user: xphan
paper_authors._id: 6718a271354948dfca7a1d23
paper_authors.name: Le Sun
paper_authors._id: 6718a271354948dfca7a1d24
paper_authors.name: Jingren Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-23T08:00:02.877Z
paper_authors._id: 602f88f5e8149a962412a667
paper_authors.avatarUrl: /avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: Jingren
paper_authors._id: 6718a271354948dfca7a1d25
paper_authors.name: Junyang Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-23T07:59:52.670Z
paper_authors._id: 620760a26e3b7210c2ff1943
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
paper_authors.fullname: Junyang Lin
paper_authors.type: user
paper_authors.user: JustinLin610
paper_title: Aligning Large Language Models via Self-Steering Optimization
paper_summary: Automated alignment develops alignment systems with minimal human
  intervention. The key to automated alignment lies in providing learnable and
  accurate preference signals for preference learning without human annotation.
  In this paper, we introduce Self-Steering Optimization (SSO), an algorithm
  that autonomously generates high-quality preference signals based on predefined
  principles during iterative training, eliminating the need for manual
  annotation. SSO maintains the accuracy of signals by ensuring a consistent
  gap between chosen and rejected responses while keeping them both on-policy to
  suit the current policy model's learning capacity. SSO can benefit the online
  and offline training of the policy model, as well as enhance the training of
  reward models. We validate the effectiveness of SSO with two foundation
  models, Qwen2 and Llama3.1, indicating that it provides accurate, on-policy
  preference signals throughout iterative training. Without any manual annotation
  or external models, SSO leads to significant performance improvements across
  six subjective or objective benchmarks. Besides, the preference data generated
  by SSO significantly enhanced the performance of the reward model on
  Rewardbench. Our work presents a scalable approach to preference optimization,
  paving the way for more efficient and effective automated alignment.
paper_ai_summary: Self-Steering Optimization autonomously generates accurate preference signals without manual annotation, improving policy and reward model performance.
paper_ai_keywords: Self-Steering Optimization SSO preference learning preference signals iterative training policy model reward model