title: Self-Exploring Language Models: Active Preference Elicitation for Online
    Alignment
summary: Preference optimization, particularly through Reinforcement Learning from
  Human Feedback (RLHF), has achieved significant success in aligning Large
  Language Models (LLMs) to adhere to human intentions. Unlike offline alignment
  with a fixed dataset, online feedback collection from humans or AI on model
  generations typically leads to more capable reward models and better-aligned
  LLMs through an iterative process. However, achieving a globally accurate
  reward model requires systematic exploration to generate diverse responses that
  span the vast space of natural language. Random sampling from standard
  reward-maximizing LLMs alone is insufficient to fulfill this requirement. To
  address this issue, we propose a bilevel objective optimistically biased
  towards potentially high-reward responses to actively explore
  out-of-distribution regions. By solving the inner-level problem with the
  reparameterized reward function, the resulting algorithm, named Self-Exploring
  Language Models (SELM), eliminates the need for a separate RM and iteratively
  updates the LLM with a straightforward objective. Compared to Direct Preference
  Optimization (DPO), the SELM objective reduces indiscriminate favor of unseen
  extrapolations and enhances exploration efficiency. Our experimental results
  demonstrate that when finetuned on Zephyr-7B-SFT and Llama-3-8B-Instruct
  models, SELM significantly boosts the performance on instruction-following
  benchmarks such as MT-Bench and AlpacaEval 2.0, as well as various standard
  academic benchmarks in different settings. Our code and models are available at
  https://github.com/shenao-zhang/SELM.
paper_authors._id: 665887d9605a5e89afda9133
paper_authors.name: Shenao Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-31T06:25:32.951Z
paper_authors._id: 661213f894e0b3bff3e80c69
paper_authors.avatarUrl: /avatars/d8febbb081825bf91e487aa8bad3a391.svg
paper_authors.fullname: Shenao Zhang
paper_authors.type: user
paper_authors.user: ZhangShenao
paper_authors._id: 665887d9605a5e89afda9134
paper_authors.name: Donghan Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T10:59:10.075Z
paper_authors._id: 65b01b8a29ae836e9ed5af24
paper_authors.avatarUrl: /avatars/a8b78a4b54d3f10858c5925521357001.svg
paper_authors.fullname: Donghan Yu
paper_authors.type: user
paper_authors.user: donghanyu
paper_authors._id: 665887d9605a5e89afda9135
paper_authors.name: Hiteshi Sharma
paper_authors._id: 665887d9605a5e89afda9136
paper_authors.name: Ziyi Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T10:59:54.022Z
paper_authors._id: 646be72031968a60a021ddb3
paper_authors.avatarUrl: /avatars/79f3766cf404fd180274dbc86ea4dac6.svg
paper_authors.fullname: Ziyi Yang
paper_authors.type: user
paper_authors.user: Ziyi-Yang
paper_authors._id: 665887d9605a5e89afda9137
paper_authors.name: Shuohang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T11:00:02.697Z
paper_authors._id: 6463b2247572c66a8e625a57
paper_authors.avatarUrl: /avatars/7722fb5649d42d966ce1e478946d5f8f.svg
paper_authors.fullname: Wang
paper_authors.type: user
paper_authors.user: Shuohang
paper_authors._id: 665887d9605a5e89afda9138
paper_authors.name: Hany Hassan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T11:00:09.148Z
paper_authors._id: 65c262065dd83d86132e1ba3
paper_authors.avatarUrl: /avatars/155fae1f1ccca4473c706b658326d36f.svg
paper_authors.fullname: Hany Hassan
paper_authors.type: user
paper_authors.user: HanyHassan
paper_authors._id: 665887d9605a5e89afda9139
paper_authors.name: Zhaoran Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T11:00:28.518Z
paper_authors._id: 651b79208bf8bf5326fa2a55
paper_authors.avatarUrl: /avatars/7e8e509861c693027d7993d066fea6ed.svg
paper_authors.fullname: Zhaoran Wang
paper_authors.type: user
paper_authors.user: zhaoranwang
paper_title: Self-Exploring Language Models: Active Preference Elicitation for Online
    Alignment
paper_summary: Preference optimization, particularly through Reinforcement Learning from
  Human Feedback (RLHF), has achieved significant success in aligning Large
  Language Models (LLMs) to adhere to human intentions. Unlike offline alignment
  with a fixed dataset, online feedback collection from humans or AI on model
  generations typically leads to more capable reward models and better-aligned
  LLMs through an iterative process. However, achieving a globally accurate
  reward model requires systematic exploration to generate diverse responses that
  span the vast space of natural language. Random sampling from standard
  reward-maximizing LLMs alone is insufficient to fulfill this requirement. To
  address this issue, we propose a bilevel objective optimistically biased
  towards potentially high-reward responses to actively explore
  out-of-distribution regions. By solving the inner-level problem with the
  reparameterized reward function, the resulting algorithm, named Self-Exploring
  Language Models (SELM), eliminates the need for a separate RM and iteratively
  updates the LLM with a straightforward objective. Compared to Direct Preference
  Optimization (DPO), the SELM objective reduces indiscriminate favor of unseen
  extrapolations and enhances exploration efficiency. Our experimental results
  demonstrate that when finetuned on Zephyr-7B-SFT and Llama-3-8B-Instruct
  models, SELM significantly boosts the performance on instruction-following
  benchmarks such as MT-Bench and AlpacaEval 2.0, as well as various standard
  academic benchmarks in different settings. Our code and models are available at
  https://github.com/shenao-zhang/SELM.
paper_ai_summary: A bilevel objective for self-exploration in preference optimization enhances Large Language Models' performance by systematically exploring diverse responses and reducing bias towards unseen extrapolations.
paper_ai_keywords: Reinforcement Learning from Human Feedback Large Language Models reward models online feedback collection bilevel objective reparameterized reward function Self-Exploring Language Models Direct Preference Optimization instruction-following benchmarks academic benchmarks