Self-Exploring Language Models: Active Preference Elicitation for Online
  Alignment
Preference optimization, particularly through Reinforcement Learning from
Human Feedback (RLHF), has achieved significant success in aligning Large
Language Models (LLMs) to adhere to human intentions. Unlike offline alignment
with a fixed dataset, online feedback collection from humans or AI on model
generations typically leads to more capable reward models and better-aligned
LLMs through an iterative process. However, achieving a globally accurate
reward model requires systematic exploration to generate diverse responses that
span the vast space of natural language. Random sampling from standard
reward-maximizing LLMs alone is insufficient to fulfill this requirement. To
address this issue, we propose a bilevel objective optimistically biased
towards potentially high-reward responses to actively explore
out-of-distribution regions. By solving the inner-level problem with the
reparameterized reward function, the resulting algorithm, named Self-Exploring
Language Models (SELM), eliminates the need for a separate RM and iteratively
updates the LLM with a straightforward objective. Compared to Direct Preference
Optimization (DPO), the SELM objective reduces indiscriminate favor of unseen
extrapolations and enhances exploration efficiency. Our experimental results
demonstrate that when finetuned on Zephyr-7B-SFT and Llama-3-8B-Instruct
models, SELM significantly boosts the performance on instruction-following
benchmarks such as MT-Bench and AlpacaEval 2.0, as well as various standard
academic benchmarks in different settings. Our code and models are available at
https://github.com/shenao-zhang/SELM.
665887d9605a5e89afda9133
Shenao Zhang
claimed_verified
2024-05-31T06:25:32.951Z
661213f894e0b3bff3e80c69
/avatars/d8febbb081825bf91e487aa8bad3a391.svg
Shenao Zhang
user
ZhangShenao
665887d9605a5e89afda9134
Donghan Yu
admin_assigned
2024-05-31T10:59:10.075Z
65b01b8a29ae836e9ed5af24
/avatars/a8b78a4b54d3f10858c5925521357001.svg
Donghan Yu
user
donghanyu
665887d9605a5e89afda9135
Hiteshi Sharma
665887d9605a5e89afda9136
Ziyi Yang
admin_assigned
2024-05-31T10:59:54.022Z
646be72031968a60a021ddb3
/avatars/79f3766cf404fd180274dbc86ea4dac6.svg
Ziyi Yang
user
Ziyi-Yang
665887d9605a5e89afda9137
Shuohang Wang
admin_assigned
2024-05-31T11:00:02.697Z
6463b2247572c66a8e625a57
/avatars/7722fb5649d42d966ce1e478946d5f8f.svg
Wang
user
Shuohang
665887d9605a5e89afda9138
Hany Hassan
admin_assigned
2024-05-31T11:00:09.148Z
65c262065dd83d86132e1ba3
/avatars/155fae1f1ccca4473c706b658326d36f.svg
Hany Hassan
user
HanyHassan
665887d9605a5e89afda9139
Zhaoran Wang
admin_assigned
2024-05-31T11:00:28.518Z
651b79208bf8bf5326fa2a55
/avatars/7e8e509861c693027d7993d066fea6ed.svg
Zhaoran Wang
user
zhaoranwang
Self-Exploring Language Models: Active Preference Elicitation for Online
  Alignment
Preference optimization, particularly through Reinforcement Learning from
Human Feedback (RLHF), has achieved significant success in aligning Large
Language Models (LLMs) to adhere to human intentions. Unlike offline alignment
with a fixed dataset, online feedback collection from humans or AI on model
generations typically leads to more capable reward models and better-aligned
LLMs through an iterative process. However, achieving a globally accurate
reward model requires systematic exploration to generate diverse responses that
span the vast space of natural language. Random sampling from standard
reward-maximizing LLMs alone is insufficient to fulfill this requirement. To
address this issue, we propose a bilevel objective optimistically biased
towards potentially high-reward responses to actively explore
out-of-distribution regions. By solving the inner-level problem with the
reparameterized reward function, the resulting algorithm, named Self-Exploring
Language Models (SELM), eliminates the need for a separate RM and iteratively
updates the LLM with a straightforward objective. Compared to Direct Preference
Optimization (DPO), the SELM objective reduces indiscriminate favor of unseen
extrapolations and enhances exploration efficiency. Our experimental results
demonstrate that when finetuned on Zephyr-7B-SFT and Llama-3-8B-Instruct
models, SELM significantly boosts the performance on instruction-following
benchmarks such as MT-Bench and AlpacaEval 2.0, as well as various standard
academic benchmarks in different settings. Our code and models are available at
https://github.com/shenao-zhang/SELM.
A bilevel objective for self-exploration in preference optimization enhances Large Language Models' performance by systematically exploring diverse responses and reducing bias towards unseen extrapolations.
Reinforcement Learning from Human Feedback Large Language Models reward models online feedback collection bilevel objective reparameterized reward function Self-Exploring Language Models Direct Preference Optimization instruction-following benchmarks academic benchmarks