Reverse Preference Optimization for Complex Instruction Following
Instruction following (IF) is a critical capability for large language models
(LLMs). However, handling complex instructions with multiple constraints
remains challenging. Previous methods typically select preference pairs based
on the number of constraints they satisfy, introducing noise where chosen
examples may fail to follow some constraints and rejected examples may excel in
certain respects over the chosen ones. To address the challenge of aligning
with multiple preferences, we propose a simple yet effective method called
Reverse Preference Optimization (RPO). It mitigates noise in preference pairs
by dynamically reversing the constraints within the instruction to ensure the
chosen response is perfect, alleviating the burden of extensive sampling and
filtering to collect perfect responses. Besides, reversal also enlarges the gap
between chosen and rejected responses, thereby clarifying the optimization
direction and making it more robust to noise. We evaluate RPO on two multi-turn
IF benchmarks, Sysbench and Multi-IF, demonstrating average improvements over
the DPO baseline of 4.6 and 2.5 points (on Llama-3.1 8B), respectively.
Moreover, RPO scales effectively across model sizes (8B to 70B parameters),
with the 70B RPO model surpassing GPT-4o.
6837b720612e76702eb756b1
Xiang Huang
claimed_verified
2025-05-29T07:43:01.218Z
66168d11123f393614f7c908
/avatars/754779a228cb2ff31d21eb1221c576de.svg
xiang huang
user
xianghuang
6837b720612e76702eb756b2
Ting-En Lin
6837b720612e76702eb756b3
Feiteng Fang
claimed_verified
2025-06-10T09:29:36.680Z
634d4284f0a69955f66489f3
/avatars/9fac134537e6ef802f26d6d54c0e0bca.svg
feitengfang
user
feltoner
6837b720612e76702eb756b4
Yuchuan Wu
6837b720612e76702eb756b5
Hangyu Li
6837b720612e76702eb756b6
Yuzhong Qu
6837b720612e76702eb756b7
Fei Huang
6837b720612e76702eb756b8
Yongbin Li
Reverse Preference Optimization for Complex Instruction Following
Instruction following (IF) is a critical capability for large language models
(LLMs). However, handling complex instructions with multiple constraints
remains challenging. Previous methods typically select preference pairs based
on the number of constraints they satisfy, introducing noise where chosen
examples may fail to follow some constraints and rejected examples may excel in
certain respects over the chosen ones. To address the challenge of aligning
with multiple preferences, we propose a simple yet effective method called
Reverse Preference Optimization (RPO). It mitigates noise in preference pairs
by dynamically reversing the constraints within the instruction to ensure the
chosen response is perfect, alleviating the burden of extensive sampling and
filtering to collect perfect responses. Besides, reversal also enlarges the gap
between chosen and rejected responses, thereby clarifying the optimization
direction and making it more robust to noise. We evaluate RPO on two multi-turn
IF benchmarks, Sysbench and Multi-IF, demonstrating average improvements over
the DPO baseline of 4.6 and 2.5 points (on Llama-3.1 8B), respectively.
Moreover, RPO scales effectively across model sizes (8B to 70B parameters),
with the 70B RPO model surpassing GPT-4o.
Reverse Preference Optimization (RPO) dynamically reverses instruction constraints to improve preference pair selection and optimize instruction following for large language models.
Instruction Following Reverse Preference Optimization preference pairs constraints multi-turn IF Sysbench Multi-IF DPO Llama-3.1 GPT-4o