title: Reverse Preference Optimization for Complex Instruction Following
summary: Instruction following (IF) is a critical capability for large language models
  (LLMs). However, handling complex instructions with multiple constraints
  remains challenging. Previous methods typically select preference pairs based
  on the number of constraints they satisfy, introducing noise where chosen
  examples may fail to follow some constraints and rejected examples may excel in
  certain respects over the chosen ones. To address the challenge of aligning
  with multiple preferences, we propose a simple yet effective method called
  Reverse Preference Optimization (RPO). It mitigates noise in preference pairs
  by dynamically reversing the constraints within the instruction to ensure the
  chosen response is perfect, alleviating the burden of extensive sampling and
  filtering to collect perfect responses. Besides, reversal also enlarges the gap
  between chosen and rejected responses, thereby clarifying the optimization
  direction and making it more robust to noise. We evaluate RPO on two multi-turn
  IF benchmarks, Sysbench and Multi-IF, demonstrating average improvements over
  the DPO baseline of 4.6 and 2.5 points (on Llama-3.1 8B), respectively.
  Moreover, RPO scales effectively across model sizes (8B to 70B parameters),
  with the 70B RPO model surpassing GPT-4o.
paper_authors._id: 6837b720612e76702eb756b1
paper_authors.name: Xiang Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T07:43:01.218Z
paper_authors._id: 66168d11123f393614f7c908
paper_authors.avatarUrl: /avatars/754779a228cb2ff31d21eb1221c576de.svg
paper_authors.fullname: xiang huang
paper_authors.type: user
paper_authors.user: xianghuang
paper_authors._id: 6837b720612e76702eb756b2
paper_authors.name: Ting-En Lin
paper_authors._id: 6837b720612e76702eb756b3
paper_authors.name: Feiteng Fang
paper_authors._id: 6837b720612e76702eb756b4
paper_authors.name: Yuchuan Wu
paper_authors._id: 6837b720612e76702eb756b5
paper_authors.name: Hangyu Li
paper_authors._id: 6837b720612e76702eb756b6
paper_authors.name: Yuzhong Qu
paper_authors._id: 6837b720612e76702eb756b7
paper_authors.name: Fei Huang
paper_authors._id: 6837b720612e76702eb756b8
paper_authors.name: Yongbin Li
paper_title: Reverse Preference Optimization for Complex Instruction Following
paper_summary: Instruction following (IF) is a critical capability for large language models
  (LLMs). However, handling complex instructions with multiple constraints
  remains challenging. Previous methods typically select preference pairs based
  on the number of constraints they satisfy, introducing noise where chosen
  examples may fail to follow some constraints and rejected examples may excel in
  certain respects over the chosen ones. To address the challenge of aligning
  with multiple preferences, we propose a simple yet effective method called
  Reverse Preference Optimization (RPO). It mitigates noise in preference pairs
  by dynamically reversing the constraints within the instruction to ensure the
  chosen response is perfect, alleviating the burden of extensive sampling and
  filtering to collect perfect responses. Besides, reversal also enlarges the gap
  between chosen and rejected responses, thereby clarifying the optimization
  direction and making it more robust to noise. We evaluate RPO on two multi-turn
  IF benchmarks, Sysbench and Multi-IF, demonstrating average improvements over
  the DPO baseline of 4.6 and 2.5 points (on Llama-3.1 8B), respectively.
  Moreover, RPO scales effectively across model sizes (8B to 70B parameters),
  with the 70B RPO model surpassing GPT-4o.
paper_ai_summary: Reverse Preference Optimization (RPO) dynamically reverses instruction constraints to improve preference pair selection and optimize instruction following for large language models.
paper_ai_keywords: Instruction Following Reverse Preference Optimization preference pairs constraints multi-turn IF Sysbench Multi-IF DPO Llama-3.1 GPT-4o