title: Iterative Length-Regularized Direct Preference Optimization: A Case
    Study on Improving 7B Language Models to GPT-4 Level
summary: Direct Preference Optimization (DPO), a standard method for aligning language
  models with human preferences, is traditionally applied to offline preferences.
  Recent studies show that DPO benefits from iterative training with online
  preferences labeled by a trained reward model. In this work, we identify a
  pitfall of vanilla iterative DPO - improved response quality can lead to
  increased verbosity. To address this, we introduce iterative length-regularized
  DPO (iLR-DPO) to penalize response length. Our empirical results show that
  iLR-DPO can enhance a 7B model to perform on par with GPT-4 without increasing
  verbosity. Specifically, our 7B model achieves a 50.5% length-controlled win
  rate against GPT-4 Preview on AlpacaEval 2.0, and excels across
  standard benchmarks including MT-Bench, Arena-Hard and OpenLLM Leaderboard.
  These results demonstrate the effectiveness of iterative DPO in aligning
  language models with human feedback.
paper_authors._id: 6670f21456709bb492312de9
paper_authors.name: Jie Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T09:11:05.671Z
paper_authors._id: 639be86b59473c6ae02ef9c4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/639be86b59473c6ae02ef9c4/gw34RBCVZCOkcAA79xUr3.png
paper_authors.fullname: Jie Liu
paper_authors.type: user
paper_authors.user: jieliu
paper_authors._id: 6670f21456709bb492312dea
paper_authors.name: Zhanhui Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T12:56:23.932Z
paper_authors._id: 642e5a7ba0b65dce1f87a7a2
paper_authors.avatarUrl: /avatars/3ae01c9330a47e98fac9f1eb0ba94073.svg
paper_authors.fullname: Zhanhui Zhou
paper_authors.type: user
paper_authors.user: ZHZisZZ
paper_authors._id: 6670f21456709bb492312deb
paper_authors.name: Jiaheng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T12:56:29.594Z
paper_authors._id: 65377c30e48353201e6fdda0
paper_authors.avatarUrl: /avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
paper_authors.fullname: Jiaheng Liu
paper_authors.type: user
paper_authors.user: CheeryLJH
paper_authors._id: 6670f21456709bb492312dec
paper_authors.name: Xingyuan Bu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T12:56:35.212Z
paper_authors._id: 6444e7765691ca69b0d95856
paper_authors.avatarUrl: /avatars/4ae5001e7c7dea89c4deaf2b05436857.svg
paper_authors.fullname: Xingyuan Bu
paper_authors.type: user
paper_authors.user: sefira32
paper_authors._id: 6670f21456709bb492312ded
paper_authors.name: Chao Yang
paper_authors._id: 6670f21456709bb492312dee
paper_authors.name: Han-Sen Zhong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T12:56:42.468Z
paper_authors._id: 663cc7a6662ad3bf0b346d78
paper_authors.avatarUrl: /avatars/15e4e187225d25051d348f3f722041ca.svg
paper_authors.fullname: Han-Sen Zhong
paper_authors.type: user
paper_authors.user: hansenzhong
paper_authors._id: 6670f21456709bb492312def
paper_authors.name: Wanli Ouyang
paper_title: Iterative Length-Regularized Direct Preference Optimization: A Case
    Study on Improving 7B Language Models to GPT-4 Level
paper_summary: Direct Preference Optimization (DPO), a standard method for aligning language
  models with human preferences, is traditionally applied to offline preferences.
  Recent studies show that DPO benefits from iterative training with online
  preferences labeled by a trained reward model. In this work, we identify a
  pitfall of vanilla iterative DPO - improved response quality can lead to
  increased verbosity. To address this, we introduce iterative length-regularized
  DPO (iLR-DPO) to penalize response length. Our empirical results show that
  iLR-DPO can enhance a 7B model to perform on par with GPT-4 without increasing
  verbosity. Specifically, our 7B model achieves a 50.5% length-controlled win
  rate against GPT-4 Preview on AlpacaEval 2.0, and excels across
  standard benchmarks including MT-Bench, Arena-Hard and OpenLLM Leaderboard.
  These results demonstrate the effectiveness of iterative DPO in aligning
  language models with human feedback.
paper_ai_summary: Iterative length-regularized DPO effectively aligns language models with human preferences while controlling verbosity, outperforming larger models on various benchmarks.
paper_ai_keywords: Direct Preference Optimization DPO iterative DPO iLR-DPO length-regularized response quality verbosity GPT-4 AlpacaEval 2.0 MT-Bench Arena-Hard OpenLLM Leaderboard