title: Weighted-Reward Preference Optimization for Implicit Model Fusion
summary: While fusing heterogeneous open-source LLMs with varying architectures and
  sizes can potentially integrate the strengths of different models, existing
  fusion methods face significant challenges, such as vocabulary alignment and
  merging distribution matrices. These procedures are not only complex but also
  prone to introducing noise and errors. In this paper, we propose an implicit
  fusion method, Weighted-Reward Preference Optimization (WRPO), which leverages
  preference optimization between the source LLMs and the target LLM to transfer
  their capabilities effectively. WRPO eliminates the need for vocabulary
  alignment and matrix fusion and can be efficiently scaled to accommodate
  various LLMs. To address distributional deviations between the source and
  target LLMs, WRPO introduces a progressive adaptation strategy that gradually
  shifts reliance on preferred examples from the target LLM to the source LLMs.
  Extensive experiments on the MT-Bench, AlpacaEval-2, and Arena-Hard benchmarks
  demonstrate that WRPO consistently outperforms existing knowledge fusion
  methods and various fine-tuning baselines. When applied to LLaMA3-8B-Instruct
  as the target model, WRPO achieves a length-controlled win rate of 55.9%
  against GPT-4-Preview-1106 on AlpacaEval-2 and a win rate of 46.2% against
  GPT-4-0314 on Arena-Hard. Our code is available at
  https://github.com/SLIT-AI/WRPO.
paper_authors._id: 675146f4372c1020c21d2046
paper_authors.name: Ziyi Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T10:37:58.424Z
paper_authors._id: 64c9b0f28d2d187c24d1e6c1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/1CPnAaB3gsupdpiNWaoDc.png
paper_authors.fullname: ZiYi Yang
paper_authors.type: user
paper_authors.user: AALF
paper_authors._id: 675146f4372c1020c21d2047
paper_authors.name: Fanqi Wan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T12:23:54.907Z
paper_authors._id: 62ecbffd99112e99c5f7fded
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62ecbffd99112e99c5f7fded/U6iXAJbpm2vaC5qksEPiH.png
paper_authors.fullname: Fanqi Wan
paper_authors.type: user
paper_authors.user: Wanfq
paper_authors._id: 675146f4372c1020c21d2048
paper_authors.name: Longguang Zhong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T12:23:49.460Z
paper_authors._id: 62b6d20416ff90e6198301b6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656148456743-noauth.png
paper_authors.fullname: Longguang Zhong
paper_authors.type: user
paper_authors.user: GGLS
paper_authors._id: 675146f4372c1020c21d2049
paper_authors.name: Tianyuan Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T12:23:43.208Z
paper_authors._id: 654c82ca47053d20c7986fe3
paper_authors.avatarUrl: /avatars/b6560247e9d6f662d36aa1c1589bb92e.svg
paper_authors.fullname: tianyuan shi
paper_authors.type: user
paper_authors.user: stycoo
paper_authors._id: 675146f4372c1020c21d204a
paper_authors.name: Xiaojun Quan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T12:23:35.922Z
paper_authors._id: 63b57d75bda8d44adf2ff3ff
paper_authors.avatarUrl: /avatars/8a387036758b2f7fc7d7529dea206669.svg
paper_authors.fullname: Xiaojun Quan
paper_authors.type: user
paper_authors.user: passerqxj
paper_title: Weighted-Reward Preference Optimization for Implicit Model Fusion
paper_summary: While fusing heterogeneous open-source LLMs with varying architectures and
  sizes can potentially integrate the strengths of different models, existing
  fusion methods face significant challenges, such as vocabulary alignment and
  merging distribution matrices. These procedures are not only complex but also
  prone to introducing noise and errors. In this paper, we propose an implicit
  fusion method, Weighted-Reward Preference Optimization (WRPO), which leverages
  preference optimization between the source LLMs and the target LLM to transfer
  their capabilities effectively. WRPO eliminates the need for vocabulary
  alignment and matrix fusion and can be efficiently scaled to accommodate
  various LLMs. To address distributional deviations between the source and
  target LLMs, WRPO introduces a progressive adaptation strategy that gradually
  shifts reliance on preferred examples from the target LLM to the source LLMs.
  Extensive experiments on the MT-Bench, AlpacaEval-2, and Arena-Hard benchmarks
  demonstrate that WRPO consistently outperforms existing knowledge fusion
  methods and various fine-tuning baselines. When applied to LLaMA3-8B-Instruct
  as the target model, WRPO achieves a length-controlled win rate of 55.9%
  against GPT-4-Preview-1106 on AlpacaEval-2 and a win rate of 46.2% against
  GPT-4-0314 on Arena-Hard. Our code is available at
  https://github.com/SLIT-AI/WRPO.
paper_ai_summary: A novel implicit fusion method, WRPO, effectively transfers capabilities between heterogeneous LLMs, improving performance on benchmarks without manual vocabulary alignment or distribution matrix fusion.
paper_ai_keywords: heterogeneous open-source LLMs Weighted-Reward Preference Optimization (WRPO) preference optimization vocabulary alignment distribution matrices progressive adaptation strategy MT-Bench AlpacaEval-2 Arena-Hard LLaMA3-8B-Instruct GPT-4-Preview-1106 GPT-4-0314