title: FuseChat-3.0: Preference Optimization Meets Heterogeneous Model Fusion
summary: We introduce FuseChat-3.0, a suite of large language models (LLMs) developed
  by integrating the strengths of heterogeneous source LLMs into more compact
  target LLMs. Our source models include the powerful Gemma-2-27B-it,
  Mistral-Large-Instruct-2407, Qwen-2.5-72B-Instruct, and Llama-3.1-70B-Instruct.
  For target models, we focus on three widely-used smaller
  variants-Llama-3.1-8B-Instruct, Gemma-2-9B-it, and Qwen-2.5-7B-Instruct-along
  with two ultra-compact options, Llama-3.2-3B-Instruct and
  Llama-3.2-1B-Instruct. To leverage the diverse capabilities of these source
  models, we develop a specialized data construction protocol tailored to various
  tasks and domains. The FuseChat-3.0 training pipeline consists of two key
  stages: (1) supervised fine-tuning (SFT) to align the target and source model
  distributions, and (2) Direct Preference Optimization (DPO) to apply
  preferences from multiple source LLMs to fine-tune the target model. The
  resulting FuseChat-3.0 models exhibit significant performance gains across
  tasks such as instruction following, general knowledge, mathematics, and
  coding. As illustrated in Figure 1, using Llama-3.1-8B-Instruct as the target
  model, our fusion approach achieves an average improvement of 6.8 points across
  14 benchmarks. Moreover, it demonstrates remarkable gains of 37.1 points and
  30.1 points on the instruction-following benchmarks AlpacaEval-2 and
  Arena-Hard, respectively. Our code, models, and datasets are available at
  https://github.com/SLIT-AI/FuseChat-3.0.
paper_authors._id: 67ca64cdd153739fa9b9dbe6
paper_authors.name: Ziyi Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-07T09:09:46.287Z
paper_authors._id: 64c9b0f28d2d187c24d1e6c1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/1CPnAaB3gsupdpiNWaoDc.png
paper_authors.fullname: ZiYi Yang
paper_authors.type: user
paper_authors.user: AALF
paper_authors._id: 67ca64cdd153739fa9b9dbe7
paper_authors.name: Fanqi Wan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:33:34.691Z
paper_authors._id: 62ecbffd99112e99c5f7fded
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62ecbffd99112e99c5f7fded/U6iXAJbpm2vaC5qksEPiH.png
paper_authors.fullname: Fanqi Wan
paper_authors.type: user
paper_authors.user: Wanfq
paper_authors._id: 67ca64cdd153739fa9b9dbe8
paper_authors.name: Longguang Zhong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:33:41.092Z
paper_authors._id: 62b6d20416ff90e6198301b6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656148456743-noauth.png
paper_authors.fullname: Longguang Zhong
paper_authors.type: user
paper_authors.user: GGLS
paper_authors._id: 67ca64cdd153739fa9b9dbe9
paper_authors.name: Canbin Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:33:47.458Z
paper_authors._id: 63b93e6921add32ac6190b5c
paper_authors.avatarUrl: /avatars/7aa6a94d48e7f7c2bc56f8734d6c4e3d.svg
paper_authors.fullname: Canbin Huang
paper_authors.type: user
paper_authors.user: OnewayLab
paper_authors._id: 67ca64cdd153739fa9b9dbea
paper_authors.name: Guosheng Liang
paper_authors._id: 67ca64cdd153739fa9b9dbeb
paper_authors.name: Xiaojun Quan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:34:05.521Z
paper_authors._id: 63b57d75bda8d44adf2ff3ff
paper_authors.avatarUrl: /avatars/8a387036758b2f7fc7d7529dea206669.svg
paper_authors.fullname: Xiaojun Quan
paper_authors.type: user
paper_authors.user: passerqxj
paper_title: FuseChat-3.0: Preference Optimization Meets Heterogeneous Model Fusion
paper_summary: We introduce FuseChat-3.0, a suite of large language models (LLMs) developed
  by integrating the strengths of heterogeneous source LLMs into more compact
  target LLMs. Our source models include the powerful Gemma-2-27B-it,
  Mistral-Large-Instruct-2407, Qwen-2.5-72B-Instruct, and Llama-3.1-70B-Instruct.
  For target models, we focus on three widely-used smaller
  variants-Llama-3.1-8B-Instruct, Gemma-2-9B-it, and Qwen-2.5-7B-Instruct-along
  with two ultra-compact options, Llama-3.2-3B-Instruct and
  Llama-3.2-1B-Instruct. To leverage the diverse capabilities of these source
  models, we develop a specialized data construction protocol tailored to various
  tasks and domains. The FuseChat-3.0 training pipeline consists of two key
  stages: (1) supervised fine-tuning (SFT) to align the target and source model
  distributions, and (2) Direct Preference Optimization (DPO) to apply
  preferences from multiple source LLMs to fine-tune the target model. The
  resulting FuseChat-3.0 models exhibit significant performance gains across
  tasks such as instruction following, general knowledge, mathematics, and
  coding. As illustrated in Figure 1, using Llama-3.1-8B-Instruct as the target
  model, our fusion approach achieves an average improvement of 6.8 points across
  14 benchmarks. Moreover, it demonstrates remarkable gains of 37.1 points and
  30.1 points on the instruction-following benchmarks AlpacaEval-2 and
  Arena-Hard, respectively. Our code, models, and datasets are available at
  https://github.com/SLIT-AI/FuseChat-3.0.
paper_ai_keywords: supervised fine-tuning (SFT) Direct Preference Optimization (DPO)