FuseChat-3.0: Preference Optimization Meets Heterogeneous Model Fusion
We introduce FuseChat-3.0, a suite of large language models (LLMs) developed
by integrating the strengths of heterogeneous source LLMs into more compact
target LLMs. Our source models include the powerful Gemma-2-27B-it,
Mistral-Large-Instruct-2407, Qwen-2.5-72B-Instruct, and Llama-3.1-70B-Instruct.
For target models, we focus on three widely-used smaller
variants-Llama-3.1-8B-Instruct, Gemma-2-9B-it, and Qwen-2.5-7B-Instruct-along
with two ultra-compact options, Llama-3.2-3B-Instruct and
Llama-3.2-1B-Instruct. To leverage the diverse capabilities of these source
models, we develop a specialized data construction protocol tailored to various
tasks and domains. The FuseChat-3.0 training pipeline consists of two key
stages: (1) supervised fine-tuning (SFT) to align the target and source model
distributions, and (2) Direct Preference Optimization (DPO) to apply
preferences from multiple source LLMs to fine-tune the target model. The
resulting FuseChat-3.0 models exhibit significant performance gains across
tasks such as instruction following, general knowledge, mathematics, and
coding. As illustrated in Figure 1, using Llama-3.1-8B-Instruct as the target
model, our fusion approach achieves an average improvement of 6.8 points across
14 benchmarks. Moreover, it demonstrates remarkable gains of 37.1 points and
30.1 points on the instruction-following benchmarks AlpacaEval-2 and
Arena-Hard, respectively. Our code, models, and datasets are available at
https://github.com/SLIT-AI/FuseChat-3.0.
67ca64cdd153739fa9b9dbe6
Ziyi Yang
claimed_verified
2025-03-07T09:09:46.287Z
64c9b0f28d2d187c24d1e6c1
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/1CPnAaB3gsupdpiNWaoDc.png
ZiYi Yang
user
AALF
67ca64cdd153739fa9b9dbe7
Fanqi Wan
admin_assigned
2025-03-07T09:33:34.691Z
62ecbffd99112e99c5f7fded
https://cdn-avatars.huggingface.co/v1/production/uploads/62ecbffd99112e99c5f7fded/U6iXAJbpm2vaC5qksEPiH.png
Fanqi Wan
user
Wanfq
67ca64cdd153739fa9b9dbe8
Longguang Zhong
admin_assigned
2025-03-07T09:33:41.092Z
62b6d20416ff90e6198301b6
https://cdn-avatars.huggingface.co/v1/production/uploads/1656148456743-noauth.png
Longguang Zhong
user
GGLS
67ca64cdd153739fa9b9dbe9
Canbin Huang
admin_assigned
2025-03-07T09:33:47.458Z
63b93e6921add32ac6190b5c
/avatars/7aa6a94d48e7f7c2bc56f8734d6c4e3d.svg
Canbin Huang
user
OnewayLab
67ca64cdd153739fa9b9dbea
Guosheng Liang
67ca64cdd153739fa9b9dbeb
Xiaojun Quan
admin_assigned
2025-03-07T09:34:05.521Z
63b57d75bda8d44adf2ff3ff
/avatars/8a387036758b2f7fc7d7529dea206669.svg
Xiaojun Quan
user
passerqxj
FuseChat-3.0: Preference Optimization Meets Heterogeneous Model Fusion
We introduce FuseChat-3.0, a suite of large language models (LLMs) developed
by integrating the strengths of heterogeneous source LLMs into more compact
target LLMs. Our source models include the powerful Gemma-2-27B-it,
Mistral-Large-Instruct-2407, Qwen-2.5-72B-Instruct, and Llama-3.1-70B-Instruct.
For target models, we focus on three widely-used smaller
variants-Llama-3.1-8B-Instruct, Gemma-2-9B-it, and Qwen-2.5-7B-Instruct-along
with two ultra-compact options, Llama-3.2-3B-Instruct and
Llama-3.2-1B-Instruct. To leverage the diverse capabilities of these source
models, we develop a specialized data construction protocol tailored to various
tasks and domains. The FuseChat-3.0 training pipeline consists of two key
stages: (1) supervised fine-tuning (SFT) to align the target and source model
distributions, and (2) Direct Preference Optimization (DPO) to apply
preferences from multiple source LLMs to fine-tune the target model. The
resulting FuseChat-3.0 models exhibit significant performance gains across
tasks such as instruction following, general knowledge, mathematics, and
coding. As illustrated in Figure 1, using Llama-3.1-8B-Instruct as the target
model, our fusion approach achieves an average improvement of 6.8 points across
14 benchmarks. Moreover, it demonstrates remarkable gains of 37.1 points and
30.1 points on the instruction-following benchmarks AlpacaEval-2 and
Arena-Hard, respectively. Our code, models, and datasets are available at
https://github.com/SLIT-AI/FuseChat-3.0.
FuseChat-3.0, a suite of large language models, integrates diverse source models into smaller targets using supervised fine-tuning and Direct Preference Optimization, achieving significant performance improvements across various tasks.
large language models heterogeneous source LLMs compact target LLMs Gemma-2-27B-it Mistral-Large-Instruct-2407 Qwen-2.5-72B-Instruct Llama-3.1-70B-Instruct supervised fine-tuning Direct Preference Optimization instruction following general knowledge mathematics coding AlpacaEval-2 Arena-Hard