TinyR1-32B-Preview: Boosting Accuracy with Branch-Merge Distillation
The challenge of reducing the size of Large Language Models (LLMs) while
maintaining their performance has gained significant attention. However,
existing methods, such as model distillation and transfer learning, often fail
to achieve high accuracy. To address this limitation, we introduce the
Branch-Merge distillation approach, which enhances model compression through
two phases: (1) the Branch Phase, where knowledge from a large teacher model is
selectively distilled into specialized student models via
domain-specific supervised fine-tuning (SFT); And (2) the Merge Phase, where
these student models are merged to enable cross-domain knowledge transfer and
improve generalization. We validate our distillation approach using DeepSeek-R1
as the teacher and DeepSeek-R1-Distill-Qwen-32B as the student. The resulting
merged model, TinyR1-32B-Preview, outperforms its counterpart
DeepSeek-R1-Distill-Qwen-32B across multiple benchmarks, including Mathematics
(+5.5 points), Coding (+4.4 points) and Science (+2.9 points), while achieving
near-equal performance to DeepSeek-R1 on AIME 2024. The Branch-Merge
distillation approach provides a scalable solution for creating smaller,
high-performing LLMs with reduced computational cost and time.
67ce5deedb623d45a95deb72
Lin Sun
claimed_verified
2025-03-10T08:00:47.601Z
632c30576bcb864974cc40a8
/avatars/96aa948ad1dd35d355e20b5765a2563a.svg
sunlin
user
lincharliesun
67ce5deedb623d45a95deb73
Guangxiang Zhao
admin_assigned
2025-03-10T10:13:04.137Z
67bc4f0bbcc20fb84d61f4fa
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/9f8n0ZCSP1AktsYh8DK5b.jpeg
Guangxiang Zhao
user
zhaoguangxiang
67ce5deedb623d45a95deb74
Xiaoqi Jian
admin_assigned
2025-03-10T10:13:10.864Z
64c7c3fb1c23fb9a2bc3950d
/avatars/84be70976902366201dd0349b1f2357c.svg
xiaoqi
user
xiaoqijiang
67ce5deedb623d45a95deb75
Yuhan Wu
admin_assigned
2025-03-10T10:13:31.059Z
660a71938695a785edd74d40
/avatars/41c96d1aec59d78d44145dc71d7b2e96.svg
wuyuhan
user
yuhanwuuu
67ce5deedb623d45a95deb76
Weihong Lin
admin_assigned
2025-03-10T10:13:37.096Z
675a69699e086bd6250a36ef
/avatars/95c72e3975d1a37f8655a2fe629746ec.svg
Weihong Lin
user
lwher1996
67ce5deedb623d45a95deb77
Yongfu Zhu
claimed_verified
2025-06-10T09:30:53.063Z
668cf9d4f7c08c3f9e12f97b
/avatars/396be7f144784aa2a1c35ec059a84997.svg
JosephZhu
user
JjjjjZzz
67ce5deedb623d45a95deb78
Change Jia
67ce5deedb623d45a95deb79
Linglin Zhang
admin_assigned
2025-03-10T10:13:58.854Z
664b0253156169f8724b8485
/avatars/4c1014b782ae298271911e7435cc5f6f.svg
Linglin zhang
user
LinglinZhang
67ce5deedb623d45a95deb7a
Jinzhu Wu
67ce5deedb623d45a95deb7b
Junfeng Ran
67ce5deedb623d45a95deb7c
Sai-er Hu
admin_assigned
2025-03-10T10:14:32.009Z
65a38f8b212d6aca9ac98573
https://cdn-avatars.huggingface.co/v1/production/uploads/65a38f8b212d6aca9ac98573/nN0XW8PujXkKy7pwQvlam.png
Sai-er Hu
user
Husserl233
67ce5deedb623d45a95deb7d
Zihan Jiang
admin_assigned
2025-03-10T10:16:14.012Z
665ee916b9e194a20c4769da
/avatars/cfb3ff26979bed1a0c65a6fa318af3ad.svg
Zihan Jiang
user
Jumbo0715
67ce5deedb623d45a95deb7e
Junting Zhou
admin_assigned
2025-03-10T10:14:47.371Z
65a374a59acab1998092a9bc
https://cdn-avatars.huggingface.co/v1/production/uploads/65a374a59acab1998092a9bc/M3s_7bSf9G-6b9nLg7N3Z.jpeg
Antonio
user
JuntingZhou
67ce5deedb623d45a95deb7f
Wenrui Liu
admin_assigned
2025-03-10T10:15:11.742Z
67160ae2ea37ca179f4f87fe
/avatars/7fc126422cfedac79e85fa6f12db8a61.svg
Wenrui Liu
user
CarenceLiu
67ce5deedb623d45a95deb80
Bin Cui
admin_assigned
2025-03-10T10:15:19.684Z
67b2795f0bd4ddcd84426bb4
/avatars/d4346ac5a0ebbaeb828d832cc6ca9f0b.svg
Bin Cui
user
lazybone128
67ce5deedb623d45a95deb81
Tong Yang
67ce5deedb623d45a95deb82
Xiangzheng Zhang
TinyR1-32B-Preview: Boosting Accuracy with Branch-Merge Distillation
The challenge of reducing the size of Large Language Models (LLMs) while
maintaining their performance has gained significant attention. However,
existing methods, such as model distillation and transfer learning, often fail
to achieve high accuracy. To address this limitation, we introduce the
Branch-Merge distillation approach, which enhances model compression through
two phases: (1) the Branch Phase, where knowledge from a large teacher model is
selectively distilled into specialized student models via
domain-specific supervised fine-tuning (SFT); And (2) the Merge Phase, where
these student models are merged to enable cross-domain knowledge transfer and
improve generalization. We validate our distillation approach using DeepSeek-R1
as the teacher and DeepSeek-R1-Distill-Qwen-32B as the student. The resulting
merged model, TinyR1-32B-Preview, outperforms its counterpart
DeepSeek-R1-Distill-Qwen-32B across multiple benchmarks, including Mathematics
(+5.5 points), Coding (+4.4 points) and Science (+2.9 points), while achieving
near-equal performance to DeepSeek-R1 on AIME 2024. The Branch-Merge
distillation approach provides a scalable solution for creating smaller,
high-performing LLMs with reduced computational cost and time.
The Branch-Merge distillation approach selectively transfers knowledge from a large teacher model to specialized student models and then merges them for improved generalization and reduced computational cost in LLMs.
Branch-Merge distillation DeepSeek-R1 DeepSeek-R1-Distill-Qwen-32B TinyR1-32B-Preview domain-specific supervised fine-tuning knowledge distillation transfer learning cross-domain knowledge transfer model compression