Small Models Struggle to Learn from Strong Reasoners
Large language models (LLMs) excel in complex reasoning tasks, and distilling
their reasoning capabilities into smaller models has shown promise. However, we
uncover an interesting phenomenon, which we term the Small Model Learnability
Gap: small models (leq3B parameters) do not consistently benefit from long
chain-of-thought (CoT) reasoning or distillation from larger models. Instead,
they perform better when fine-tuned on shorter, simpler reasoning chains that
better align with their intrinsic learning capacity. To address this, we
propose Mix Distillation, a simple yet effective strategy that balances
reasoning complexity by combining long and short CoT examples or reasoning from
both larger and smaller models. Our experiments demonstrate that Mix
Distillation significantly improves small model reasoning performance compared
to training on either data alone. These findings highlight the limitations of
direct strong model distillation and underscore the importance of adapting
reasoning complexity for effective reasoning capability transfer.
67b4d05a9f8a8ab661450397
Yuetai Li
67b4d05a9f8a8ab661450398
Xiang Yue
claimed_verified
2025-07-02T16:08:22.034Z
6230d750d93e84e233882dbc
https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
Xiang Yue
user
yuexiang96
67b4d05a9f8a8ab661450399
Zhangchen Xu
claimed_verified
2025-02-20T09:37:32.715Z
653df1323479e9ebbe3eb6cc
https://cdn-avatars.huggingface.co/v1/production/uploads/653df1323479e9ebbe3eb6cc/K_g-r1iMRNKj99LXPuYF3.jpeg
Zhangchen Xu
user
zhangchenxu
67b4d05a9f8a8ab66145039a
Fengqing Jiang
admin_assigned
2025-02-20T16:09:45.083Z
6531e1021dd8ebbdc1a6fd8e
https://cdn-avatars.huggingface.co/v1/production/uploads/6531e1021dd8ebbdc1a6fd8e/lIcl7zCPtzRsfiUh6uY1o.jpeg
Fengqing Jiang
user
fqjiang
67b4d05a9f8a8ab66145039b
Luyao Niu
admin_assigned
2025-02-20T16:09:50.890Z
666dfd4770f5a2cb4aefd12f
/avatars/fa0e0dbc203a21e58dda8fdb4cbc67ad.svg
Luyao Niu
user
LNIU
67b4d05a9f8a8ab66145039c
Bill Yuchen Lin
admin_assigned
2025-02-20T16:09:56.614Z
607f666a4ad99100d63ce35c
https://cdn-avatars.huggingface.co/v1/production/uploads/607f666a4ad99100d63ce35c/QxhxnvfeV6efkxwUFHwjI.png
Bill Yuchen Lin
user
yuchenlin
67b4d05a9f8a8ab66145039d
Bhaskar Ramasubramanian
67b4d05a9f8a8ab66145039e
Radha Poovendran
Small Models Struggle to Learn from Strong Reasoners
Large language models (LLMs) excel in complex reasoning tasks, and distilling
their reasoning capabilities into smaller models has shown promise. However, we
uncover an interesting phenomenon, which we term the Small Model Learnability
Gap: small models (leq3B parameters) do not consistently benefit from long
chain-of-thought (CoT) reasoning or distillation from larger models. Instead,
they perform better when fine-tuned on shorter, simpler reasoning chains that
better align with their intrinsic learning capacity. To address this, we
propose Mix Distillation, a simple yet effective strategy that balances
reasoning complexity by combining long and short CoT examples or reasoning from
both larger and smaller models. Our experiments demonstrate that Mix
Distillation significantly improves small model reasoning performance compared
to training on either data alone. These findings highlight the limitations of
direct strong model distillation and underscore the importance of adapting
reasoning complexity for effective reasoning capability transfer.
Mix Distillation improves small model reasoning by balancing long and short chain-of-thought examples, addressing the Small Model Learnability Gap.
large language models (LLMs) chain-of-thought (CoT) distillation small models parameter-efficient fine-tuning Mix Distillation