Through the Valley: Path to Effective Long CoT Training for Small
  Language Models
Long chain-of-thought (CoT) supervision has become a common strategy to
enhance reasoning in language models. While effective for large models, we
identify a phenomenon we call Long CoT Degradation, in which small language
models (SLMs; <=3B parameters) trained on limited long CoT data experience
significant performance deterioration. Through extensive experiments on the
Qwen2.5, LLaMA3 and Gemma3 families, we demonstrate that this degradation is
widespread across SLMs. In some settings, models trained on only 8k long CoT
examples lose up to 75% of their original performance before fine-tuning.
Strikingly, we further observe that for some particularly small models, even
training on 220k long CoT examples fails to recover or surpass their original
performance prior to fine-tuning. Our analysis attributes this effect to error
accumulation: while longer responses increase the capacity for multi-step
reasoning, they also amplify the risk of compounding mistakes. Furthermore, we
find that Long CoT Degradation may negatively impacts downstream reinforcement
learning (RL), although this can be alleviated by sufficiently scaled
supervised fine-tuning (SFT). Our findings challenge common assumptions about
the benefits of long CoT training for SLMs and offer practical guidance for
building more effective small-scale reasoning models.
684790cd3ec10bdd8ab4ddaa
Renjie Luo
extracted_pending
2025-06-10T01:56:30.026Z
66dfb6bac93721c02f75f37e
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/zzoX0tpkHCNsFGjtnATSt.png
Renjie
user
RogerLos
684790cd3ec10bdd8ab4ddab
Jiaxi Li
claimed_verified
2025-06-27T12:57:55.121Z
64c90edf0986bd6fa23f3c2d
/avatars/de778ea9ef17397f3839fc4d4bbf6c06.svg
Jiaxi
user
jxjessieli
684790cd3ec10bdd8ab4ddac
Chen Huang
claimed_verified
2025-06-10T09:19:43.446Z
65d7b983baa72790a1151923
/avatars/938531e84ca01a0c5a2a174057e3e9c5.svg
Chen Huang
user
Albus-Chen
684790cd3ec10bdd8ab4ddad
Wei Lu
Through the Valley: Path to Effective Long CoT Training for Small
  Language Models
Long chain-of-thought (CoT) supervision has become a common strategy to
enhance reasoning in language models. While effective for large models, we
identify a phenomenon we call Long CoT Degradation, in which small language
models (SLMs; <=3B parameters) trained on limited long CoT data experience
significant performance deterioration. Through extensive experiments on the
Qwen2.5, LLaMA3 and Gemma3 families, we demonstrate that this degradation is
widespread across SLMs. In some settings, models trained on only 8k long CoT
examples lose up to 75% of their original performance before fine-tuning.
Strikingly, we further observe that for some particularly small models, even
training on 220k long CoT examples fails to recover or surpass their original
performance prior to fine-tuning. Our analysis attributes this effect to error
accumulation: while longer responses increase the capacity for multi-step
reasoning, they also amplify the risk of compounding mistakes. Furthermore, we
find that Long CoT Degradation may negatively impacts downstream reinforcement
learning (RL), although this can be alleviated by sufficiently scaled
supervised fine-tuning (SFT). Our findings challenge common assumptions about
the benefits of long CoT training for SLMs and offer practical guidance for
building more effective small-scale reasoning models.
Small language models experience significant performance declines when trained on long chain-of-thought data due to error accumulation, impacting downstream reinforcement learning but potentially mitigated by extensive supervised fine-tuning.
Long chain-of-thought Long CoT Degradation small language models SLMs Qwen2.5 LLaMA3 Gemma3 error accumulation supervised fine-tuning SFT reinforcement learning