Mind the Gap: Bridging Thought Leap for Improved Chain-of-Thought Tuning
Large language models (LLMs) have achieved remarkable progress on
mathematical tasks through Chain-of-Thought (CoT) reasoning. However, existing
mathematical CoT datasets often suffer from Thought Leaps due to experts
omitting intermediate steps, which negatively impacts model learning and
generalization. We propose the CoT Thought Leap Bridge Task, which aims to
automatically detect leaps and generate missing intermediate reasoning steps to
restore the completeness and coherence of CoT. To facilitate this, we
constructed a specialized training dataset called ScaleQM+, based on the
structured ScaleQuestMath dataset, and trained CoT-Bridge to bridge thought
leaps. Through comprehensive experiments on mathematical reasoning benchmarks,
we demonstrate that models fine-tuned on bridged datasets consistently
outperform those trained on original datasets, with improvements of up to
+5.87% on NuminaMath. Our approach effectively enhances distilled data (+3.02%)
and provides better starting points for reinforcement learning (+3.1%),
functioning as a plug-and-play module compatible with existing optimization
techniques. Furthermore, CoT-Bridge demonstrate improved generalization to
out-of-domain logical reasoning tasks, confirming that enhancing reasoning
completeness yields broadly applicable benefits.
682f474c9ee0bb0cc953b885
Haolei Xu
claimed_verified
2025-05-26T08:17:23.733Z
6692aff88db712bad780f02a
/avatars/5dc4b1c27c70f6a64864711dbff4910f.svg
xhl
user
zjuxhl
682f474c9ee0bb0cc953b886
Yuchen Yan
claimed_verified
2025-05-26T08:17:21.363Z
64098738342c26884c792c93
https://cdn-avatars.huggingface.co/v1/production/uploads/64098738342c26884c792c93/SxBUd-wLrl-PjQsrVYJte.jpeg
Yuchen Yan
user
yanyc
682f474c9ee0bb0cc953b887
Yongliang Shen
claimed_verified
2025-05-26T08:17:18.955Z
5e1058e9fcf41d740b69966d
/avatars/ce74839ba871f2b54313a670a233ba82.svg
Yongliang Shen
user
tricktreat
682f474c9ee0bb0cc953b888
Wenqi Zhang
682f474c9ee0bb0cc953b889
Guiyang Hou
682f474c9ee0bb0cc953b88a
Shengpei Jiang
682f474c9ee0bb0cc953b88b
Kaitao Song
682f474c9ee0bb0cc953b88c
Weiming Lu
682f474c9ee0bb0cc953b88d
Jun Xiao
682f474c9ee0bb0cc953b88e
Yueting Zhuang
Mind the Gap: Bridging Thought Leap for Improved Chain-of-Thought Tuning
Large language models (LLMs) have achieved remarkable progress on
mathematical tasks through Chain-of-Thought (CoT) reasoning. However, existing
mathematical CoT datasets often suffer from Thought Leaps due to experts
omitting intermediate steps, which negatively impacts model learning and
generalization. We propose the CoT Thought Leap Bridge Task, which aims to
automatically detect leaps and generate missing intermediate reasoning steps to
restore the completeness and coherence of CoT. To facilitate this, we
constructed a specialized training dataset called ScaleQM+, based on the
structured ScaleQuestMath dataset, and trained CoT-Bridge to bridge thought
leaps. Through comprehensive experiments on mathematical reasoning benchmarks,
we demonstrate that models fine-tuned on bridged datasets consistently
outperform those trained on original datasets, with improvements of up to
+5.87% on NuminaMath. Our approach effectively enhances distilled data (+3.02%)
and provides better starting points for reinforcement learning (+3.1%),
functioning as a plug-and-play module compatible with existing optimization
techniques. Furthermore, CoT-Bridge demonstrate improved generalization to
out-of-domain logical reasoning tasks, confirming that enhancing reasoning
completeness yields broadly applicable benefits.
A model for detecting and generating missing intermediate steps in mathematical Chain-of-Thought reasoning improves performance and generalization on mathematical and logical reasoning tasks.
Chain-of-Thought reasoning thought leaps ScaleQM+ ScaleQuestMath NuminaMath distilled data reinforcement learning generalization logical reasoning tasks