title: What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A
    Gradient Perspective
summary: What makes a difference in the post-training of LLMs? We investigate the
  training patterns of different layers in large language models (LLMs), through
  the lens of gradient, when training with different responses and initial
  models. We are specifically interested in how fast vs. slow thinking affects
  the layer-wise gradients, given the recent popularity of training LLMs on
  reasoning paths such as chain-of-thoughts (CoT) and process rewards. In our
  study, fast thinking without CoT leads to larger gradients and larger
  differences of gradients across layers than slow thinking (Detailed CoT),
  indicating the learning stability brought by the latter. Moreover, pre-trained
  LLMs are less affected by the instability of fast thinking than
  instruction-tuned LLMs. Additionally, we study whether the gradient patterns
  can reflect the correctness of responses when training different LLMs using
  slow vs. fast thinking paths. The results show that the gradients of slow
  thinking can distinguish correct and irrelevant reasoning paths. As a
  comparison, we conduct similar gradient analyses on non-reasoning knowledge
  learning tasks, on which, however, trivially increasing the response length
  does not lead to similar behaviors of slow thinking. Our study strengthens
  fundamental understandings of LLM training and sheds novel insights on its
  efficiency and stability, which pave the way towards building a generalizable
  System-2 agent. Our code, data, and gradient statistics can be found in:
  https://github.com/MingLiiii/Layer_Gradient.
paper_authors._id: 6724491cccb533a2edfce752
paper_authors.name: Ming Li
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-11-01T03:21:06.593Z
paper_authors._id: 65031d01cccc7b28a388c719
paper_authors.avatarUrl: /avatars/9d8c94b6ab8ad8b4faba3221b7e76053.svg
paper_authors.fullname: Ming Li
paper_authors.type: user
paper_authors.user: MingLiiii
paper_authors._id: 6724491cccb533a2edfce753
paper_authors.name: Yanhong Li
paper_authors._id: 6724491cccb533a2edfce754
paper_authors.name: Tianyi Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-03T14:22:09.660Z
paper_authors._id: 647f5af5b0e96764589f3b2a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
paper_authors.fullname: Tianyi Zhou
paper_authors.type: user
paper_authors.user: zhoutianyi
paper_title: What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A
    Gradient Perspective
paper_summary: What makes a difference in the post-training of LLMs? We investigate the
  training patterns of different layers in large language models (LLMs), through
  the lens of gradient, when training with different responses and initial
  models. We are specifically interested in how fast vs. slow thinking affects
  the layer-wise gradients, given the recent popularity of training LLMs on
  reasoning paths such as chain-of-thoughts (CoT) and process rewards. In our
  study, fast thinking without CoT leads to larger gradients and larger
  differences of gradients across layers than slow thinking (Detailed CoT),
  indicating the learning stability brought by the latter. Moreover, pre-trained
  LLMs are less affected by the instability of fast thinking than
  instruction-tuned LLMs. Additionally, we study whether the gradient patterns
  can reflect the correctness of responses when training different LLMs using
  slow vs. fast thinking paths. The results show that the gradients of slow
  thinking can distinguish correct and irrelevant reasoning paths. As a
  comparison, we conduct similar gradient analyses on non-reasoning knowledge
  learning tasks, on which, however, trivially increasing the response length
  does not lead to similar behaviors of slow thinking. Our study strengthens
  fundamental understandings of LLM training and sheds novel insights on its
  efficiency and stability, which pave the way towards building a generalizable
  System-2 agent. Our code, data, and gradient statistics can be found in:
  https://github.com/MingLiiii/Layer_Gradient.
paper_ai_summary: Investigating gradient patterns in LLMs during fast vs. slow training reveals that slow thinking (chain-of-thoughts) enhances learning stability and distinguishes correct reasoning paths.
paper_ai_keywords: large language models LLMs gradient chain-of-thoughts CoT learning stability reasoning paths post-training pre-trained LLMs instruction-tuned LLMs gradient patterns response correctness