title: Scale-Distribution Decoupling: Enabling Stable and Effective Training of
    Large Language Models
summary: Training stability is a persistent challenge in the pre-training of large
  language models (LLMs), particularly for architectures such as Post-Norm
  Transformers, which are prone to gradient explosion and dissipation. In this
  paper, we propose Scale-Distribution Decoupling (SDD), a novel approach that
  stabilizes training by explicitly decoupling the scale and distribution of the
  weight matrix in fully-connected layers. SDD applies a normalization mechanism
  to regulate activations and a learnable scaling vector to maintain
  well-conditioned gradients, effectively preventing gradient explosion
  and dissipation. This separation improves optimization efficiency,
  particularly in deep networks, by ensuring stable gradient propagation.
  Experimental results demonstrate that our method stabilizes training across
  various LLM architectures and outperforms existing techniques in different
  normalization configurations. Furthermore, the proposed method is lightweight
  and compatible with existing frameworks, making it a practical solution for
  stabilizing LLM training. Code is available at https://github.com/kaihemo/SDD.
paper_authors._id: 67be86743ea16c7e9491ff16
paper_authors.name: Ya Wang
paper_authors._id: 67be86743ea16c7e9491ff17
paper_authors.name: Zhijian Zhuo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-26T08:58:10.556Z
paper_authors._id: 66335b9c95c5b79ebf306f30
paper_authors.avatarUrl: /avatars/d57784ee65cbef014360c9bac1ad4119.svg
paper_authors.fullname: Zhijian Zhuo
paper_authors.type: user
paper_authors.user: BryceZhuo
paper_authors._id: 67be86743ea16c7e9491ff18
paper_authors.name: Yutao Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-26T08:25:55.016Z
paper_authors._id: 6371128eafbe42caa5a5222b
paper_authors.avatarUrl: /avatars/c3b2ab35949c38aa3dfb2657a1300aac.svg
paper_authors.fullname: Yutao Zeng
paper_authors.type: user
paper_authors.user: Taoer
paper_authors._id: 67be86743ea16c7e9491ff19
paper_authors.name: Xun Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-26T08:52:26.974Z
paper_authors._id: 62533db4a06ec75172eeabe7
paper_authors.avatarUrl: /avatars/b1a4dad90afae5c00df97233a97777db.svg
paper_authors.fullname: xunzhou
paper_authors.type: user
paper_authors.user: xunzhou
paper_authors._id: 67be86743ea16c7e9491ff1a
paper_authors.name: Jian Yang
paper_authors._id: 67be86743ea16c7e9491ff1b
paper_authors.name: Xiaoqing Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-26T08:51:57.314Z
paper_authors._id: 64648638351adef1a847a7ad
paper_authors.avatarUrl: /avatars/7518e058fcf81ee81a06c96e996531e9.svg
paper_authors.fullname: Xiaoqing Li
paper_authors.type: user
paper_authors.user: LLIXQ
paper_title: Scale-Distribution Decoupling: Enabling Stable and Effective Training of
    Large Language Models
paper_summary: Training stability is a persistent challenge in the pre-training of large
  language models (LLMs), particularly for architectures such as Post-Norm
  Transformers, which are prone to gradient explosion and dissipation. In this
  paper, we propose Scale-Distribution Decoupling (SDD), a novel approach that
  stabilizes training by explicitly decoupling the scale and distribution of the
  weight matrix in fully-connected layers. SDD applies a normalization mechanism
  to regulate activations and a learnable scaling vector to maintain
  well-conditioned gradients, effectively preventing gradient explosion
  and dissipation. This separation improves optimization efficiency,
  particularly in deep networks, by ensuring stable gradient propagation.
  Experimental results demonstrate that our method stabilizes training across
  various LLM architectures and outperforms existing techniques in different
  normalization configurations. Furthermore, the proposed method is lightweight
  and compatible with existing frameworks, making it a practical solution for
  stabilizing LLM training. Code is available at https://github.com/kaihemo/SDD.
paper_ai_summary: Scale-Distribution Decoupling stabilizes training of large language models by decoupling weight matrix scale and distribution to prevent gradient issues.
paper_ai_keywords: Training stability Post-Norm Transformers Scale-Distribution Decoupling gradient explosion gradient dissipation normalization scaling vector well-conditioned gradients deep networks optimization efficiency