Scale-Distribution Decoupling: Enabling Stable and Effective Training of
  Large Language Models
Training stability is a persistent challenge in the pre-training of large
language models (LLMs), particularly for architectures such as Post-Norm
Transformers, which are prone to gradient explosion and dissipation. In this
paper, we propose Scale-Distribution Decoupling (SDD), a novel approach that
stabilizes training by explicitly decoupling the scale and distribution of the
weight matrix in fully-connected layers. SDD applies a normalization mechanism
to regulate activations and a learnable scaling vector to maintain
well-conditioned gradients, effectively preventing gradient explosion
and dissipation. This separation improves optimization efficiency,
particularly in deep networks, by ensuring stable gradient propagation.
Experimental results demonstrate that our method stabilizes training across
various LLM architectures and outperforms existing techniques in different
normalization configurations. Furthermore, the proposed method is lightweight
and compatible with existing frameworks, making it a practical solution for
stabilizing LLM training. Code is available at https://github.com/kaihemo/SDD.
67be86743ea16c7e9491ff16
Ya Wang
67be86743ea16c7e9491ff17
Zhijian Zhuo
admin_assigned
2025-02-26T08:58:10.556Z
66335b9c95c5b79ebf306f30
/avatars/d57784ee65cbef014360c9bac1ad4119.svg
Zhijian Zhuo
user
BryceZhuo
67be86743ea16c7e9491ff18
Yutao Zeng
claimed_verified
2025-02-26T08:25:55.016Z
6371128eafbe42caa5a5222b
/avatars/c3b2ab35949c38aa3dfb2657a1300aac.svg
Yutao Zeng
user
Taoer
67be86743ea16c7e9491ff19
Xun Zhou
admin_assigned
2025-02-26T08:52:26.974Z
62533db4a06ec75172eeabe7
/avatars/b1a4dad90afae5c00df97233a97777db.svg
xunzhou
user
xunzhou
67be86743ea16c7e9491ff1a
Jian Yang
67be86743ea16c7e9491ff1b
Xiaoqing Li
admin_assigned
2025-02-26T08:51:57.314Z
64648638351adef1a847a7ad
/avatars/7518e058fcf81ee81a06c96e996531e9.svg
Xiaoqing Li
user
LLIXQ
Scale-Distribution Decoupling: Enabling Stable and Effective Training of
  Large Language Models
Training stability is a persistent challenge in the pre-training of large
language models (LLMs), particularly for architectures such as Post-Norm
Transformers, which are prone to gradient explosion and dissipation. In this
paper, we propose Scale-Distribution Decoupling (SDD), a novel approach that
stabilizes training by explicitly decoupling the scale and distribution of the
weight matrix in fully-connected layers. SDD applies a normalization mechanism
to regulate activations and a learnable scaling vector to maintain
well-conditioned gradients, effectively preventing gradient explosion
and dissipation. This separation improves optimization efficiency,
particularly in deep networks, by ensuring stable gradient propagation.
Experimental results demonstrate that our method stabilizes training across
various LLM architectures and outperforms existing techniques in different
normalization configurations. Furthermore, the proposed method is lightweight
and compatible with existing frameworks, making it a practical solution for
stabilizing LLM training. Code is available at https://github.com/kaihemo/SDD.
Scale-Distribution Decoupling stabilizes training of large language models by decoupling weight matrix scale and distribution to prevent gradient issues.
Training stability Post-Norm Transformers Scale-Distribution Decoupling gradient explosion gradient dissipation normalization scaling vector well-conditioned gradients deep networks optimization efficiency