GPAS: Accelerating Convergence of LLM Pretraining via
  Gradient-Preserving Activation Scaling
Modern Large Language Models, such as the LLaMA, Qwen and DeepSeek series,
predominantly adopt the Pre-LayerNorm (Pre-LN) Transformer architecture. While
being stable during pretraining and scalable to large model sizes, Pre-LN
suffers from an exponential growth in activation variance across layers,
causing the residual path to dominate over sub-layer outputs and limiting the
learning capacity of deeper layers. To mitigate this issue, we propose
Gradient-Preserving Activation Scaling (GPAS), a simple technique that can be
used in combination with existing approaches. GPAS works by scaling down the
intermediate activations while keeping their gradients unchanged. This leaves
information in the activations intact, and avoids the gradient vanishing
problem associated with gradient downscaling. Extensive experiments across
various model sizes from 71M to 1B show that GPAS achieves consistent
performance gains. Beyond enhancing Pre-LN Transformers, GPAS also shows
promise in improving alternative architectures such as Sandwich-LN and
DeepNorm, demonstrating its versatility and potential for improving training
dynamics in a wide range of settings.
6862a0ce3b7fc785c7ca5dea
Tianhao Chen
6862a0ce3b7fc785c7ca5deb
Xin Xu
6862a0ce3b7fc785c7ca5dec
Zijing Liu
6862a0ce3b7fc785c7ca5ded
Pengxiang Li
6862a0ce3b7fc785c7ca5dee
Xinyuan Song
6862a0ce3b7fc785c7ca5def
Ajay Kumar Jaiswal
6862a0ce3b7fc785c7ca5df0
Fan Zhang
6862a0ce3b7fc785c7ca5df1
Jishan Hu
6862a0ce3b7fc785c7ca5df2
Yang Wang
6862a0ce3b7fc785c7ca5df3
Hao Chen
6862a0ce3b7fc785c7ca5df4
Shizhe Diao
6862a0ce3b7fc785c7ca5df5
Shiwei Liu
6862a0ce3b7fc785c7ca5df6
Yu Li
6862a0ce3b7fc785c7ca5df7
Yin Lu
6862a0ce3b7fc785c7ca5df8
Can Yang
GPAS: Accelerating Convergence of LLM Pretraining via
  Gradient-Preserving Activation Scaling
Modern Large Language Models, such as the LLaMA, Qwen and DeepSeek series,
predominantly adopt the Pre-LayerNorm (Pre-LN) Transformer architecture. While
being stable during pretraining and scalable to large model sizes, Pre-LN
suffers from an exponential growth in activation variance across layers,
causing the residual path to dominate over sub-layer outputs and limiting the
learning capacity of deeper layers. To mitigate this issue, we propose
Gradient-Preserving Activation Scaling (GPAS), a simple technique that can be
used in combination with existing approaches. GPAS works by scaling down the
intermediate activations while keeping their gradients unchanged. This leaves
information in the activations intact, and avoids the gradient vanishing
problem associated with gradient downscaling. Extensive experiments across
various model sizes from 71M to 1B show that GPAS achieves consistent
performance gains. Beyond enhancing Pre-LN Transformers, GPAS also shows
promise in improving alternative architectures such as Sandwich-LN and
DeepNorm, demonstrating its versatility and potential for improving training
dynamics in a wide range of settings.
Gradient-Preserving Activation Scaling (GPAS) mitigates activation variance issues in Pre-LayerNorm Transformers and enhances training dynamics across different architectures.
Pre-LayerNorm Transformer architecture activation variance residual path learning capacity Gradient-Preserving Activation Scaling GPAS gradient vanishing problem Sandwich-LN DeepNorm training dynamics