HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid
  Normalization
Transformers have become the de facto architecture for a wide range of
machine learning tasks, particularly in large language models (LLMs). Despite
their remarkable performance, challenges remain in training deep transformer
networks, especially regarding the location of layer normalization. While
Pre-Norm structures facilitate easier training due to their more prominent
identity path, they often yield suboptimal performance compared to Post-Norm.
In this paper, we propose HybridNorm, a straightforward yet
effective hybrid normalization strategy that integrates the advantages of both
Pre-Norm and Post-Norm approaches. Specifically, HybridNorm employs QKV
normalization within the attention mechanism and Post-Norm in the feed-forward
network (FFN) of each transformer block. This design not only stabilizes
training but also enhances performance, particularly in the context of LLMs.
Comprehensive experiments in both dense and sparse architectures show that
HybridNorm consistently outperforms both Pre-Norm and Post-Norm approaches,
achieving state-of-the-art results across various benchmarks. These findings
highlight the potential of HybridNorm as a more stable and effective technique
for improving the training and performance of deep transformer models. %Code
will be made publicly available. Code is available at
https://github.com/BryceZhuo/HybridNorm.
67ca69063a6e3e8656bcc1d2
Zhijian Zhuo
admin_assigned
2025-03-07T09:27:08.401Z
66335b9c95c5b79ebf306f30
/avatars/d57784ee65cbef014360c9bac1ad4119.svg
Zhijian Zhuo
user
BryceZhuo
67ca69063a6e3e8656bcc1d3
Yutao Zeng
claimed_verified
2025-03-07T09:09:43.334Z
6371128eafbe42caa5a5222b
/avatars/c3b2ab35949c38aa3dfb2657a1300aac.svg
Yutao Zeng
user
Taoer
67ca69063a6e3e8656bcc1d4
Ya Wang
67ca69063a6e3e8656bcc1d5
Sijun Zhang
67ca69063a6e3e8656bcc1d6
Jian Yang
67ca69063a6e3e8656bcc1d7
Xiaoqing Li
admin_assigned
2025-03-07T09:26:45.058Z
64648638351adef1a847a7ad
/avatars/7518e058fcf81ee81a06c96e996531e9.svg
Xiaoqing Li
user
LLIXQ
67ca69063a6e3e8656bcc1d8
Xun Zhou
67ca69063a6e3e8656bcc1d9
Jinwen Ma
admin_assigned
2025-03-07T09:27:23.639Z
663a684d08778abaf0556df8
/avatars/d95b517df5b80a8b42bac2b171604742.svg
Majinwen
user
Breeze0417
HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid
  Normalization
Transformers have become the de facto architecture for a wide range of
machine learning tasks, particularly in large language models (LLMs). Despite
their remarkable performance, challenges remain in training deep transformer
networks, especially regarding the location of layer normalization. While
Pre-Norm structures facilitate easier training due to their more prominent
identity path, they often yield suboptimal performance compared to Post-Norm.
In this paper, we propose HybridNorm, a straightforward yet
effective hybrid normalization strategy that integrates the advantages of both
Pre-Norm and Post-Norm approaches. Specifically, HybridNorm employs QKV
normalization within the attention mechanism and Post-Norm in the feed-forward
network (FFN) of each transformer block. This design not only stabilizes
training but also enhances performance, particularly in the context of LLMs.
Comprehensive experiments in both dense and sparse architectures show that
HybridNorm consistently outperforms both Pre-Norm and Post-Norm approaches,
achieving state-of-the-art results across various benchmarks. These findings
highlight the potential of HybridNorm as a more stable and effective technique
for improving the training and performance of deep transformer models. %Code
will be made publicly available. Code is available at
https://github.com/BryceZhuo/HybridNorm.
HybridNorm, combining pre-normalization and post-normalization, enhances training stability and performance of deep transformer models, especially in LLMs.
transformers layer normalization Pre-Norm Post-Norm HybridNorm attention mechanism feed-forward network dense architectures sparse architectures state-of-the-art results