title: HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid
    Normalization
summary: Transformers have become the de facto architecture for a wide range of
  machine learning tasks, particularly in large language models (LLMs). Despite
  their remarkable performance, challenges remain in training deep transformer
  networks, especially regarding the location of layer normalization. While
  Pre-Norm structures facilitate easier training due to their more prominent
  identity path, they often yield suboptimal performance compared to Post-Norm.
  In this paper, we propose HybridNorm, a straightforward yet
  effective hybrid normalization strategy that integrates the advantages of both
  Pre-Norm and Post-Norm approaches. Specifically, HybridNorm employs QKV
  normalization within the attention mechanism and Post-Norm in the feed-forward
  network (FFN) of each transformer block. This design not only stabilizes
  training but also enhances performance, particularly in the context of LLMs.
  Comprehensive experiments in both dense and sparse architectures show that
  HybridNorm consistently outperforms both Pre-Norm and Post-Norm approaches,
  achieving state-of-the-art results across various benchmarks. These findings
  highlight the potential of HybridNorm as a more stable and effective technique
  for improving the training and performance of deep transformer models. %Code
  will be made publicly available. Code is available at
  https://github.com/BryceZhuo/HybridNorm.
paper_authors._id: 67ca69063a6e3e8656bcc1d2
paper_authors.name: Zhijian Zhuo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:27:08.401Z
paper_authors._id: 66335b9c95c5b79ebf306f30
paper_authors.avatarUrl: /avatars/d57784ee65cbef014360c9bac1ad4119.svg
paper_authors.fullname: Zhijian Zhuo
paper_authors.type: user
paper_authors.user: BryceZhuo
paper_authors._id: 67ca69063a6e3e8656bcc1d3
paper_authors.name: Yutao Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-07T09:09:43.334Z
paper_authors._id: 6371128eafbe42caa5a5222b
paper_authors.avatarUrl: /avatars/c3b2ab35949c38aa3dfb2657a1300aac.svg
paper_authors.fullname: Yutao Zeng
paper_authors.type: user
paper_authors.user: Taoer
paper_authors._id: 67ca69063a6e3e8656bcc1d4
paper_authors.name: Ya Wang
paper_authors._id: 67ca69063a6e3e8656bcc1d5
paper_authors.name: Sijun Zhang
paper_authors._id: 67ca69063a6e3e8656bcc1d6
paper_authors.name: Jian Yang
paper_authors._id: 67ca69063a6e3e8656bcc1d7
paper_authors.name: Xiaoqing Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:26:45.058Z
paper_authors._id: 64648638351adef1a847a7ad
paper_authors.avatarUrl: /avatars/7518e058fcf81ee81a06c96e996531e9.svg
paper_authors.fullname: Xiaoqing Li
paper_authors.type: user
paper_authors.user: LLIXQ
paper_authors._id: 67ca69063a6e3e8656bcc1d8
paper_authors.name: Xun Zhou
paper_authors._id: 67ca69063a6e3e8656bcc1d9
paper_authors.name: Jinwen Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:27:23.639Z
paper_authors._id: 663a684d08778abaf0556df8
paper_authors.avatarUrl: /avatars/d95b517df5b80a8b42bac2b171604742.svg
paper_authors.fullname: Majinwen
paper_authors.type: user
paper_authors.user: Breeze0417
paper_title: HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid
    Normalization
paper_summary: Transformers have become the de facto architecture for a wide range of
  machine learning tasks, particularly in large language models (LLMs). Despite
  their remarkable performance, challenges remain in training deep transformer
  networks, especially regarding the location of layer normalization. While
  Pre-Norm structures facilitate easier training due to their more prominent
  identity path, they often yield suboptimal performance compared to Post-Norm.
  In this paper, we propose HybridNorm, a straightforward yet
  effective hybrid normalization strategy that integrates the advantages of both
  Pre-Norm and Post-Norm approaches. Specifically, HybridNorm employs QKV
  normalization within the attention mechanism and Post-Norm in the feed-forward
  network (FFN) of each transformer block. This design not only stabilizes
  training but also enhances performance, particularly in the context of LLMs.
  Comprehensive experiments in both dense and sparse architectures show that
  HybridNorm consistently outperforms both Pre-Norm and Post-Norm approaches,
  achieving state-of-the-art results across various benchmarks. These findings
  highlight the potential of HybridNorm as a more stable and effective technique
  for improving the training and performance of deep transformer models. %Code
  will be made publicly available. Code is available at
  https://github.com/BryceZhuo/HybridNorm.
paper_ai_summary: HybridNorm, combining pre-normalization and post-normalization, enhances training stability and performance of deep transformer models, especially in LLMs.
paper_ai_keywords: transformers layer normalization Pre-Norm Post-Norm HybridNorm attention mechanism feed-forward network dense architectures sparse architectures state-of-the-art results