title: BranchNorm: Robustly Scaling Extremely Deep Transformers
summary: Recently, DeepNorm scales Transformers into extremely deep (i.e., 1000
  layers) and reveals the promising potential of deep scaling. To stabilize the
  training of deep models, DeepNorm (Wang et al., 2022) attempts to constrain the
  model update to a constant value. Although applying such a constraint can
  benefit the early stage of model training, it may lead to undertrained models
  during the whole training procedure. In this paper, we propose BranchNorm,
  which dynamically rescales the non-residual branch of Transformer in accordance
  with the training period. BranchNorm not only theoretically stabilizes the
  training with smooth gradient norms at the early stage, but also encourages
  better convergence in the subsequent training stage. Experiment results on
  multiple translation tasks demonstrate that BranchNorm achieves a better
  trade-off between training stability and converge performance.
paper_authors._id: 64547a49a7ce0a8fde817551
paper_authors.name: Yijin Liu
paper_authors._id: 64547a49a7ce0a8fde817552
paper_authors.name: Xianfeng Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T13:22:45.218Z
paper_authors._id: 63c210b5a0ffa3857eb591ea
paper_authors.avatarUrl: /avatars/7d663ea8e8dce3fa4ebe86cea758ef74.svg
paper_authors.fullname: zengxianfeng
paper_authors.type: user
paper_authors.user: zreading
paper_authors._id: 64547a49a7ce0a8fde817553
paper_authors.name: Fandong Meng
paper_authors._id: 64547a49a7ce0a8fde817554
paper_authors.name: Jie Zhou
paper_title: BranchNorm: Robustly Scaling Extremely Deep Transformers
paper_summary: Recently, DeepNorm scales Transformers into extremely deep (i.e., 1000
  layers) and reveals the promising potential of deep scaling. To stabilize the
  training of deep models, DeepNorm (Wang et al., 2022) attempts to constrain the
  model update to a constant value. Although applying such a constraint can
  benefit the early stage of model training, it may lead to undertrained models
  during the whole training procedure. In this paper, we propose BranchNorm,
  which dynamically rescales the non-residual branch of Transformer in accordance
  with the training period. BranchNorm not only theoretically stabilizes the
  training with smooth gradient norms at the early stage, but also encourages
  better convergence in the subsequent training stage. Experiment results on
  multiple translation tasks demonstrate that BranchNorm achieves a better
  trade-off between training stability and converge performance.
paper_ai_summary: BranchNorm dynamically adjusts the scaling of the non-residual branch in Transformers to improve training stability and convergence in deep models.
paper_ai_keywords: DeepNorm Transformers deep scaling model update BranchNorm gradient norms training stability convergence