title: SLAB: Efficient Transformers with Simplified Linear Attention and
    Progressive Re-parameterized Batch Normalization
summary: Transformers have become foundational architectures for both natural language
  and computer vision tasks. However, the high computational cost makes it quite
  challenging to deploy on resource-constraint devices. This paper investigates
  the computational bottleneck modules of efficient transformer, i.e.,
  normalization layers and attention modules. LayerNorm is commonly used in
  transformer architectures but is not computational friendly due to statistic
  calculation during inference. However, replacing LayerNorm with more efficient
  BatchNorm in transformer often leads to inferior performance and collapse in
  training. To address this problem, we propose a novel method named PRepBN to
  progressively replace LayerNorm with re-parameterized BatchNorm in training.
  Moreover, we propose a simplified linear attention (SLA) module that is simple
  yet effective to achieve strong performance. Extensive experiments on image
  classification as well as object detection demonstrate the effectiveness of our
  proposed method. For example, our SLAB-Swin obtains 83.6% top-1 accuracy on
  ImageNet-1K with 16.2ms latency, which is 2.4ms less than that of
  Flatten-Swin with 0.1% higher accuracy. We also evaluated our method for
  language modeling task and obtain comparable performance and lower
  latency.Codes are publicly available at https://github.com/xinghaochen/SLAB and
  https://github.com/mindspore-lab/models/tree/master/research/huawei-noah/SLAB.
paper_authors._id: 664c0e14e60e8ad392878afd
paper_authors.name: Jialong Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T09:13:53.812Z
paper_authors._id: 6621d02fc50869c1e91522ef
paper_authors.avatarUrl: /avatars/db7e40095cbb694e5df148bd2cddad37.svg
paper_authors.fullname: guojialong
paper_authors.type: user
paper_authors.user: guojialong
paper_authors._id: 664c0e14e60e8ad392878afe
paper_authors.name: Xinghao Chen
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-05-21T02:59:33.252Z
paper_authors._id: 65878c774f9d2b955e025019
paper_authors.avatarUrl: /avatars/c039c29e131a8cba96b38048da94971b.svg
paper_authors.fullname: Xinghao Chen
paper_authors.type: user
paper_authors.user: xinghaochen
paper_authors._id: 664c0e14e60e8ad392878aff
paper_authors.name: Yehui Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T09:14:01.241Z
paper_authors._id: 64d5deb154bb9eb704f83122
paper_authors.avatarUrl: /avatars/86ce09bcca903319051e2307581a43f4.svg
paper_authors.fullname: Yehui Tang
paper_authors.type: user
paper_authors.user: tangyehui
paper_authors._id: 664c0e14e60e8ad392878b00
paper_authors.name: Yunhe Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T09:14:07.049Z
paper_authors._id: 658bdf7b925aadd43304f05c
paper_authors.avatarUrl: /avatars/64d9e9dea27c376c3bc7b2a54efc2a46.svg
paper_authors.fullname: Yunhe Wang
paper_authors.type: user
paper_authors.user: MightyCrane
paper_title: SLAB: Efficient Transformers with Simplified Linear Attention and
    Progressive Re-parameterized Batch Normalization
paper_summary: Transformers have become foundational architectures for both natural language
  and computer vision tasks. However, the high computational cost makes it quite
  challenging to deploy on resource-constraint devices. This paper investigates
  the computational bottleneck modules of efficient transformer, i.e.,
  normalization layers and attention modules. LayerNorm is commonly used in
  transformer architectures but is not computational friendly due to statistic
  calculation during inference. However, replacing LayerNorm with more efficient
  BatchNorm in transformer often leads to inferior performance and collapse in
  training. To address this problem, we propose a novel method named PRepBN to
  progressively replace LayerNorm with re-parameterized BatchNorm in training.
  Moreover, we propose a simplified linear attention (SLA) module that is simple
  yet effective to achieve strong performance. Extensive experiments on image
  classification as well as object detection demonstrate the effectiveness of our
  proposed method. For example, our SLAB-Swin obtains 83.6% top-1 accuracy on
  ImageNet-1K with 16.2ms latency, which is 2.4ms less than that of
  Flatten-Swin with 0.1% higher accuracy. We also evaluated our method for
  language modeling task and obtain comparable performance and lower
  latency.Codes are publicly available at https://github.com/xinghaochen/SLAB and
  https://github.com/mindspore-lab/models/tree/master/research/huawei-noah/SLAB.
paper_ai_keywords: normalization layers attention modules LayerNorm BatchNorm PRepBN simplified linear attention (SLA) image classification object detection ImageNet-1K FLatten-Swin language modeling