title: EfficientViT: Memory Efficient Vision Transformer with Cascaded Group
    Attention
summary: Vision transformers have shown great success due to their high model
  capabilities. However, their remarkable performance is accompanied by heavy
  computation costs, which makes them unsuitable for real-time applications. In
  this paper, we propose a family of high-speed vision transformers named
  EfficientViT. We find that the speed of existing transformer models is commonly
  bounded by memory inefficient operations, especially the tensor reshaping and
  element-wise functions in MHSA. Therefore, we design a new building block with
  a sandwich layout, i.e., using a single memory-bound MHSA between efficient FFN
  layers, which improves memory efficiency while enhancing channel communication.
  Moreover, we discover that the attention maps share high similarities across
  heads, leading to computational redundancy. To address this, we present a
  cascaded group attention module feeding attention heads with different splits
  of the full feature, which not only saves computation cost but also improves
  attention diversity. Comprehensive experiments demonstrate EfficientViT
  outperforms existing efficient models, striking a good trade-off between speed
  and accuracy. For instance, our EfficientViT-M5 surpasses MobileNetV3-Large by
  1.9% in accuracy, while getting 40.4% and 45.2% higher throughput on Nvidia
  V100 GPU and Intel Xeon CPU, respectively. Compared to the recent efficient
  model MobileViT-XXS, EfficientViT-M2 achieves 1.8% superior accuracy, while
  running 5.8x/3.7x faster on the GPU/CPU, and 7.4x faster when converted to ONNX
  format. Code and models are available at
  https://github.com/microsoft/Cream/tree/main/EfficientViT.
paper_authors._id: 645dc520ce72244df7b52bc3
paper_authors.name: Xinyu Liu
paper_authors._id: 645dc520ce72244df7b52bc4
paper_authors.name: Houwen Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T12:57:40.098Z
paper_authors._id: 631952b6f18d0b5d999ca397
paper_authors.avatarUrl: /avatars/eb87dfc142a7602f8fb888f7b7b60d38.svg
paper_authors.fullname: Peng
paper_authors.type: user
paper_authors.user: Houwen
paper_authors._id: 645dc520ce72244df7b52bc5
paper_authors.name: Ningxin Zheng
paper_authors._id: 645dc520ce72244df7b52bc6
paper_authors.name: Yuqing Yang
paper_authors._id: 645dc520ce72244df7b52bc7
paper_authors.name: Han Hu
paper_authors._id: 645dc520ce72244df7b52bc8
paper_authors.name: Yixuan Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T12:59:27.708Z
paper_authors._id: 640fdfc9f2d7c41a1ea112ef
paper_authors.avatarUrl: /avatars/780328c388ac4bc9acdf063e7833259d.svg
paper_authors.fullname: yxyuan
paper_authors.type: user
paper_authors.user: yixuanyuan
paper_title: EfficientViT: Memory Efficient Vision Transformer with Cascaded Group
    Attention
paper_summary: Vision transformers have shown great success due to their high model
  capabilities. However, their remarkable performance is accompanied by heavy
  computation costs, which makes them unsuitable for real-time applications. In
  this paper, we propose a family of high-speed vision transformers named
  EfficientViT. We find that the speed of existing transformer models is commonly
  bounded by memory inefficient operations, especially the tensor reshaping and
  element-wise functions in MHSA. Therefore, we design a new building block with
  a sandwich layout, i.e., using a single memory-bound MHSA between efficient FFN
  layers, which improves memory efficiency while enhancing channel communication.
  Moreover, we discover that the attention maps share high similarities across
  heads, leading to computational redundancy. To address this, we present a
  cascaded group attention module feeding attention heads with different splits
  of the full feature, which not only saves computation cost but also improves
  attention diversity. Comprehensive experiments demonstrate EfficientViT
  outperforms existing efficient models, striking a good trade-off between speed
  and accuracy. For instance, our EfficientViT-M5 surpasses MobileNetV3-Large by
  1.9% in accuracy, while getting 40.4% and 45.2% higher throughput on Nvidia
  V100 GPU and Intel Xeon CPU, respectively. Compared to the recent efficient
  model MobileViT-XXS, EfficientViT-M2 achieves 1.8% superior accuracy, while
  running 5.8x/3.7x faster on the GPU/CPU, and 7.4x faster when converted to ONNX
  format. Code and models are available at
  https://github.com/microsoft/Cream/tree/main/EfficientViT.
paper_ai_keywords: Vision transformers EfficientViT MHSA (Multi-Head Self-Attention) tensor reshaping element-wise functions sandwich layout FFN (Feed-Forward Network) memory efficiency channel communication attention maps computational redundancy cascaded group attention module feature growing throughput Nvidia V100 GPU Intel Xeon CPU ONNX format