title: FMViT: A multiple-frequency mixing Vision Transformer
summary: The transformer model has gained widespread adoption in computer vision tasks
  in recent times. However, due to the quadratic time and memory complexity of
  self-attention, which is proportional to the number of input tokens, most
  existing Vision Transformers (ViTs) encounter challenges in achieving efficient
  performance in practical industrial deployment scenarios, such as TensorRT and
  CoreML, where traditional CNNs excel. Although some recent attempts have been
  made to design CNN-Transformer hybrid architectures to tackle this problem,
  their overall performance has not met expectations. To tackle these challenges,
  we propose an efficient hybrid ViT architecture named FMViT. This approach
  enhances the model's expressive power by blending high-frequency features and
  low-frequency features with varying frequencies, enabling it to capture both
  local and global information effectively. Additionally, we introduce
  deploy-friendly mechanisms such as Convolutional Multigroup Reparameterization
  (gMLP), Lightweight Multi-head Self-Attention (RLMHSA), and Convolutional
  Fusion Block (CFB) to further improve the model's performance and reduce
  computational overhead. Our experiments demonstrate that FMViT surpasses
  existing CNNs, ViTs, and CNNTransformer hybrid architectures in terms of
  latency/accuracy trade-offs for various vision tasks. On the TensorRT platform,
  FMViT outperforms Resnet101 by 2.5% (83.3% vs. 80.8%) in top-1 accuracy on the
  ImageNet dataset while maintaining similar inference latency. Moreover, FMViT
  achieves comparable performance with EfficientNet-B5, but with a 43%
  improvement in inference speed. On CoreML, FMViT outperforms MobileOne by 2.6%
  in top-1 accuracy on the ImageNet dataset, with inference latency comparable to
  MobileOne (78.5% vs. 75.9%). Our code can be found at
  https://github.com/tany0699/FMViT.
paper_authors._id: 65519f8752c59e60d5d0b258
paper_authors.name: Wei Tan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-13T10:33:44.143Z
paper_authors._id: 63ec43ebc81b6a52391c268e
paper_authors.avatarUrl: /avatars/de5926d598f1d818c42f1e4a1c544b7b.svg
paper_authors.fullname: tanwei
paper_authors.type: user
paper_authors.user: tany0699
paper_authors._id: 65519f8752c59e60d5d0b259
paper_authors.name: Yifeng Geng
paper_authors._id: 65519f8752c59e60d5d0b25a
paper_authors.name: Xuansong Xie
paper_title: FMViT: A multiple-frequency mixing Vision Transformer
paper_summary: The transformer model has gained widespread adoption in computer vision tasks
  in recent times. However, due to the quadratic time and memory complexity of
  self-attention, which is proportional to the number of input tokens, most
  existing Vision Transformers (ViTs) encounter challenges in achieving efficient
  performance in practical industrial deployment scenarios, such as TensorRT and
  CoreML, where traditional CNNs excel. Although some recent attempts have been
  made to design CNN-Transformer hybrid architectures to tackle this problem,
  their overall performance has not met expectations. To tackle these challenges,
  we propose an efficient hybrid ViT architecture named FMViT. This approach
  enhances the model's expressive power by blending high-frequency features and
  low-frequency features with varying frequencies, enabling it to capture both
  local and global information effectively. Additionally, we introduce
  deploy-friendly mechanisms such as Convolutional Multigroup Reparameterization
  (gMLP), Lightweight Multi-head Self-Attention (RLMHSA), and Convolutional
  Fusion Block (CFB) to further improve the model's performance and reduce
  computational overhead. Our experiments demonstrate that FMViT surpasses
  existing CNNs, ViTs, and CNNTransformer hybrid architectures in terms of
  latency/accuracy trade-offs for various vision tasks. On the TensorRT platform,
  FMViT outperforms Resnet101 by 2.5% (83.3% vs. 80.8%) in top-1 accuracy on the
  ImageNet dataset while maintaining similar inference latency. Moreover, FMViT
  achieves comparable performance with EfficientNet-B5, but with a 43%
  improvement in inference speed. On CoreML, FMViT outperforms MobileOne by 2.6%
  in top-1 accuracy on the ImageNet dataset, with inference latency comparable to
  MobileOne (78.5% vs. 75.9%). Our code can be found at
  https://github.com/tany0699/FMViT.
paper_ai_summary: An FMViT hybrid architecture combines Convolutional Multigroup Reparameterization, Lightweight Multi-head Self-Attention, and Convolutional Fusion Block to achieve efficient vision tasks with superior accuracy and speed.
paper_ai_keywords: transformer Vision Transformers self-attention Covolutional Multigroup Reparameterization Lightweight Multi-head Self-Attention Convolutional Fusion Block