title: Scale-Aware Modulation Meet Transformer
summary: This paper presents a new vision Transformer, Scale-Aware Modulation
  Transformer (SMT), that can handle various downstream tasks efficiently by
  combining the convolutional network and vision Transformer. The proposed
  Scale-Aware Modulation (SAM) in the SMT includes two primary novel designs.
  Firstly, we introduce the Multi-Head Mixed Convolution (MHMC) module, which can
  capture multi-scale features and expand the receptive field. Secondly, we
  propose the Scale-Aware Aggregation (SAA) module, which is lightweight but
  effective, enabling information fusion across different heads. By leveraging
  these two modules, convolutional modulation is further enhanced. Furthermore,
  in contrast to prior works that utilized modulations throughout all stages to
  build an attention-free network, we propose an Evolutionary Hybrid Network
  (EHN), which can effectively simulate the shift from capturing local to global
  dependencies as the network becomes deeper, resulting in superior performance.
  Extensive experiments demonstrate that SMT significantly outperforms existing
  state-of-the-art models across a wide range of visual tasks. Specifically, SMT
  with 11.5M / 2.4GFLOPs and 32M / 7.7GFLOPs can achieve 82.2% and 84.3% top-1
  accuracy on ImageNet-1K, respectively. After pretrained on ImageNet-22K in
  224^2 resolution, it attains 87.1% and 88.1% top-1 accuracy when finetuned with
  resolution 224^2 and 384^2, respectively. For object detection with Mask R-CNN,
  the SMT base trained with 1x and 3x schedule outperforms the Swin Transformer
  counterpart by 4.2 and 1.3 mAP on COCO, respectively. For semantic segmentation
  with UPerNet, the SMT base test at single- and multi-scale surpasses Swin by
  2.0 and 1.1 mIoU respectively on the ADE20K.
paper_authors._id: 64b60dae786aaee1929d8a47
paper_authors.name: Weifeng Lin
paper_authors._id: 64b60dae786aaee1929d8a48
paper_authors.name: Ziheng Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-18T08:23:13.847Z
paper_authors._id: 63b7bf7fca2f378e7103e182
paper_authors.avatarUrl: /avatars/73b5179b888293a7039e9bc42d129c61.svg
paper_authors.fullname: wuziheng
paper_authors.type: user
paper_authors.user: wuziheng
paper_authors._id: 64b60dae786aaee1929d8a49
paper_authors.name: Jiayu Chen
paper_authors._id: 64b60dae786aaee1929d8a4a
paper_authors.name: Jun Huang
paper_authors._id: 64b60dae786aaee1929d8a4b
paper_authors.name: Lianwen Jin
paper_title: Scale-Aware Modulation Meet Transformer
paper_summary: This paper presents a new vision Transformer, Scale-Aware Modulation
  Transformer (SMT), that can handle various downstream tasks efficiently by
  combining the convolutional network and vision Transformer. The proposed
  Scale-Aware Modulation (SAM) in the SMT includes two primary novel designs.
  Firstly, we introduce the Multi-Head Mixed Convolution (MHMC) module, which can
  capture multi-scale features and expand the receptive field. Secondly, we
  propose the Scale-Aware Aggregation (SAA) module, which is lightweight but
  effective, enabling information fusion across different heads. By leveraging
  these two modules, convolutional modulation is further enhanced. Furthermore,
  in contrast to prior works that utilized modulations throughout all stages to
  build an attention-free network, we propose an Evolutionary Hybrid Network
  (EHN), which can effectively simulate the shift from capturing local to global
  dependencies as the network becomes deeper, resulting in superior performance.
  Extensive experiments demonstrate that SMT significantly outperforms existing
  state-of-the-art models across a wide range of visual tasks. Specifically, SMT
  with 11.5M / 2.4GFLOPs and 32M / 7.7GFLOPs can achieve 82.2% and 84.3% top-1
  accuracy on ImageNet-1K, respectively. After pretrained on ImageNet-22K in
  224^2 resolution, it attains 87.1% and 88.1% top-1 accuracy when finetuned with
  resolution 224^2 and 384^2, respectively. For object detection with Mask R-CNN,
  the SMT base trained with 1x and 3x schedule outperforms the Swin Transformer
  counterpart by 4.2 and 1.3 mAP on COCO, respectively. For semantic segmentation
  with UPerNet, the SMT base test at single- and multi-scale surpasses Swin by
  2.0 and 1.1 mIoU respectively on the ADE20K.
paper_ai_summary: A Scale-Aware Modulation Transformer combines convolutional networks with attention mechanisms using Multi-Head Mixed Convolution and Scale-Aware Aggregation modules, achieving superior performance across visual tasks compared to existing models.
paper_ai_keywords: vision Transformer Scale-Aware Modulation Multi-Head Mixed Convolution Scale-Aware Aggregation Evolutionary Hybrid Network attention-free network ImageNet top-1 accuracy Mask R-CNN mAP UPerNet mIoU ADE20K