title: DualToken-ViT: Position-aware Efficient Vision Transformer with Dual
    Token Fusion
summary: Self-attention-based vision transformers (ViTs) have emerged as a highly
  competitive architecture in computer vision. Unlike convolutional neural
  networks (CNNs), ViTs are capable of global information sharing. With the
  development of various structures of ViTs, ViTs are increasingly advantageous
  for many vision tasks. However, the quadratic complexity of self-attention
  renders ViTs computationally intensive, and their lack of inductive biases of
  locality and translation equivariance demands larger model sizes compared to
  CNNs to effectively learn visual features. In this paper, we propose a
  light-weight and efficient vision transformer model called DualToken-ViT that
  leverages the advantages of CNNs and ViTs. DualToken-ViT effectively fuses the
  token with local information obtained by convolution-based structure and the
  token with global information obtained by self-attention-based structure to
  achieve an efficient attention structure. In addition, we use position-aware
  global tokens throughout all stages to enrich the global information, which
  further strengthening the effect of DualToken-ViT. Position-aware global tokens
  also contain the position information of the image, which makes our model
  better for vision tasks. We conducted extensive experiments on image
  classification, object detection and semantic segmentation tasks to demonstrate
  the effectiveness of DualToken-ViT. On the ImageNet-1K dataset, our models of
  different scales achieve accuracies of 75.4% and 79.4% with only 0.5G and 1.0G
  FLOPs, respectively, and our model with 1.0G FLOPs outperforms LightViT-T using
  global tokens by 0.7%.
paper_authors._id: 65111a8d9d93eec4a773cb6f
paper_authors.name: Zhenzhen Chu
paper_authors._id: 65111a8d9d93eec4a773cb70
paper_authors.name: Jiayu Chen
paper_authors._id: 65111a8d9d93eec4a773cb71
paper_authors.name: Cen Chen
paper_authors._id: 65111a8d9d93eec4a773cb72
paper_authors.name: Chengyu Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-25T11:25:49.484Z
paper_authors._id: 62a1ee69e497a54ddb32a998
paper_authors.avatarUrl: /avatars/bf6ceaba75f09e9e30cd52a803acfd32.svg
paper_authors.fullname: Chengyu Wang
paper_authors.type: user
paper_authors.user: chywang
paper_authors._id: 65111a8d9d93eec4a773cb73
paper_authors.name: Ziheng Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-25T11:25:21.986Z
paper_authors._id: 63b7bf7fca2f378e7103e182
paper_authors.avatarUrl: /avatars/73b5179b888293a7039e9bc42d129c61.svg
paper_authors.fullname: wuziheng
paper_authors.type: user
paper_authors.user: wuziheng
paper_authors._id: 65111a8d9d93eec4a773cb74
paper_authors.name: Jun Huang
paper_authors._id: 65111a8d9d93eec4a773cb75
paper_authors.name: Weining Qian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-25T11:24:50.188Z
paper_authors._id: 643916835aa69077ffb27c8a
paper_authors.avatarUrl: /avatars/3eeb0b8ea703d3c873839ad0fd062352.svg
paper_authors.fullname: Weining Qian
paper_authors.type: user
paper_authors.user: wnqian
paper_title: DualToken-ViT: Position-aware Efficient Vision Transformer with Dual
    Token Fusion
paper_summary: Self-attention-based vision transformers (ViTs) have emerged as a highly
  competitive architecture in computer vision. Unlike convolutional neural
  networks (CNNs), ViTs are capable of global information sharing. With the
  development of various structures of ViTs, ViTs are increasingly advantageous
  for many vision tasks. However, the quadratic complexity of self-attention
  renders ViTs computationally intensive, and their lack of inductive biases of
  locality and translation equivariance demands larger model sizes compared to
  CNNs to effectively learn visual features. In this paper, we propose a
  light-weight and efficient vision transformer model called DualToken-ViT that
  leverages the advantages of CNNs and ViTs. DualToken-ViT effectively fuses the
  token with local information obtained by convolution-based structure and the
  token with global information obtained by self-attention-based structure to
  achieve an efficient attention structure. In addition, we use position-aware
  global tokens throughout all stages to enrich the global information, which
  further strengthening the effect of DualToken-ViT. Position-aware global tokens
  also contain the position information of the image, which makes our model
  better for vision tasks. We conducted extensive experiments on image
  classification, object detection and semantic segmentation tasks to demonstrate
  the effectiveness of DualToken-ViT. On the ImageNet-1K dataset, our models of
  different scales achieve accuracies of 75.4% and 79.4% with only 0.5G and 1.0G
  FLOPs, respectively, and our model with 1.0G FLOPs outperforms LightViT-T using
  global tokens by 0.7%.
paper_ai_keywords: self-attention-based vision transformers (ViTs) global information sharing convolutional neural networks (CNNs) quadratic complexity of self-attention computationally intensive inductive biases of locality and translation equivariance light-weight and efficient vision transformer model DualToken-ViT continuous attention structure position-aware global tokens image classification object detection semantic segmentation tasks ImageNet-1K FLOPs (FLOP operations per second)