title: RMT: Retentive Networks Meet Vision Transformers
summary: Transformer first appears in the field of natural language processing and is
  later migrated to the computer vision domain, where it demonstrates excellent
  performance in vision tasks. However, recently, Retentive Network (RetNet) has
  emerged as an architecture with the potential to replace Transformer,
  attracting widespread attention in the NLP community. Therefore, we raise the
  question of whether transferring RetNet's idea to vision can also bring
  outstanding performance to vision tasks. To address this, we combine RetNet and
  Transformer to propose RMT. Inspired by RetNet, RMT introduces explicit decay
  into the vision backbone, bringing prior knowledge related to spatial distances
  to the vision model. This distance-related spatial prior allows for explicit
  control of the range of tokens that each token can attend to. Additionally, to
  reduce the computational cost of global modeling, we decompose this modeling
  process along the two coordinate axes of the image. Abundant experiments have
  demonstrated that our RMT exhibits exceptional performance across various
  computer vision tasks. For example, RMT achieves 84.1% Top1-acc on ImageNet-1k
  using merely 4.5G FLOPs. To the best of our knowledge, among all models, RMT
  achieves the highest Top1-acc when models are of similar size and trained with
  the same strategy. Moreover, RMT significantly outperforms existing vision
  backbones in downstream tasks such as object detection, instance segmentation,
  and semantic segmentation. Our work is still in progress.
paper_authors._id: 650d142844ab290c35942b62
paper_authors.name: Qihang Fan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-22T15:00:25.024Z
paper_authors._id: 64e632d4c5d101d654b2a037
paper_authors.avatarUrl: /avatars/fef1142e96effd9e32842e8531b7dd57.svg
paper_authors.fullname: Qihang Fang
paper_authors.type: user
paper_authors.user: qihfang
paper_authors._id: 650d142844ab290c35942b63
paper_authors.name: Huaibo Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-22T15:00:32.564Z
paper_authors._id: 642259feeaad1bcb28b44f55
paper_authors.avatarUrl: /avatars/685791fc7e66d685e9446228c9e278b1.svg
paper_authors.fullname: Huaibo Huang
paper_authors.type: user
paper_authors.user: Bob072
paper_authors._id: 650d142844ab290c35942b64
paper_authors.name: Mingrui Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-22T15:00:39.417Z
paper_authors._id: 64049e1d0ab5e22719f37be2
paper_authors.avatarUrl: /avatars/1902cd469d231dc252ac92d33af8f5b2.svg
paper_authors.fullname: Mingrui Chen
paper_authors.type: user
paper_authors.user: Walnutes
paper_authors._id: 650d142844ab290c35942b65
paper_authors.name: Hongmin Liu
paper_authors._id: 650d142844ab290c35942b66
paper_authors.name: Ran He
paper_title: RMT: Retentive Networks Meet Vision Transformers
paper_summary: Transformer first appears in the field of natural language processing and is
  later migrated to the computer vision domain, where it demonstrates excellent
  performance in vision tasks. However, recently, Retentive Network (RetNet) has
  emerged as an architecture with the potential to replace Transformer,
  attracting widespread attention in the NLP community. Therefore, we raise the
  question of whether transferring RetNet's idea to vision can also bring
  outstanding performance to vision tasks. To address this, we combine RetNet and
  Transformer to propose RMT. Inspired by RetNet, RMT introduces explicit decay
  into the vision backbone, bringing prior knowledge related to spatial distances
  to the vision model. This distance-related spatial prior allows for explicit
  control of the range of tokens that each token can attend to. Additionally, to
  reduce the computational cost of global modeling, we decompose this modeling
  process along the two coordinate axes of the image. Abundant experiments have
  demonstrated that our RMT exhibits exceptional performance across various
  computer vision tasks. For example, RMT achieves 84.1% Top1-acc on ImageNet-1k
  using merely 4.5G FLOPs. To the best of our knowledge, among all models, RMT
  achieves the highest Top1-acc when models are of similar size and trained with
  the same strategy. Moreover, RMT significantly outperforms existing vision
  backbones in downstream tasks such as object detection, instance segmentation,
  and semantic segmentation. Our work is still in progress.
paper_ai_summary: The proposed RMT model, combining RetNet and Transformer architectures, introduces explicit spatial distance priors and coordinate-wise decomposition to achieve exceptional performance in computer vision tasks.
paper_ai_keywords: Transformer Retentive Network (RetNet) RMT explicit decay spatial distances tokens computational cost global modeling ImageNet-1k Top1-acc object detection instance segmentation semantic segmentation