RMT: Retentive Networks Meet Vision Transformers
Transformer first appears in the field of natural language processing and is
later migrated to the computer vision domain, where it demonstrates excellent
performance in vision tasks. However, recently, Retentive Network (RetNet) has
emerged as an architecture with the potential to replace Transformer,
attracting widespread attention in the NLP community. Therefore, we raise the
question of whether transferring RetNet's idea to vision can also bring
outstanding performance to vision tasks. To address this, we combine RetNet and
Transformer to propose RMT. Inspired by RetNet, RMT introduces explicit decay
into the vision backbone, bringing prior knowledge related to spatial distances
to the vision model. This distance-related spatial prior allows for explicit
control of the range of tokens that each token can attend to. Additionally, to
reduce the computational cost of global modeling, we decompose this modeling
process along the two coordinate axes of the image. Abundant experiments have
demonstrated that our RMT exhibits exceptional performance across various
computer vision tasks. For example, RMT achieves 84.1% Top1-acc on ImageNet-1k
using merely 4.5G FLOPs. To the best of our knowledge, among all models, RMT
achieves the highest Top1-acc when models are of similar size and trained with
the same strategy. Moreover, RMT significantly outperforms existing vision
backbones in downstream tasks such as object detection, instance segmentation,
and semantic segmentation. Our work is still in progress.
650d142844ab290c35942b62
Qihang Fan
admin_assigned
2023-09-22T15:00:25.024Z
64e632d4c5d101d654b2a037
/avatars/fef1142e96effd9e32842e8531b7dd57.svg
Qihang Fang
user
qihfang
650d142844ab290c35942b63
Huaibo Huang
admin_assigned
2023-09-22T15:00:32.564Z
642259feeaad1bcb28b44f55
/avatars/685791fc7e66d685e9446228c9e278b1.svg
Huaibo Huang
user
Bob072
650d142844ab290c35942b64
Mingrui Chen
admin_assigned
2023-09-22T15:00:39.417Z
64049e1d0ab5e22719f37be2
/avatars/1902cd469d231dc252ac92d33af8f5b2.svg
Mingrui Chen
user
Walnutes
650d142844ab290c35942b65
Hongmin Liu
650d142844ab290c35942b66
Ran He
RMT: Retentive Networks Meet Vision Transformers
Transformer first appears in the field of natural language processing and is
later migrated to the computer vision domain, where it demonstrates excellent
performance in vision tasks. However, recently, Retentive Network (RetNet) has
emerged as an architecture with the potential to replace Transformer,
attracting widespread attention in the NLP community. Therefore, we raise the
question of whether transferring RetNet's idea to vision can also bring
outstanding performance to vision tasks. To address this, we combine RetNet and
Transformer to propose RMT. Inspired by RetNet, RMT introduces explicit decay
into the vision backbone, bringing prior knowledge related to spatial distances
to the vision model. This distance-related spatial prior allows for explicit
control of the range of tokens that each token can attend to. Additionally, to
reduce the computational cost of global modeling, we decompose this modeling
process along the two coordinate axes of the image. Abundant experiments have
demonstrated that our RMT exhibits exceptional performance across various
computer vision tasks. For example, RMT achieves 84.1% Top1-acc on ImageNet-1k
using merely 4.5G FLOPs. To the best of our knowledge, among all models, RMT
achieves the highest Top1-acc when models are of similar size and trained with
the same strategy. Moreover, RMT significantly outperforms existing vision
backbones in downstream tasks such as object detection, instance segmentation,
and semantic segmentation. Our work is still in progress.
The proposed RMT model, combining RetNet and Transformer architectures, introduces explicit spatial distance priors and coordinate-wise decomposition to achieve exceptional performance in computer vision tasks.
Transformer Retentive Network (RetNet) RMT explicit decay spatial distances tokens computational cost global modeling ImageNet-1k Top1-acc object detection instance segmentation semantic segmentation