  Transformers have shown superior performance on various vision tasks. Their
large receptive field endows Transformer models with higher representation
power than their CNN counterparts. Nevertheless, simply enlarging the receptive
field also raises several concerns. On the one hand, using dense attention in
ViT leads to excessive memory and computational cost, and features can be
influenced by irrelevant parts that are beyond the region of interests. On the
other hand, the handcrafted attention adopted in PVT or Swin Transformer is
data agnostic and may limit the ability to model long-range relations. To solve
this dilemma, we propose a novel deformable multi-head attention module, where
the positions of key and value pairs in self-attention are adaptively allocated
in a data-dependent way. This flexible scheme enables the proposed deformable
attention to dynamically focus on relevant regions while maintains the
representation power of global attention. On this basis, we present Deformable
Attention Transformer (DAT), a general vision backbone efficient and effective
for visual recognition. We further build an enhanced version DAT++. Extensive
experiments show that our DAT++ achieves state-of-the-art results on various
visual recognition benchmarks, with 85.9% ImageNet accuracy, 54.5 and 47.0
MS-COCO instance segmentation mAP, and 51.5 ADE20K semantic segmentation mIoU.

Computer Vision and Pattern Recognition