DETR Doesn't Need Multi-Scale or Locality Design
This paper presents an improved DETR detector that maintains a "plain"
nature: using a single-scale feature map and global cross-attention
calculations without specific locality constraints, in contrast to previous
leading DETR-based detectors that reintroduce architectural inductive biases of
multi-scale and locality into the decoder. We show that two simple technologies
are surprisingly effective within a plain design to compensate for the lack of
multi-scale feature maps and locality constraints. The first is a box-to-pixel
relative position bias (BoxRPB) term added to the cross-attention formulation,
which well guides each query to attend to the corresponding object region while
also providing encoding flexibility. The second is masked image modeling
(MIM)-based backbone pre-training which helps learn representation with
fine-grained localization ability and proves crucial for remedying dependencies
on the multi-scale feature maps. By incorporating these technologies and recent
advancements in training and problem formation, the improved "plain" DETR
showed exceptional improvements over the original DETR detector. By leveraging
the Object365 dataset for pre-training, it achieved 63.9 mAP accuracy using a
Swin-L backbone, which is highly competitive with state-of-the-art detectors
which all heavily rely on multi-scale feature maps and region-based feature
extraction. Code is available at https://github.com/impiga/Plain-DETR .
64cc839f275c763046155511
Yutong Lin
admin_assigned
2023-08-04T12:47:06.770Z
61f25d7b317f58c6f33e15e2
/avatars/47b93ffb40bc8cbdad21906012e8011a.svg
Yutong Lin
user
impiga
64cc839f275c763046155512
Yuhui Yuan
admin_assigned
2023-08-04T12:47:40.228Z
646f69a6041e48e1c4728de3
https://cdn-avatars.huggingface.co/v1/production/uploads/646f69a6041e48e1c4728de3/U5OaW6PgsXTXnfG03xs9Q.png
GlyphByT5
user
GlyphByT5
64cc839f275c763046155513
Zheng Zhang
64cc839f275c763046155514
Chen Li
64cc839f275c763046155515
Nanning Zheng
64cc839f275c763046155516
Han Hu
DETR Doesn't Need Multi-Scale or Locality Design
This paper presents an improved DETR detector that maintains a "plain"
nature: using a single-scale feature map and global cross-attention
calculations without specific locality constraints, in contrast to previous
leading DETR-based detectors that reintroduce architectural inductive biases of
multi-scale and locality into the decoder. We show that two simple technologies
are surprisingly effective within a plain design to compensate for the lack of
multi-scale feature maps and locality constraints. The first is a box-to-pixel
relative position bias (BoxRPB) term added to the cross-attention formulation,
which well guides each query to attend to the corresponding object region while
also providing encoding flexibility. The second is masked image modeling
(MIM)-based backbone pre-training which helps learn representation with
fine-grained localization ability and proves crucial for remedying dependencies
on the multi-scale feature maps. By incorporating these technologies and recent
advancements in training and problem formation, the improved "plain" DETR
showed exceptional improvements over the original DETR detector. By leveraging
the Object365 dataset for pre-training, it achieved 63.9 mAP accuracy using a
Swin-L backbone, which is highly competitive with state-of-the-art detectors
which all heavily rely on multi-scale feature maps and region-based feature
extraction. Code is available at https://github.com/impiga/Plain-DETR .
A plain DETR detector with global cross-attention and box-to-pixel relative position bias achieves competitive performance using masked image modeling and a single-scale feature map.
DETR single-scale feature map global cross-attention box-to-pixel relative position bias masked image modeling Object365 dataset Swin-L backbone