title: DETR Doesn't Need Multi-Scale or Locality Design
summary: This paper presents an improved DETR detector that maintains a "plain"
  nature: using a single-scale feature map and global cross-attention
  calculations without specific locality constraints, in contrast to previous
  leading DETR-based detectors that reintroduce architectural inductive biases of
  multi-scale and locality into the decoder. We show that two simple technologies
  are surprisingly effective within a plain design to compensate for the lack of
  multi-scale feature maps and locality constraints. The first is a box-to-pixel
  relative position bias (BoxRPB) term added to the cross-attention formulation,
  which well guides each query to attend to the corresponding object region while
  also providing encoding flexibility. The second is masked image modeling
  (MIM)-based backbone pre-training which helps learn representation with
  fine-grained localization ability and proves crucial for remedying dependencies
  on the multi-scale feature maps. By incorporating these technologies and recent
  advancements in training and problem formation, the improved "plain" DETR
  showed exceptional improvements over the original DETR detector. By leveraging
  the Object365 dataset for pre-training, it achieved 63.9 mAP accuracy using a
  Swin-L backbone, which is highly competitive with state-of-the-art detectors
  which all heavily rely on multi-scale feature maps and region-based feature
  extraction. Code is available at https://github.com/impiga/Plain-DETR .
paper_authors._id: 64cc839f275c763046155511
paper_authors.name: Yutong Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-04T12:47:06.770Z
paper_authors._id: 61f25d7b317f58c6f33e15e2
paper_authors.avatarUrl: /avatars/47b93ffb40bc8cbdad21906012e8011a.svg
paper_authors.fullname: Yutong Lin
paper_authors.type: user
paper_authors.user: impiga
paper_authors._id: 64cc839f275c763046155512
paper_authors.name: Yuhui Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-04T12:47:40.228Z
paper_authors._id: 646f69a6041e48e1c4728de3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646f69a6041e48e1c4728de3/U5OaW6PgsXTXnfG03xs9Q.png
paper_authors.fullname: GlyphByT5
paper_authors.type: user
paper_authors.user: GlyphByT5
paper_authors._id: 64cc839f275c763046155513
paper_authors.name: Zheng Zhang
paper_authors._id: 64cc839f275c763046155514
paper_authors.name: Chen Li
paper_authors._id: 64cc839f275c763046155515
paper_authors.name: Nanning Zheng
paper_authors._id: 64cc839f275c763046155516
paper_authors.name: Han Hu
paper_title: DETR Doesn't Need Multi-Scale or Locality Design
paper_summary: This paper presents an improved DETR detector that maintains a "plain"
  nature: using a single-scale feature map and global cross-attention
  calculations without specific locality constraints, in contrast to previous
  leading DETR-based detectors that reintroduce architectural inductive biases of
  multi-scale and locality into the decoder. We show that two simple technologies
  are surprisingly effective within a plain design to compensate for the lack of
  multi-scale feature maps and locality constraints. The first is a box-to-pixel
  relative position bias (BoxRPB) term added to the cross-attention formulation,
  which well guides each query to attend to the corresponding object region while
  also providing encoding flexibility. The second is masked image modeling
  (MIM)-based backbone pre-training which helps learn representation with
  fine-grained localization ability and proves crucial for remedying dependencies
  on the multi-scale feature maps. By incorporating these technologies and recent
  advancements in training and problem formation, the improved "plain" DETR
  showed exceptional improvements over the original DETR detector. By leveraging
  the Object365 dataset for pre-training, it achieved 63.9 mAP accuracy using a
  Swin-L backbone, which is highly competitive with state-of-the-art detectors
  which all heavily rely on multi-scale feature maps and region-based feature
  extraction. Code is available at https://github.com/impiga/Plain-DETR .
paper_ai_keywords: DETR global cross-attention box-to-pixel relative position bias (BoxRPB) masked image modeling (MIM) backbone pre-training Object365 dataset Swin-L backbone object region mAP accuracy region-based feature extraction multi-scale feature maps