title: Less is More: Focus Attention for Efficient DETR
summary: DETR-like models have significantly boosted the performance of detectors and
  even outperformed classical convolutional models. However, all tokens are
  treated equally without discrimination brings a redundant computational burden
  in the traditional encoder structure. The recent sparsification strategies
  exploit a subset of informative tokens to reduce attention complexity
  maintaining performance through the sparse encoder. But these methods tend to
  rely on unreliable model statistics. Moreover, simply reducing the token
  population hinders the detection performance to a large extent, limiting the
  application of these sparse models. We propose Focus-DETR, which focuses
  attention on more informative tokens for a better trade-off between computation
  efficiency and model accuracy. Specifically, we reconstruct the encoder with
  dual attention, which includes a token scoring mechanism that considers both
  localization and category semantic information of the objects from multi-scale
  feature maps. We efficiently abandon the background queries and enhance the
  semantic interaction of the fine-grained object queries based on the scores.
  Compared with the state-of-the-art sparse DETR-like detectors under the same
  setting, our Focus-DETR gets comparable complexity while achieving 50.4AP
  (+2.2) on COCO. The code is available at
  https://github.com/huawei-noah/noah-research/tree/master/Focus-DETR and
  https://gitee.com/mindspore/models/tree/master/research/cv/Focus-DETR.
paper_authors._id: 64bf8dd95b8d8261461a0a3a
paper_authors.name: Dehua Zheng
paper_authors._id: 64bf8dd95b8d8261461a0a3b
paper_authors.name: Wenhui Dong
paper_authors._id: 64bf8dd95b8d8261461a0a3c
paper_authors.name: Hailin Hu
paper_authors._id: 64bf8dd95b8d8261461a0a3d
paper_authors.name: Xinghao Chen
paper_authors._id: 64bf8dd95b8d8261461a0a3e
paper_authors.name: Yunhe Wang
paper_title: Less is More: Focus Attention for Efficient DETR
paper_summary: DETR-like models have significantly boosted the performance of detectors and
  even outperformed classical convolutional models. However, all tokens are
  treated equally without discrimination brings a redundant computational burden
  in the traditional encoder structure. The recent sparsification strategies
  exploit a subset of informative tokens to reduce attention complexity
  maintaining performance through the sparse encoder. But these methods tend to
  rely on unreliable model statistics. Moreover, simply reducing the token
  population hinders the detection performance to a large extent, limiting the
  application of these sparse models. We propose Focus-DETR, which focuses
  attention on more informative tokens for a better trade-off between computation
  efficiency and model accuracy. Specifically, we reconstruct the encoder with
  dual attention, which includes a token scoring mechanism that considers both
  localization and category semantic information of the objects from multi-scale
  feature maps. We efficiently abandon the background queries and enhance the
  semantic interaction of the fine-grained object queries based on the scores.
  Compared with the state-of-the-art sparse DETR-like detectors under the same
  setting, our Focus-DETR gets comparable complexity while achieving 50.4AP
  (+2.2) on COCO. The code is available at
  https://github.com/huawei-noah/noah-research/tree/master/Focus-DETR and
  https://gitee.com/mindspore/models/tree/master/research/cv/Focus-DETR.
paper_ai_keywords: DETR-like models tokens traditional encoder structure sparsification strategies attention complexity sparse encoder token scoring mechanism localization category semantic information multi-scale feature maps background queries fine-grained object queries dual attention state-of-the-art sparse DETR-like detectors COCO 50.4AP