title: Revisiting DETR Pre-training for Object Detection
summary: Motivated by that DETR-based approaches have established new records on COCO
  detection and segmentation benchmarks, many recent endeavors show increasing
  interest in how to further improve DETR-based approaches by pre-training the
  Transformer in a self-supervised manner while keeping the backbone frozen. Some
  studies already claimed significant improvements in accuracy. In this paper, we
  take a closer look at their experimental methodology and check if their
  approaches are still effective on the very recent state-of-the-art such as
  H-Deformable-DETR. We conduct thorough experiments on COCO object
  detection tasks to study the influence of the choice of pre-training datasets,
  localization, and classification target generation schemes. Unfortunately, we
  find the previous representative self-supervised approach such as DETReg, fails
  to boost the performance of the strong DETR-based approaches on full data
  regimes. We further analyze the reasons and find that simply combining a more
  accurate box predictor and Objects365 benchmark can significantly improve the
  results in follow-up experiments. We demonstrate the effectiveness of our
  approach by achieving strong object detection results of AP=59.3% on COCO
  val set, which surpasses H-Deformable-DETR + Swin-L by +1.4%.
  Last, we generate a series of synthetic pre-training datasets by combining the
  very recent image-to-text captioning models (LLaVA) and text-to-image
  generative models (SDXL). Notably, pre-training on these synthetic datasets
  leads to notable improvements in object detection performance. Looking ahead,
  we anticipate substantial advantages through the future expansion of the
  synthetic pre-training dataset.
paper_authors._id: 64cb33d500377e28486f8486
paper_authors.name: Yan Ma
paper_authors._id: 64cb33d500377e28486f8487
paper_authors.name: Weicong Liang
paper_authors._id: 64cb33d500377e28486f8488
paper_authors.name: Yiduo Hao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-29T08:55:43.434Z
paper_authors._id: 64d26a067a4b4217eff1effb
paper_authors.avatarUrl: /avatars/b684a6061b71fde92939cb1f73351d21.svg
paper_authors.fullname: Yiduo Hao
paper_authors.type: user
paper_authors.user: Scherzando
paper_authors._id: 64cb33d500377e28486f8489
paper_authors.name: Bohan Chen
paper_authors._id: 64cb33d500377e28486f848a
paper_authors.name: Xiangyu Yue
paper_authors._id: 64cb33d500377e28486f848b
paper_authors.name: Chao Zhang
paper_authors._id: 64cb33d500377e28486f848c
paper_authors.name: Yuhui Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-28T08:39:35.114Z
paper_authors._id: 631f108bb45367a05fe74260
paper_authors.avatarUrl: /avatars/c20da6800b643728062712d9a2771648.svg
paper_authors.fullname: Researcher
paper_authors.type: user
paper_authors.user: YuanYuhui
paper_title: Revisiting DETR Pre-training for Object Detection
paper_summary: Motivated by that DETR-based approaches have established new records on COCO
  detection and segmentation benchmarks, many recent endeavors show increasing
  interest in how to further improve DETR-based approaches by pre-training the
  Transformer in a self-supervised manner while keeping the backbone frozen. Some
  studies already claimed significant improvements in accuracy. In this paper, we
  take a closer look at their experimental methodology and check if their
  approaches are still effective on the very recent state-of-the-art such as
  H-Deformable-DETR. We conduct thorough experiments on COCO object
  detection tasks to study the influence of the choice of pre-training datasets,
  localization, and classification target generation schemes. Unfortunately, we
  find the previous representative self-supervised approach such as DETReg, fails
  to boost the performance of the strong DETR-based approaches on full data
  regimes. We further analyze the reasons and find that simply combining a more
  accurate box predictor and Objects365 benchmark can significantly improve the
  results in follow-up experiments. We demonstrate the effectiveness of our
  approach by achieving strong object detection results of AP=59.3% on COCO
  val set, which surpasses H-Deformable-DETR + Swin-L by +1.4%.
  Last, we generate a series of synthetic pre-training datasets by combining the
  very recent image-to-text captioning models (LLaVA) and text-to-image
  generative models (SDXL). Notably, pre-training on these synthetic datasets
  leads to notable improvements in object detection performance. Looking ahead,
  we anticipate substantial advantages through the future expansion of the
  synthetic pre-training dataset.
paper_ai_summary: Thorough evaluation reveals that synthetic pre-training datasets derived from recent captioning and generative models significantly improve the performance of DETR-based object detection approaches.
paper_ai_keywords: DETR Transformer self-supervised pre-training backbone COCO object detection pre-training datasets localization classification target generation DETReg $\mathcal{H}$-Deformable-DETR Swin-L LLaVA SDXL synthetic pre-training datasets