title: InstaGen: Enhancing Object Detection by Training on Synthetic Dataset
summary: In this paper, we introduce a novel paradigm to enhance the ability of object
  detector, e.g., expanding categories or improving detection performance, by
  training on synthetic dataset generated from diffusion models. Specifically, we
  integrate an instance-level grounding head into a pre-trained, generative
  diffusion model, to augment it with the ability of localising arbitrary
  instances in the generated images. The grounding head is trained to align the
  text embedding of category names with the regional visual feature of the
  diffusion model, using supervision from an off-the-shelf object detector, and a
  novel self-training scheme on (novel) categories not covered by the detector.
  This enhanced version of diffusion model, termed as InstaGen, can serve as a
  data synthesizer for object detection. We conduct thorough experiments to show
  that, object detector can be enhanced while training on the synthetic dataset
  from InstaGen, demonstrating superior performance over existing
  state-of-the-art methods in open-vocabulary (+4.5 AP) and data-sparse (+1.2 to
  5.2 AP) scenarios.
paper_authors._id: 65c5a386eabb2d3066caf282
paper_authors.name: Chengjian Feng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:07:54.572Z
paper_authors._id: 639c3582e62c431aa8d9028e
paper_authors.avatarUrl: /avatars/b23c73d0b7ea062f5292c5838b83555a.svg
paper_authors.fullname: Chengjian Feng
paper_authors.type: user
paper_authors.user: jian223
paper_authors._id: 65c5a386eabb2d3066caf283
paper_authors.name: Yujie Zhong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:08:01.808Z
paper_authors._id: 64b4adc90a54158d663268a5
paper_authors.avatarUrl: /avatars/885b8c05d070c69f28879290c33c0aa7.svg
paper_authors.fullname: Yujie Zhong
paper_authors.type: user
paper_authors.user: zhongyujie
paper_authors._id: 65c5a386eabb2d3066caf284
paper_authors.name: Zequn Jie
paper_authors._id: 65c5a386eabb2d3066caf285
paper_authors.name: Weidi Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:08:26.940Z
paper_authors._id: 63f1841f2f7c0152e8748ec4
paper_authors.avatarUrl: /avatars/5398658c7d0cf556531a625a4ca5d18a.svg
paper_authors.fullname: Xie
paper_authors.type: user
paper_authors.user: Weidi
paper_authors._id: 65c5a386eabb2d3066caf286
paper_authors.name: Lin Ma
paper_title: InstaGen: Enhancing Object Detection by Training on Synthetic Dataset
paper_summary: In this paper, we introduce a novel paradigm to enhance the ability of object
  detector, e.g., expanding categories or improving detection performance, by
  training on synthetic dataset generated from diffusion models. Specifically, we
  integrate an instance-level grounding head into a pre-trained, generative
  diffusion model, to augment it with the ability of localising arbitrary
  instances in the generated images. The grounding head is trained to align the
  text embedding of category names with the regional visual feature of the
  diffusion model, using supervision from an off-the-shelf object detector, and a
  novel self-training scheme on (novel) categories not covered by the detector.
  This enhanced version of diffusion model, termed as InstaGen, can serve as a
  data synthesizer for object detection. We conduct thorough experiments to show
  that, object detector can be enhanced while training on the synthetic dataset
  from InstaGen, demonstrating superior performance over existing
  state-of-the-art methods in open-vocabulary (+4.5 AP) and data-sparse (+1.2 to
  5.2 AP) scenarios.
paper_ai_keywords: diffusion models instance-level grounding head text embedding visual features Res diffusion model InstaGen open-vocabulary data-sparse