title: Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion
    Models
summary: Beyond high-fidelity image synthesis, diffusion models have recently
  exhibited promising results in dense visual perception tasks. However, most
  existing work treats diffusion models as a standalone component for perception
  tasks, employing them either solely for off-the-shelf data augmentation or as
  mere feature extractors. In contrast to these isolated and thus sub-optimal
  efforts, we introduce a unified, versatile, diffusion-based framework,
  Diff-2-in-1, that can simultaneously handle both multi-modal data generation
  and dense visual perception, through a unique exploitation of the
  diffusion-denoising process. Within this framework, we further enhance
  discriminative visual perception via multi-modal generation, by utilizing the
  denoising network to create multi-modal data that mirror the distribution of
  the original training set. Importantly, Diff-2-in-1 optimizes the utilization
  of the created diverse and faithful data by leveraging a novel self-improving
  learning mechanism. Comprehensive experimental evaluations validate the
  effectiveness of our framework, showcasing consistent performance improvements
  across various discriminative backbones and high-quality multi-modal data
  generation characterized by both realism and usefulness.
paper_authors._id: 672e3a87b0d4d3107840222e
paper_authors.name: Shuhong Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-19T18:05:07.095Z
paper_authors._id: 64361417a4bd75c62cc1e534
paper_authors.avatarUrl: /avatars/b9b43985a0190ccf418c6853f9c3eb16.svg
paper_authors.fullname: Shuhong Zheng
paper_authors.type: user
paper_authors.user: ShuhongZheng
paper_authors._id: 672e3a87b0d4d3107840222f
paper_authors.name: Zhipeng Bao
paper_authors._id: 672e3a87b0d4d31078402230
paper_authors.name: Ruoyu Zhao
paper_authors._id: 672e3a87b0d4d31078402231
paper_authors.name: Martial Hebert
paper_authors._id: 672e3a87b0d4d31078402232
paper_authors.name: Yu-Xiong Wang
paper_title: Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion
    Models
paper_summary: Beyond high-fidelity image synthesis, diffusion models have recently
  exhibited promising results in dense visual perception tasks. However, most
  existing work treats diffusion models as a standalone component for perception
  tasks, employing them either solely for off-the-shelf data augmentation or as
  mere feature extractors. In contrast to these isolated and thus sub-optimal
  efforts, we introduce a unified, versatile, diffusion-based framework,
  Diff-2-in-1, that can simultaneously handle both multi-modal data generation
  and dense visual perception, through a unique exploitation of the
  diffusion-denoising process. Within this framework, we further enhance
  discriminative visual perception via multi-modal generation, by utilizing the
  denoising network to create multi-modal data that mirror the distribution of
  the original training set. Importantly, Diff-2-in-1 optimizes the utilization
  of the created diverse and faithful data by leveraging a novel self-improving
  learning mechanism. Comprehensive experimental evaluations validate the
  effectiveness of our framework, showcasing consistent performance improvements
  across various discriminative backbones and high-quality multi-modal data
  generation characterized by both realism and usefulness.
paper_ai_keywords: diffusion models dense visual perception data augmentation feature extractors diffusion-denoising process denoising network multi-modal data generation self-improving learning mechanism