Aligning Generative Denoising with Discriminative Objectives Unleashes
  Diffusion for Visual Perception
With the success of image generation, generative diffusion models are
increasingly adopted for discriminative tasks, as pixel generation provides a
unified perception interface. However, directly repurposing the generative
denoising process for discriminative objectives reveals critical gaps rarely
addressed previously. Generative models tolerate intermediate sampling errors
if the final distribution remains plausible, but discriminative tasks require
rigorous accuracy throughout, as evidenced in challenging multi-modal tasks
like referring image segmentation. Motivated by this gap, we analyze and
enhance alignment between generative diffusion processes and perception tasks,
focusing on how perception quality evolves during denoising. We find: (1)
earlier denoising steps contribute disproportionately to perception quality,
prompting us to propose tailored learning objectives reflecting varying
timestep contributions; (2) later denoising steps show unexpected perception
degradation, highlighting sensitivity to training-denoising distribution
shifts, addressed by our diffusion-tailored data augmentation; and (3)
generative processes uniquely enable interactivity, serving as controllable
user interfaces adaptable to correctional prompts in multi-round interactions.
Our insights significantly improve diffusion-based perception models without
architectural changes, achieving state-of-the-art performance on depth
estimation, referring image segmentation, and generalist perception tasks. Code
available at https://github.com/ziqipang/ADDP.
67ffcf5874193e65a7831a17
Ziqi Pang
admin_assigned
2025-04-17T11:16:13.569Z
642a33ea5673845d9854f458
/avatars/ea8ed29a0b4ad629a2ba6e7d26cbd923.svg
Ziqi Pang
user
ziqipang
67ffcf5874193e65a7831a18
Xin Xu
67ffcf5874193e65a7831a19
Yu-Xiong Wang
Aligning Generative Denoising with Discriminative Objectives Unleashes
  Diffusion for Visual Perception
With the success of image generation, generative diffusion models are
increasingly adopted for discriminative tasks, as pixel generation provides a
unified perception interface. However, directly repurposing the generative
denoising process for discriminative objectives reveals critical gaps rarely
addressed previously. Generative models tolerate intermediate sampling errors
if the final distribution remains plausible, but discriminative tasks require
rigorous accuracy throughout, as evidenced in challenging multi-modal tasks
like referring image segmentation. Motivated by this gap, we analyze and
enhance alignment between generative diffusion processes and perception tasks,
focusing on how perception quality evolves during denoising. We find: (1)
earlier denoising steps contribute disproportionately to perception quality,
prompting us to propose tailored learning objectives reflecting varying
timestep contributions; (2) later denoising steps show unexpected perception
degradation, highlighting sensitivity to training-denoising distribution
shifts, addressed by our diffusion-tailored data augmentation; and (3)
generative processes uniquely enable interactivity, serving as controllable
user interfaces adaptable to correctional prompts in multi-round interactions.
Our insights significantly improve diffusion-based perception models without
architectural changes, achieving state-of-the-art performance on depth
estimation, referring image segmentation, and generalist perception tasks. Code
available at https://github.com/ziqipang/ADDP.
Enhancements to generative diffusion models address gaps in discriminative tasks by focusing on perception quality during denoising, improving performance on depth estimation, referring image segmentation, and generalist perception tasks.
generative diffusion models discriminative tasks pixel generation generative denoising referring image segmentation perception quality denoising steps learning objectives diffusion-tailored data augmentation interactivity correctional prompts depth estimation generalist perception tasks