ControlNet++: Improving Conditional Controls with Efficient Consistency
  Feedback
To enhance the controllability of text-to-image diffusion models, existing
efforts like ControlNet incorporated image-based conditional controls. In this
paper, we reveal that existing methods still face significant challenges in
generating images that align with the image conditional controls. To this end,
we propose ControlNet++, a novel approach that improves controllable generation
by explicitly optimizing pixel-level cycle consistency between generated images
and conditional controls. Specifically, for an input conditional control, we
use a pre-trained discriminative reward model to extract the corresponding
condition of the generated images, and then optimize the consistency loss
between the input conditional control and extracted condition. A
straightforward implementation would be generating images from random noises
and then calculating the consistency loss, but such an approach requires
storing gradients for multiple sampling timesteps, leading to considerable time
and memory costs. To address this, we introduce an efficient reward strategy
that deliberately disturbs the input images by adding noise, and then uses the
single-step denoised images for reward fine-tuning. This avoids the extensive
costs associated with image sampling, allowing for more efficient reward
fine-tuning. Extensive experiments show that ControlNet++ significantly
improves controllability under various conditional controls. For example, it
achieves improvements over ControlNet by 7.9% mIoU, 13.4% SSIM, and 7.6% RMSE,
respectively, for segmentation mask, line-art edge, and depth conditions.
6618975e4f47c19ab9b6b891
Ming Li
claimed_verified
2024-04-16T00:02:52.951Z
637f0eb22438d7485b8ef5d7
https://cdn-avatars.huggingface.co/v1/production/uploads/637f0eb22438d7485b8ef5d7/70h7dekqj7LuBobOXckmJ.jpeg
Ming Li
user
limingcv
6618975e4f47c19ab9b6b892
Taojiannan Yang
6618975e4f47c19ab9b6b893
Huafeng Kuang
6618975e4f47c19ab9b6b894
Jie Wu
6618975e4f47c19ab9b6b895
Zhaoning Wang
6618975e4f47c19ab9b6b896
Xuefeng Xiao
admin_assigned
2024-04-18T11:55:00.608Z
646b7f71df2609a541c1ab9f
/avatars/48b82e5fd9b06f41ff825507c36816cd.svg
Xuefeng Xiao
user
xiaoxuefeng
6618975e4f47c19ab9b6b897
Chen Chen
ControlNet++: Improving Conditional Controls with Efficient Consistency
  Feedback
To enhance the controllability of text-to-image diffusion models, existing
efforts like ControlNet incorporated image-based conditional controls. In this
paper, we reveal that existing methods still face significant challenges in
generating images that align with the image conditional controls. To this end,
we propose ControlNet++, a novel approach that improves controllable generation
by explicitly optimizing pixel-level cycle consistency between generated images
and conditional controls. Specifically, for an input conditional control, we
use a pre-trained discriminative reward model to extract the corresponding
condition of the generated images, and then optimize the consistency loss
between the input conditional control and extracted condition. A
straightforward implementation would be generating images from random noises
and then calculating the consistency loss, but such an approach requires
storing gradients for multiple sampling timesteps, leading to considerable time
and memory costs. To address this, we introduce an efficient reward strategy
that deliberately disturbs the input images by adding noise, and then uses the
single-step denoised images for reward fine-tuning. This avoids the extensive
costs associated with image sampling, allowing for more efficient reward
fine-tuning. Extensive experiments show that ControlNet++ significantly
improves controllability under various conditional controls. For example, it
achieves improvements over ControlNet by 7.9% mIoU, 13.4% SSIM, and 7.6% RMSE,
respectively, for segmentation mask, line-art edge, and depth conditions.
ControlNet++ enhances text-to-image generation controllability through pixel-level cycle consistency optimization using an efficient noise-disturbed reward strategy.
ControlNet text-to-image diffusion models pixel-level cycle consistency discriminative reward model consistency loss single-step denoised images reward fine-tuning segmentation mask line-art edge depth conditions mIoU SSIM RMSE