title: ControlNet++: Improving Conditional Controls with Efficient Consistency
    Feedback
summary: To enhance the controllability of text-to-image diffusion models, existing
  efforts like ControlNet incorporated image-based conditional controls. In this
  paper, we reveal that existing methods still face significant challenges in
  generating images that align with the image conditional controls. To this end,
  we propose ControlNet++, a novel approach that improves controllable generation
  by explicitly optimizing pixel-level cycle consistency between generated images
  and conditional controls. Specifically, for an input conditional control, we
  use a pre-trained discriminative reward model to extract the corresponding
  condition of the generated images, and then optimize the consistency loss
  between the input conditional control and extracted condition. A
  straightforward implementation would be generating images from random noises
  and then calculating the consistency loss, but such an approach requires
  storing gradients for multiple sampling timesteps, leading to considerable time
  and memory costs. To address this, we introduce an efficient reward strategy
  that deliberately disturbs the input images by adding noise, and then uses the
  single-step denoised images for reward fine-tuning. This avoids the extensive
  costs associated with image sampling, allowing for more efficient reward
  fine-tuning. Extensive experiments show that ControlNet++ significantly
  improves controllability under various conditional controls. For example, it
  achieves improvements over ControlNet by 7.9% mIoU, 13.4% SSIM, and 7.6% RMSE,
  respectively, for segmentation mask, line-art edge, and depth conditions.
paper_authors._id: 6618975e4f47c19ab9b6b891
paper_authors.name: Ming Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-16T00:02:52.951Z
paper_authors._id: 637f0eb22438d7485b8ef5d7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637f0eb22438d7485b8ef5d7/70h7dekqj7LuBobOXckmJ.jpeg
paper_authors.fullname: Ming Li
paper_authors.type: user
paper_authors.user: limingcv
paper_authors._id: 6618975e4f47c19ab9b6b892
paper_authors.name: Taojiannan Yang
paper_authors._id: 6618975e4f47c19ab9b6b893
paper_authors.name: Huafeng Kuang
paper_authors._id: 6618975e4f47c19ab9b6b894
paper_authors.name: Jie Wu
paper_authors._id: 6618975e4f47c19ab9b6b895
paper_authors.name: Zhaoning Wang
paper_authors._id: 6618975e4f47c19ab9b6b896
paper_authors.name: Xuefeng Xiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T11:55:00.608Z
paper_authors._id: 646b7f71df2609a541c1ab9f
paper_authors.avatarUrl: /avatars/48b82e5fd9b06f41ff825507c36816cd.svg
paper_authors.fullname: Xuefeng Xiao
paper_authors.type: user
paper_authors.user: xiaoxuefeng
paper_authors._id: 6618975e4f47c19ab9b6b897
paper_authors.name: Chen Chen
paper_title: ControlNet++: Improving Conditional Controls with Efficient Consistency
    Feedback
paper_summary: To enhance the controllability of text-to-image diffusion models, existing
  efforts like ControlNet incorporated image-based conditional controls. In this
  paper, we reveal that existing methods still face significant challenges in
  generating images that align with the image conditional controls. To this end,
  we propose ControlNet++, a novel approach that improves controllable generation
  by explicitly optimizing pixel-level cycle consistency between generated images
  and conditional controls. Specifically, for an input conditional control, we
  use a pre-trained discriminative reward model to extract the corresponding
  condition of the generated images, and then optimize the consistency loss
  between the input conditional control and extracted condition. A
  straightforward implementation would be generating images from random noises
  and then calculating the consistency loss, but such an approach requires
  storing gradients for multiple sampling timesteps, leading to considerable time
  and memory costs. To address this, we introduce an efficient reward strategy
  that deliberately disturbs the input images by adding noise, and then uses the
  single-step denoised images for reward fine-tuning. This avoids the extensive
  costs associated with image sampling, allowing for more efficient reward
  fine-tuning. Extensive experiments show that ControlNet++ significantly
  improves controllability under various conditional controls. For example, it
  achieves improvements over ControlNet by 7.9% mIoU, 13.4% SSIM, and 7.6% RMSE,
  respectively, for segmentation mask, line-art edge, and depth conditions.
paper_ai_summary: ControlNet++ enhances text-to-image generation controllability through pixel-level cycle consistency optimization using an efficient noise-disturbed reward strategy.
paper_ai_keywords: ControlNet text-to-image diffusion models pixel-level cycle consistency discriminative reward model consistency loss single-step denoised images reward fine-tuning segmentation mask line-art edge depth conditions mIoU SSIM RMSE