CCM: Adding Conditional Controls to Text-to-Image Consistency Models
Consistency Models (CMs) have showed a promise in creating visual content
efficiently and with high quality. However, the way to add new conditional
controls to the pretrained CMs has not been explored. In this technical report,
we consider alternative strategies for adding ControlNet-like conditional
control to CMs and present three significant findings. 1) ControlNet trained
for diffusion models (DMs) can be directly applied to CMs for high-level
semantic controls but struggles with low-level detail and realism control. 2)
CMs serve as an independent class of generative models, based on which
ControlNet can be trained from scratch using Consistency Training proposed by
Song et al. 3) A lightweight adapter can be jointly optimized under multiple
conditions through Consistency Training, allowing for the swift transfer of
DMs-based ControlNet to CMs. We study these three solutions across various
conditional controls, including edge, depth, human pose, low-resolution image
and masked image with text-to-image latent consistency models.
65793e78c17736d75f832c85
Jie Xiao
admin_assigned
2023-12-13T09:20:18.435Z
646b2767ec9a61e8717a41a7
/avatars/5503aefdfd6cddd6f83bf9fbcced4c90.svg
jiexiao
user
jiexiao
65793e78c17736d75f832c86
Kai Zhu
claimed_verified
2023-12-13T14:39:43.277Z
643d06a1043254ce109108a4
/avatars/5d58816cf76f2648af05b6a293de834d.svg
kai zhu
user
kaizhu111
65793e78c17736d75f832c87
Han Zhang
65793e78c17736d75f832c88
Zhiheng Liu
claimed_verified
2024-04-18T08:11:19.586Z
6479925ab77e18dbf640bd67
/avatars/bb52ecd22ca4b49157f8668be35409e7.svg
Zhiheng Liu
user
Johanan0528
65793e78c17736d75f832c89
Yujun Shen
65793e78c17736d75f832c8a
Yu Liu
65793e78c17736d75f832c8b
Xueyang Fu
65793e78c17736d75f832c8c
Zheng-Jun Zha
CCM: Adding Conditional Controls to Text-to-Image Consistency Models
Consistency Models (CMs) have showed a promise in creating visual content
efficiently and with high quality. However, the way to add new conditional
controls to the pretrained CMs has not been explored. In this technical report,
we consider alternative strategies for adding ControlNet-like conditional
control to CMs and present three significant findings. 1) ControlNet trained
for diffusion models (DMs) can be directly applied to CMs for high-level
semantic controls but struggles with low-level detail and realism control. 2)
CMs serve as an independent class of generative models, based on which
ControlNet can be trained from scratch using Consistency Training proposed by
Song et al. 3) A lightweight adapter can be jointly optimized under multiple
conditions through Consistency Training, allowing for the swift transfer of
DMs-based ControlNet to CMs. We study these three solutions across various
conditional controls, including edge, depth, human pose, low-resolution image
and masked image with text-to-image latent consistency models.
Research explores methods to integrate ControlNet-like conditional control into Consistency Models, finding approaches for high-level semantic control, alternative training methods, and lightweight adapters.
Consistency Models ControlNet diffusion models high-level semantic controls low-level detail control Consistency Training lightweight adapter edge depth human pose low-resolution image masked image text-to-image latent consistency models