title: CCM: Adding Conditional Controls to Text-to-Image Consistency Models
summary: Consistency Models (CMs) have showed a promise in creating visual content
  efficiently and with high quality. However, the way to add new conditional
  controls to the pretrained CMs has not been explored. In this technical report,
  we consider alternative strategies for adding ControlNet-like conditional
  control to CMs and present three significant findings. 1) ControlNet trained
  for diffusion models (DMs) can be directly applied to CMs for high-level
  semantic controls but struggles with low-level detail and realism control. 2)
  CMs serve as an independent class of generative models, based on which
  ControlNet can be trained from scratch using Consistency Training proposed by
  Song et al. 3) A lightweight adapter can be jointly optimized under multiple
  conditions through Consistency Training, allowing for the swift transfer of
  DMs-based ControlNet to CMs. We study these three solutions across various
  conditional controls, including edge, depth, human pose, low-resolution image
  and masked image with text-to-image latent consistency models.
paper_authors._id: 65793e78c17736d75f832c85
paper_authors.name: Jie Xiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-13T09:20:18.435Z
paper_authors._id: 646b2767ec9a61e8717a41a7
paper_authors.avatarUrl: /avatars/5503aefdfd6cddd6f83bf9fbcced4c90.svg
paper_authors.fullname: jiexiao
paper_authors.type: user
paper_authors.user: jiexiao
paper_authors._id: 65793e78c17736d75f832c86
paper_authors.name: Kai Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-13T14:39:43.277Z
paper_authors._id: 643d06a1043254ce109108a4
paper_authors.avatarUrl: /avatars/5d58816cf76f2648af05b6a293de834d.svg
paper_authors.fullname: kai zhu
paper_authors.type: user
paper_authors.user: kaizhu111
paper_authors._id: 65793e78c17736d75f832c87
paper_authors.name: Han Zhang
paper_authors._id: 65793e78c17736d75f832c88
paper_authors.name: Zhiheng Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-18T08:11:19.586Z
paper_authors._id: 6479925ab77e18dbf640bd67
paper_authors.avatarUrl: /avatars/bb52ecd22ca4b49157f8668be35409e7.svg
paper_authors.fullname: Zhiheng Liu
paper_authors.type: user
paper_authors.user: Johanan0528
paper_authors._id: 65793e78c17736d75f832c89
paper_authors.name: Yujun Shen
paper_authors._id: 65793e78c17736d75f832c8a
paper_authors.name: Yu Liu
paper_authors._id: 65793e78c17736d75f832c8b
paper_authors.name: Xueyang Fu
paper_authors._id: 65793e78c17736d75f832c8c
paper_authors.name: Zheng-Jun Zha
paper_title: CCM: Adding Conditional Controls to Text-to-Image Consistency Models
paper_summary: Consistency Models (CMs) have showed a promise in creating visual content
  efficiently and with high quality. However, the way to add new conditional
  controls to the pretrained CMs has not been explored. In this technical report,
  we consider alternative strategies for adding ControlNet-like conditional
  control to CMs and present three significant findings. 1) ControlNet trained
  for diffusion models (DMs) can be directly applied to CMs for high-level
  semantic controls but struggles with low-level detail and realism control. 2)
  CMs serve as an independent class of generative models, based on which
  ControlNet can be trained from scratch using Consistency Training proposed by
  Song et al. 3) A lightweight adapter can be jointly optimized under multiple
  conditions through Consistency Training, allowing for the swift transfer of
  DMs-based ControlNet to CMs. We study these three solutions across various
  conditional controls, including edge, depth, human pose, low-resolution image
  and masked image with text-to-image latent consistency models.
paper_ai_keywords: Consistency Models ControlNet diffusion models high-level semantic controls low-level detail realism control ControlNet-like conditional control lightweight adapter Consistency Training text-to-image latent consistency models