title: Cocktail: Mixing Multi-Modality Controls for Text-Conditional Image
    Generation
summary: Text-conditional diffusion models are able to generate high-fidelity images
  with diverse contents. However, linguistic representations frequently exhibit
  ambiguous descriptions of the envisioned objective imagery, requiring the
  incorporation of additional control signals to bolster the efficacy of
  text-guided diffusion models. In this work, we propose Cocktail, a pipeline to
  mix various modalities into one embedding, amalgamated with a generalized
  ControlNet (gControlNet), a controllable normalisation (ControlNorm), and a
  spatial guidance sampling method, to actualize multi-modal and
  spatially-refined control for text-conditional diffusion models. Specifically,
  we introduce a hyper-network gControlNet, dedicated to the alignment and
  infusion of the control signals from disparate modalities into the pre-trained
  diffusion model. gControlNet is capable of accepting flexible modality signals,
  encompassing the simultaneous reception of any combination of modality signals,
  or the supplementary fusion of multiple modality signals. The control signals
  are then fused and injected into the backbone model according to our proposed
  ControlNorm. Furthermore, our advanced spatial guidance sampling methodology
  proficiently incorporates the control signal into the designated region,
  thereby circumventing the manifestation of undesired objects within the
  generated image. We demonstrate the results of our method in controlling
  various modalities, proving high-quality synthesis and fidelity to multiple
  external signals.
paper_authors._id: 647947835fa05d12dc348cf2
paper_authors.name: Minghui Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-01T05:39:07.174Z
paper_authors._id: 630b77f68b327c7b8b98c409
paper_authors.avatarUrl: /avatars/2fe95c9ac95f34dbb031f2ec018f68b0.svg
paper_authors.fullname: Minghui Hu
paper_authors.type: user
paper_authors.user: h1t
paper_authors._id: 647947835fa05d12dc348cf3
paper_authors.name: Jianbin Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-07T13:03:22.267Z
paper_authors._id: 641870135d6f3d15c64d074e
paper_authors.avatarUrl: /avatars/35f4b33fd1d4db326e4ee4300b26db72.svg
paper_authors.fullname: Jianbin Zheng
paper_authors.type: user
paper_authors.user: jabir-zheng
paper_authors._id: 647947835fa05d12dc348cf4
paper_authors.name: Daqing Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-05T10:06:53.304Z
paper_authors._id: 63ad3c1c6ee60ca58a4083d0
paper_authors.avatarUrl: /avatars/7e3a5cc39c9508fab3330d92a327d6e2.svg
paper_authors.fullname: Daqing Liu
paper_authors.type: user
paper_authors.user: daqingliu
paper_authors._id: 647947835fa05d12dc348cf5
paper_authors.name: Chuanxia Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-05T10:07:40.664Z
paper_authors._id: 635e16911e2d8c6673e39785
paper_authors.avatarUrl: /avatars/905bccbe96b72862515ae4c27e7de642.svg
paper_authors.fullname: chuanxia
paper_authors.type: user
paper_authors.user: lyndonzheng
paper_authors._id: 647947835fa05d12dc348cf6
paper_authors.name: Chaoyue Wang
paper_authors._id: 647947835fa05d12dc348cf7
paper_authors.name: Dacheng Tao
paper_authors._id: 647947835fa05d12dc348cf8
paper_authors.name: Tat-Jen Cham
paper_title: Cocktail: Mixing Multi-Modality Controls for Text-Conditional Image
    Generation
paper_summary: Text-conditional diffusion models are able to generate high-fidelity images
  with diverse contents. However, linguistic representations frequently exhibit
  ambiguous descriptions of the envisioned objective imagery, requiring the
  incorporation of additional control signals to bolster the efficacy of
  text-guided diffusion models. In this work, we propose Cocktail, a pipeline to
  mix various modalities into one embedding, amalgamated with a generalized
  ControlNet (gControlNet), a controllable normalisation (ControlNorm), and a
  spatial guidance sampling method, to actualize multi-modal and
  spatially-refined control for text-conditional diffusion models. Specifically,
  we introduce a hyper-network gControlNet, dedicated to the alignment and
  infusion of the control signals from disparate modalities into the pre-trained
  diffusion model. gControlNet is capable of accepting flexible modality signals,
  encompassing the simultaneous reception of any combination of modality signals,
  or the supplementary fusion of multiple modality signals. The control signals
  are then fused and injected into the backbone model according to our proposed
  ControlNorm. Furthermore, our advanced spatial guidance sampling methodology
  proficiently incorporates the control signal into the designated region,
  thereby circumventing the manifestation of undesired objects within the
  generated image. We demonstrate the results of our method in controlling
  various modalities, proving high-quality synthesis and fidelity to multiple
  external signals.
paper_ai_summary: Cocktail enhances text-conditional diffusion models with a multi-modal and spatially-refined control system, including gControlNet, ControlNorm, and spatial guidance sampling, to improve image synthesis quality and fidelity.
paper_ai_keywords: text-conditional diffusion models gControlNet Controllable normalisation ControlNorm spatial guidance sampling