Cocktail: Mixing Multi-Modality Controls for Text-Conditional Image
  Generation
Text-conditional diffusion models are able to generate high-fidelity images
with diverse contents. However, linguistic representations frequently exhibit
ambiguous descriptions of the envisioned objective imagery, requiring the
incorporation of additional control signals to bolster the efficacy of
text-guided diffusion models. In this work, we propose Cocktail, a pipeline to
mix various modalities into one embedding, amalgamated with a generalized
ControlNet (gControlNet), a controllable normalisation (ControlNorm), and a
spatial guidance sampling method, to actualize multi-modal and
spatially-refined control for text-conditional diffusion models. Specifically,
we introduce a hyper-network gControlNet, dedicated to the alignment and
infusion of the control signals from disparate modalities into the pre-trained
diffusion model. gControlNet is capable of accepting flexible modality signals,
encompassing the simultaneous reception of any combination of modality signals,
or the supplementary fusion of multiple modality signals. The control signals
are then fused and injected into the backbone model according to our proposed
ControlNorm. Furthermore, our advanced spatial guidance sampling methodology
proficiently incorporates the control signal into the designated region,
thereby circumventing the manifestation of undesired objects within the
generated image. We demonstrate the results of our method in controlling
various modalities, proving high-quality synthesis and fidelity to multiple
external signals.
647947835fa05d12dc348cf2
Minghui Hu
claimed_verified
2024-03-01T05:39:07.174Z
630b77f68b327c7b8b98c409
/avatars/2fe95c9ac95f34dbb031f2ec018f68b0.svg
Minghui Hu
user
h1t
647947835fa05d12dc348cf3
Jianbin Zheng
admin_assigned
2023-06-07T13:03:22.267Z
641870135d6f3d15c64d074e
/avatars/35f4b33fd1d4db326e4ee4300b26db72.svg
Jianbin Zheng
user
jabir-zheng
647947835fa05d12dc348cf4
Daqing Liu
claimed_verified
2023-06-05T10:06:53.304Z
63ad3c1c6ee60ca58a4083d0
/avatars/7e3a5cc39c9508fab3330d92a327d6e2.svg
Daqing Liu
user
daqingliu
647947835fa05d12dc348cf5
Chuanxia Zheng
claimed_verified
2023-06-05T10:07:40.664Z
635e16911e2d8c6673e39785
/avatars/905bccbe96b72862515ae4c27e7de642.svg
chuanxia
user
lyndonzheng
647947835fa05d12dc348cf6
Chaoyue Wang
647947835fa05d12dc348cf7
Dacheng Tao
647947835fa05d12dc348cf8
Tat-Jen Cham
Cocktail: Mixing Multi-Modality Controls for Text-Conditional Image
  Generation
Text-conditional diffusion models are able to generate high-fidelity images
with diverse contents. However, linguistic representations frequently exhibit
ambiguous descriptions of the envisioned objective imagery, requiring the
incorporation of additional control signals to bolster the efficacy of
text-guided diffusion models. In this work, we propose Cocktail, a pipeline to
mix various modalities into one embedding, amalgamated with a generalized
ControlNet (gControlNet), a controllable normalisation (ControlNorm), and a
spatial guidance sampling method, to actualize multi-modal and
spatially-refined control for text-conditional diffusion models. Specifically,
we introduce a hyper-network gControlNet, dedicated to the alignment and
infusion of the control signals from disparate modalities into the pre-trained
diffusion model. gControlNet is capable of accepting flexible modality signals,
encompassing the simultaneous reception of any combination of modality signals,
or the supplementary fusion of multiple modality signals. The control signals
are then fused and injected into the backbone model according to our proposed
ControlNorm. Furthermore, our advanced spatial guidance sampling methodology
proficiently incorporates the control signal into the designated region,
thereby circumventing the manifestation of undesired objects within the
generated image. We demonstrate the results of our method in controlling
various modalities, proving high-quality synthesis and fidelity to multiple
external signals.
Cocktail enhances text-conditional diffusion models with a multi-modal and spatially-refined control system, including gControlNet, ControlNorm, and spatial guidance sampling, to improve image synthesis quality and fidelity.
text-conditional diffusion models gControlNet Controllable normalisation ControlNorm spatial guidance sampling