OmniBooth: Learning Latent Control for Image Synthesis with Multi-modal
  Instruction
We present OmniBooth, an image generation framework that enables spatial
control with instance-level multi-modal customization. For all instances, the
multimodal instruction can be described through text prompts or image
references. Given a set of user-defined masks and associated text or image
guidance, our objective is to generate an image, where multiple objects are
positioned at specified coordinates and their attributes are precisely aligned
with the corresponding guidance. This approach significantly expands the scope
of text-to-image generation, and elevates it to a more versatile and practical
dimension in controllability. In this paper, our core contribution lies in the
proposed latent control signals, a high-dimensional spatial feature that
provides a unified representation to integrate the spatial, textual, and image
conditions seamlessly. The text condition extends ControlNet to provide
instance-level open-vocabulary generation. The image condition further enables
fine-grained control with personalized identity. In practice, our method
empowers users with more flexibility in controllable generation, as users can
choose multi-modal conditions from text or images as needed. Furthermore,
thorough experiments demonstrate our enhanced performance in image synthesis
fidelity and alignment across different tasks and datasets. Project page:
https://len-li.github.io/omnibooth-web/
6704c3af5788bbf517ed087f
Leheng Li
admin_assigned
2024-10-08T09:31:52.781Z
6343e37f73b4f9cedab1c846
/avatars/2638af4626e8a4e3a95f845b94ad94f6.svg
Leheng Li
user
lilelife
6704c3af5788bbf517ed0880
Weichao Qiu
6704c3af5788bbf517ed0881
Xu Yan
6704c3af5788bbf517ed0882
Jing He
6704c3af5788bbf517ed0883
Kaiqiang Zhou
6704c3af5788bbf517ed0884
Yingjie Cai
admin_assigned
2024-10-08T09:32:20.081Z
65f41e75a76fa48877923f9e
/avatars/b93f42ebc74bc59974f957b551bf1b8d.svg
Yingjie Cai
user
Evenc
6704c3af5788bbf517ed0885
Qing Lian
6704c3af5788bbf517ed0886
Bingbing Liu
admin_assigned
2024-10-08T09:32:41.707Z
656957ecef451c6ba6480b92
/avatars/fe8b8e68d0c1ea67063848ebfebb1965.svg
bingbing liu
user
bingbl
6704c3af5788bbf517ed0887
Ying-Cong Chen
OmniBooth: Learning Latent Control for Image Synthesis with Multi-modal
  Instruction
We present OmniBooth, an image generation framework that enables spatial
control with instance-level multi-modal customization. For all instances, the
multimodal instruction can be described through text prompts or image
references. Given a set of user-defined masks and associated text or image
guidance, our objective is to generate an image, where multiple objects are
positioned at specified coordinates and their attributes are precisely aligned
with the corresponding guidance. This approach significantly expands the scope
of text-to-image generation, and elevates it to a more versatile and practical
dimension in controllability. In this paper, our core contribution lies in the
proposed latent control signals, a high-dimensional spatial feature that
provides a unified representation to integrate the spatial, textual, and image
conditions seamlessly. The text condition extends ControlNet to provide
instance-level open-vocabulary generation. The image condition further enables
fine-grained control with personalized identity. In practice, our method
empowers users with more flexibility in controllable generation, as users can
choose multi-modal conditions from text or images as needed. Furthermore,
thorough experiments demonstrate our enhanced performance in image synthesis
fidelity and alignment across different tasks and datasets. Project page:
https://len-li.github.io/omnibooth-web/
OmniBooth is an image generation framework that uses latent control signals to enable flexible spatial control and instance-level customization using text prompts or image references.
OmniBooth latent control signals spatial feature text-to-image generation ControlNet open-vocabulary generation fine-grained control image synthesis fidelity