title: OmniBooth: Learning Latent Control for Image Synthesis with Multi-modal
    Instruction
summary: We present OmniBooth, an image generation framework that enables spatial
  control with instance-level multi-modal customization. For all instances, the
  multimodal instruction can be described through text prompts or image
  references. Given a set of user-defined masks and associated text or image
  guidance, our objective is to generate an image, where multiple objects are
  positioned at specified coordinates and their attributes are precisely aligned
  with the corresponding guidance. This approach significantly expands the scope
  of text-to-image generation, and elevates it to a more versatile and practical
  dimension in controllability. In this paper, our core contribution lies in the
  proposed latent control signals, a high-dimensional spatial feature that
  provides a unified representation to integrate the spatial, textual, and image
  conditions seamlessly. The text condition extends ControlNet to provide
  instance-level open-vocabulary generation. The image condition further enables
  fine-grained control with personalized identity. In practice, our method
  empowers users with more flexibility in controllable generation, as users can
  choose multi-modal conditions from text or images as needed. Furthermore,
  thorough experiments demonstrate our enhanced performance in image synthesis
  fidelity and alignment across different tasks and datasets. Project page:
  https://len-li.github.io/omnibooth-web/
paper_authors._id: 6704c3af5788bbf517ed087f
paper_authors.name: Leheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-08T09:31:52.781Z
paper_authors._id: 6343e37f73b4f9cedab1c846
paper_authors.avatarUrl: /avatars/2638af4626e8a4e3a95f845b94ad94f6.svg
paper_authors.fullname: Leheng Li
paper_authors.type: user
paper_authors.user: lilelife
paper_authors._id: 6704c3af5788bbf517ed0880
paper_authors.name: Weichao Qiu
paper_authors._id: 6704c3af5788bbf517ed0881
paper_authors.name: Xu Yan
paper_authors._id: 6704c3af5788bbf517ed0882
paper_authors.name: Jing He
paper_authors._id: 6704c3af5788bbf517ed0883
paper_authors.name: Kaiqiang Zhou
paper_authors._id: 6704c3af5788bbf517ed0884
paper_authors.name: Yingjie Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-08T09:32:20.081Z
paper_authors._id: 65f41e75a76fa48877923f9e
paper_authors.avatarUrl: /avatars/b93f42ebc74bc59974f957b551bf1b8d.svg
paper_authors.fullname: Yingjie Cai
paper_authors.type: user
paper_authors.user: Evenc
paper_authors._id: 6704c3af5788bbf517ed0885
paper_authors.name: Qing Lian
paper_authors._id: 6704c3af5788bbf517ed0886
paper_authors.name: Bingbing Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-08T09:32:41.707Z
paper_authors._id: 656957ecef451c6ba6480b92
paper_authors.avatarUrl: /avatars/fe8b8e68d0c1ea67063848ebfebb1965.svg
paper_authors.fullname: bingbing liu
paper_authors.type: user
paper_authors.user: bingbl
paper_authors._id: 6704c3af5788bbf517ed0887
paper_authors.name: Ying-Cong Chen
paper_title: OmniBooth: Learning Latent Control for Image Synthesis with Multi-modal
    Instruction
paper_summary: We present OmniBooth, an image generation framework that enables spatial
  control with instance-level multi-modal customization. For all instances, the
  multimodal instruction can be described through text prompts or image
  references. Given a set of user-defined masks and associated text or image
  guidance, our objective is to generate an image, where multiple objects are
  positioned at specified coordinates and their attributes are precisely aligned
  with the corresponding guidance. This approach significantly expands the scope
  of text-to-image generation, and elevates it to a more versatile and practical
  dimension in controllability. In this paper, our core contribution lies in the
  proposed latent control signals, a high-dimensional spatial feature that
  provides a unified representation to integrate the spatial, textual, and image
  conditions seamlessly. The text condition extends ControlNet to provide
  instance-level open-vocabulary generation. The image condition further enables
  fine-grained control with personalized identity. In practice, our method
  empowers users with more flexibility in controllable generation, as users can
  choose multi-modal conditions from text or images as needed. Furthermore,
  thorough experiments demonstrate our enhanced performance in image synthesis
  fidelity and alignment across different tasks and datasets. Project page:
  https://len-li.github.io/omnibooth-web/
paper_ai_keywords: latent control signals spatial feature unified representation ControlNet open-vocabulary generation fine-grained control personalized identity spatial control instance-level multi-modal customization multi-modal instruction text prompts image references user-defined masks image synthesis fidelity text-to-image Label