title: UNIMO-G: Unified Image Generation through Multimodal Conditional
    Diffusion
summary: Existing text-to-image diffusion models primarily generate images from text
  prompts. However, the inherent conciseness of textual descriptions poses
  challenges in faithfully synthesizing images with intricate details, such as
  specific entities or scenes. This paper presents UNIMO-G, a simple
  multimodal conditional diffusion framework that operates on multimodal prompts
  with interleaved textual and visual inputs, which demonstrates a unified
  ability for both text-driven and subject-driven image generation. UNIMO-G
  comprises two core components: a Multimodal Large Language Model (MLLM) for
  encoding multimodal prompts, and a conditional denoising diffusion network for
  generating images based on the encoded multimodal input. We leverage a
  two-stage training strategy to effectively train the framework: firstly
  pre-training on large-scale text-image pairs to develop conditional image
  generation capabilities, and then instruction tuning with multimodal prompts to
  achieve unified image generation proficiency. A well-designed data processing
  pipeline involving language grounding and image segmentation is employed to
  construct multi-modal prompts. UNIMO-G excels in both text-to-image generation
  and zero-shot subject-driven synthesis, and is notably effective in generating
  high-fidelity images from complex multimodal prompts involving multiple image
  entities.
paper_authors._id: 65b1cf4cfcead433ff15b9a4
paper_authors.name: Wei Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-26T09:52:51.136Z
paper_authors._id: 63044e025c70c21d0eaf08bc
paper_authors.avatarUrl: /avatars/a2d39973d7fbcbe9d4cce5648b3149c2.svg
paper_authors.fullname: Wei Li
paper_authors.type: user
paper_authors.user: Wiley085
paper_authors._id: 65b1cf4cfcead433ff15b9a5
paper_authors.name: Xue Xu
paper_authors._id: 65b1cf4cfcead433ff15b9a6
paper_authors.name: Jiachen Liu
paper_authors._id: 65b1cf4cfcead433ff15b9a7
paper_authors.name: Xinyan Xiao
paper_title: UNIMO-G: Unified Image Generation through Multimodal Conditional
    Diffusion
paper_summary: Existing text-to-image diffusion models primarily generate images from text
  prompts. However, the inherent conciseness of textual descriptions poses
  challenges in faithfully synthesizing images with intricate details, such as
  specific entities or scenes. This paper presents UNIMO-G, a simple
  multimodal conditional diffusion framework that operates on multimodal prompts
  with interleaved textual and visual inputs, which demonstrates a unified
  ability for both text-driven and subject-driven image generation. UNIMO-G
  comprises two core components: a Multimodal Large Language Model (MLLM) for
  encoding multimodal prompts, and a conditional denoising diffusion network for
  generating images based on the encoded multimodal input. We leverage a
  two-stage training strategy to effectively train the framework: firstly
  pre-training on large-scale text-image pairs to develop conditional image
  generation capabilities, and then instruction tuning with multimodal prompts to
  achieve unified image generation proficiency. A well-designed data processing
  pipeline involving language grounding and image segmentation is employed to
  construct multi-modal prompts. UNIMO-G excels in both text-to-image generation
  and zero-shot subject-driven synthesis, and is notably effective in generating
  high-fidelity images from complex multimodal prompts involving multiple image
  entities.
paper_ai_summary: A multimodal conditional diffusion framework named UNIMO-G generates images from both text and visual prompts, capable of high-fidelity synthesis of intricate details and diverse image entities.
paper_ai_keywords: Multimodal Large Language Model conditional denoising diffusion network text-driven subject-driven image generation large-scale text-image pairs language grounding image segmentation multimodal prompts zero-shot synthesis high-fidelity images