title: Instruct-Imagen: Image Generation with Multi-modal Instruction
summary: This paper presents instruct-imagen, a model that tackles heterogeneous image
  generation tasks and generalizes across unseen tasks. We introduce *multi-modal
  instruction* for image generation, a task representation articulating a range
  of generation intents with precision. It uses natural language to amalgamate
  disparate modalities (e.g., text, edge, style, subject, etc.), such that
  abundant generation intents can be standardized in a uniform format.
    We then build instruct-imagen by fine-tuning a pre-trained text-to-image
  diffusion model with a two-stage framework. First, we adapt the model using the
  retrieval-augmented training, to enhance model's capabilities to ground its
  generation on external multimodal context. Subsequently, we fine-tune the
  adapted model on diverse image generation tasks that requires vision-language
  understanding (e.g., subject-driven generation, etc.), each paired with a
  multi-modal instruction encapsulating the task's essence. Human evaluation on
  various image generation datasets reveals that instruct-imagen matches or
  surpasses prior task-specific models in-domain and demonstrates promising
  generalization to unseen and more complex tasks.
paper_authors._id: 659772f1ce6e887177e45dfa
paper_authors.name: Hexiang Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-12T08:06:43.586Z
paper_authors._id: 643441ccd55dea2d0ec2c309
paper_authors.avatarUrl: /avatars/82e99d445e2b513ad7270fa852adbcbb.svg
paper_authors.fullname: Hexiang Hu
paper_authors.type: user
paper_authors.user: hexianghu
paper_authors._id: 659772f1ce6e887177e45dfb
paper_authors.name: Kelvin C. K. Chan
paper_authors._id: 659772f1ce6e887177e45dfc
paper_authors.name: Yu-Chuan Su
paper_authors._id: 659772f1ce6e887177e45dfd
paper_authors.name: Wenhu Chen
paper_authors._id: 659772f1ce6e887177e45dfe
paper_authors.name: Yandong Li
paper_authors._id: 659772f1ce6e887177e45dff
paper_authors.name: Kihyuk Sohn
paper_authors._id: 659772f1ce6e887177e45e00
paper_authors.name: Yang Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-08T08:48:00.827Z
paper_authors._id: 631ba6aaf6bc4be4a653b2c7
paper_authors.avatarUrl: /avatars/c026f47a042711127d3b8431f6217f55.svg
paper_authors.fullname: Yang Zhao
paper_authors.type: user
paper_authors.user: YangNaruto
paper_authors._id: 659772f1ce6e887177e45e01
paper_authors.name: Xue Ben
paper_authors._id: 659772f1ce6e887177e45e02
paper_authors.name: Boqing Gong
paper_authors._id: 659772f1ce6e887177e45e03
paper_authors.name: William Cohen
paper_authors._id: 659772f1ce6e887177e45e04
paper_authors.name: Ming-Wei Chang
paper_authors._id: 659772f1ce6e887177e45e05
paper_authors.name: Xuhui Jia
paper_title: Instruct-Imagen: Image Generation with Multi-modal Instruction
paper_summary: This paper presents instruct-imagen, a model that tackles heterogeneous image
  generation tasks and generalizes across unseen tasks. We introduce *multi-modal
  instruction* for image generation, a task representation articulating a range
  of generation intents with precision. It uses natural language to amalgamate
  disparate modalities (e.g., text, edge, style, subject, etc.), such that
  abundant generation intents can be standardized in a uniform format.
    We then build instruct-imagen by fine-tuning a pre-trained text-to-image
  diffusion model with a two-stage framework. First, we adapt the model using the
  retrieval-augmented training, to enhance model's capabilities to ground its
  generation on external multimodal context. Subsequently, we fine-tune the
  adapted model on diverse image generation tasks that requires vision-language
  understanding (e.g., subject-driven generation, etc.), each paired with a
  multi-modal instruction encapsulating the task's essence. Human evaluation on
  various image generation datasets reveals that instruct-imagen matches or
  surpasses prior task-specific models in-domain and demonstrates promising
  generalization to unseen and more complex tasks.
paper_ai_summary: Instruct-Imagen achieves robust and generalized image generation through fine-tuning a pre-trained text-to-image diffusion model with multi-modal instructions and retrieval-augmented training.
paper_ai_keywords: multi-modal instruction text-to-image diffusion model retrieval-augmented training subject-driven generation vision-language understanding