title: MIGE: A Unified Framework for Multimodal Instruction-Based Image
    Generation and Editing
summary: Despite significant progress in diffusion-based image generation,
  subject-driven generation and instruction-based editing remain challenging.
  Existing methods typically treat them separately, struggling with limited
  high-quality data and poor generalization. However, both tasks require
  capturing complex visual variations while maintaining consistency between
  inputs and outputs. Therefore, we propose MIGE, a unified framework that
  standardizes task representations using multimodal instructions. It treats
  subject-driven generation as creation on a blank canvas and instruction-based
  editing as modification of an existing image, establishing a shared
  input-output formulation. MIGE introduces a novel multimodal encoder that maps
  free-form multimodal instructions into a unified vision-language space,
  integrating visual and semantic features through a feature fusion
  mechanism.This unification enables joint training of both tasks, providing two
  key advantages: (1) Cross-Task Enhancement: By leveraging shared visual and
  semantic representations, joint training improves instruction adherence and
  visual consistency in both subject-driven generation and instruction-based
  editing. (2) Generalization: Learning in a unified format facilitates
  cross-task knowledge transfer, enabling MIGE to generalize to novel
  compositional tasks, including instruction-based subject-driven editing.
  Experiments show that MIGE excels in both subject-driven generation and
  instruction-based editing while setting a state-of-the-art in the new task of
  instruction-based subject-driven editing. Code and model have been publicly
  available at https://github.com/Eureka-Maggie/MIGE.
paper_authors._id: 67c5aad632a7208c9ae1d020
paper_authors.name: Xueyun Tian
paper_authors._id: 67c5aad632a7208c9ae1d021
paper_authors.name: Wei Li
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-03-03T13:12:57.839Z
paper_authors._id: 63044e025c70c21d0eaf08bc
paper_authors.avatarUrl: /avatars/a2d39973d7fbcbe9d4cce5648b3149c2.svg
paper_authors.fullname: Wei Li
paper_authors.type: user
paper_authors.user: Wiley085
paper_authors._id: 67c5aad632a7208c9ae1d022
paper_authors.name: Bingbing Xu
paper_authors._id: 67c5aad632a7208c9ae1d023
paper_authors.name: Yige Yuan
paper_authors._id: 67c5aad632a7208c9ae1d024
paper_authors.name: Yuanzhuo Wang
paper_authors._id: 67c5aad632a7208c9ae1d025
paper_authors.name: Huawei Shen
paper_title: MIGE: A Unified Framework for Multimodal Instruction-Based Image
    Generation and Editing
paper_summary: Despite significant progress in diffusion-based image generation,
  subject-driven generation and instruction-based editing remain challenging.
  Existing methods typically treat them separately, struggling with limited
  high-quality data and poor generalization. However, both tasks require
  capturing complex visual variations while maintaining consistency between
  inputs and outputs. Therefore, we propose MIGE, a unified framework that
  standardizes task representations using multimodal instructions. It treats
  subject-driven generation as creation on a blank canvas and instruction-based
  editing as modification of an existing image, establishing a shared
  input-output formulation. MIGE introduces a novel multimodal encoder that maps
  free-form multimodal instructions into a unified vision-language space,
  integrating visual and semantic features through a feature fusion
  mechanism.This unification enables joint training of both tasks, providing two
  key advantages: (1) Cross-Task Enhancement: By leveraging shared visual and
  semantic representations, joint training improves instruction adherence and
  visual consistency in both subject-driven generation and instruction-based
  editing. (2) Generalization: Learning in a unified format facilitates
  cross-task knowledge transfer, enabling MIGE to generalize to novel
  compositional tasks, including instruction-based subject-driven editing.
  Experiments show that MIGE excels in both subject-driven generation and
  instruction-based editing while setting a state-of-the-art in the new task of
  instruction-based subject-driven editing. Code and model have been publicly
  available at https://github.com/Eureka-Maggie/MIGE.
paper_ai_keywords: diffusion-based image generation subject-driven generation instruction-based editing multimodal instructions unified vision-language space feature fusion mechanism cross-task enhancement instruction adherence visual consistency cross-task knowledge transfer instruction-based subject-driven editing