MIGE: A Unified Framework for Multimodal Instruction-Based Image
  Generation and Editing
Despite significant progress in diffusion-based image generation,
subject-driven generation and instruction-based editing remain challenging.
Existing methods typically treat them separately, struggling with limited
high-quality data and poor generalization. However, both tasks require
capturing complex visual variations while maintaining consistency between
inputs and outputs. Therefore, we propose MIGE, a unified framework that
standardizes task representations using multimodal instructions. It treats
subject-driven generation as creation on a blank canvas and instruction-based
editing as modification of an existing image, establishing a shared
input-output formulation. MIGE introduces a novel multimodal encoder that maps
free-form multimodal instructions into a unified vision-language space,
integrating visual and semantic features through a feature fusion
mechanism.This unification enables joint training of both tasks, providing two
key advantages: (1) Cross-Task Enhancement: By leveraging shared visual and
semantic representations, joint training improves instruction adherence and
visual consistency in both subject-driven generation and instruction-based
editing. (2) Generalization: Learning in a unified format facilitates
cross-task knowledge transfer, enabling MIGE to generalize to novel
compositional tasks, including instruction-based subject-driven editing.
Experiments show that MIGE excels in both subject-driven generation and
instruction-based editing while setting a state-of-the-art in the new task of
instruction-based subject-driven editing. Code and model have been publicly
available at https://github.com/Eureka-Maggie/MIGE.
67c5aad632a7208c9ae1d020
Xueyun Tian
67c5aad632a7208c9ae1d021
Wei Li
extracted_pending
2025-03-03T13:12:57.839Z
63044e025c70c21d0eaf08bc
/avatars/a2d39973d7fbcbe9d4cce5648b3149c2.svg
Wei Li
user
Wiley085
67c5aad632a7208c9ae1d022
Bingbing Xu
67c5aad632a7208c9ae1d023
Yige Yuan
67c5aad632a7208c9ae1d024
Yuanzhuo Wang
67c5aad632a7208c9ae1d025
Huawei Shen
MIGE: A Unified Framework for Multimodal Instruction-Based Image
  Generation and Editing
Despite significant progress in diffusion-based image generation,
subject-driven generation and instruction-based editing remain challenging.
Existing methods typically treat them separately, struggling with limited
high-quality data and poor generalization. However, both tasks require
capturing complex visual variations while maintaining consistency between
inputs and outputs. Therefore, we propose MIGE, a unified framework that
standardizes task representations using multimodal instructions. It treats
subject-driven generation as creation on a blank canvas and instruction-based
editing as modification of an existing image, establishing a shared
input-output formulation. MIGE introduces a novel multimodal encoder that maps
free-form multimodal instructions into a unified vision-language space,
integrating visual and semantic features through a feature fusion
mechanism.This unification enables joint training of both tasks, providing two
key advantages: (1) Cross-Task Enhancement: By leveraging shared visual and
semantic representations, joint training improves instruction adherence and
visual consistency in both subject-driven generation and instruction-based
editing. (2) Generalization: Learning in a unified format facilitates
cross-task knowledge transfer, enabling MIGE to generalize to novel
compositional tasks, including instruction-based subject-driven editing.
Experiments show that MIGE excels in both subject-driven generation and
instruction-based editing while setting a state-of-the-art in the new task of
instruction-based subject-driven editing. Code and model have been publicly
available at https://github.com/Eureka-Maggie/MIGE.
MIGE, a unified framework using multimodal instructions, enhances both subject-driven generation and instruction-based editing through shared visual and semantic representations.
diffusion-based image generation multimodal instructions MIGE feature fusion mechanism cross-Task Enhancement instruction adherence visual consistency generalization cross-task knowledge transfer instruction-based subject-driven editing