title: MMIG-Bench: Towards Comprehensive and Explainable Evaluation of
    Multi-Modal Image Generation Models
summary: Recent multimodal image generators such as GPT-4o, Gemini 2.0 Flash, and
  Gemini 2.5 Pro excel at following complex instructions, editing images and
  maintaining concept consistency. However, they are still evaluated by disjoint
  toolkits: text-to-image (T2I) benchmarks that lacks multi-modal conditioning,
  and customized image generation benchmarks that overlook compositional
  semantics and common knowledge. We propose MMIG-Bench, a comprehensive
  Multi-Modal Image Generation Benchmark that unifies these tasks by pairing
  4,850 richly annotated text prompts with 1,750 multi-view reference images
  across 380 subjects, spanning humans, animals, objects, and artistic styles.
  MMIG-Bench is equipped with a three-level evaluation framework: (1) low-level
  metrics for visual artifacts and identity preservation of objects; (2) novel
  Aspect Matching Score (AMS): a VQA-based mid-level metric that delivers
  fine-grained prompt-image alignment and shows strong correlation with human
  judgments; and (3) high-level metrics for aesthetics and human preference.
  Using MMIG-Bench, we benchmark 17 state-of-the-art models, including Gemini 2.5
  Pro, FLUX, DreamBooth, and IP-Adapter, and validate our metrics with 32k human
  ratings, yielding in-depth insights into architecture and data design. We will
  release the dataset and evaluation code to foster rigorous, unified evaluation
  and accelerate future innovations in multi-modal image generation.
paper_authors._id: 68360763eccf5a50adcddade
paper_authors.name: Hang Hua
paper_authors._id: 68360763eccf5a50adcddadf
paper_authors.name: Ziyun Zeng
paper_authors._id: 68360763eccf5a50adcddae0
paper_authors.name: Yizhi Song
paper_authors._id: 68360763eccf5a50adcddae1
paper_authors.name: Yunlong Tang
paper_authors._id: 68360763eccf5a50adcddae2
paper_authors.name: Liu He
paper_authors._id: 68360763eccf5a50adcddae3
paper_authors.name: Daniel Aliaga
paper_authors._id: 68360763eccf5a50adcddae4
paper_authors.name: Wei Xiong
paper_authors._id: 68360763eccf5a50adcddae5
paper_authors.name: Jiebo Luo
paper_title: MMIG-Bench: Towards Comprehensive and Explainable Evaluation of
    Multi-Modal Image Generation Models
paper_summary: Recent multimodal image generators such as GPT-4o, Gemini 2.0 Flash, and
  Gemini 2.5 Pro excel at following complex instructions, editing images and
  maintaining concept consistency. However, they are still evaluated by disjoint
  toolkits: text-to-image (T2I) benchmarks that lacks multi-modal conditioning,
  and customized image generation benchmarks that overlook compositional
  semantics and common knowledge. We propose MMIG-Bench, a comprehensive
  Multi-Modal Image Generation Benchmark that unifies these tasks by pairing
  4,850 richly annotated text prompts with 1,750 multi-view reference images
  across 380 subjects, spanning humans, animals, objects, and artistic styles.
  MMIG-Bench is equipped with a three-level evaluation framework: (1) low-level
  metrics for visual artifacts and identity preservation of objects; (2) novel
  Aspect Matching Score (AMS): a VQA-based mid-level metric that delivers
  fine-grained prompt-image alignment and shows strong correlation with human
  judgments; and (3) high-level metrics for aesthetics and human preference.
  Using MMIG-Bench, we benchmark 17 state-of-the-art models, including Gemini 2.5
  Pro, FLUX, DreamBooth, and IP-Adapter, and validate our metrics with 32k human
  ratings, yielding in-depth insights into architecture and data design. We will
  release the dataset and evaluation code to foster rigorous, unified evaluation
  and accelerate future innovations in multi-modal image generation.
paper_ai_summary: A comprehensive benchmark, MMIG-Bench, evaluates multi-modal image generators using text prompts and reference images, providing detailed insights through low-level, mid-level, and high-level metrics.
paper_ai_keywords: multi-modal image generators GPT-4o Gemini 2.0 Flash Gemini 2.5 Pro MMIG-Bench text-to-image (T2I) multi-view reference images Aspect Matching Score (AMS) VQA-based visual artifacts identity preservation aesthetics human preference