title: MMMG: a Comprehensive and Reliable Evaluation Suite for Multitask
    Multimodal Generation
summary: Automatically evaluating multimodal generation presents a significant
  challenge, as automated metrics often struggle to align reliably with human
  evaluation, especially for complex tasks that involve multiple modalities. To
  address this, we present MMMG, a comprehensive and human-aligned benchmark for
  multimodal generation across 4 modality combinations (image, audio, interleaved
  text and image, interleaved text and audio), with a focus on tasks that present
  significant challenges for generation models, while still enabling reliable
  automatic evaluation through a combination of models and programs. MMMG
  encompasses 49 tasks (including 29 newly developed ones), each with a carefully
  designed evaluation pipeline, and 937 instructions to systematically assess
  reasoning, controllability, and other key capabilities of multimodal generation
  models. Extensive validation demonstrates that MMMG is highly aligned with
  human evaluation, achieving an average agreement of 94.3%. Benchmarking results
  on 24 multimodal generation models reveal that even though the state-of-the-art
  model, GPT Image, achieves 78.3% accuracy for image generation, it falls short
  on multimodal reasoning and interleaved generation. Furthermore, results
  suggest considerable headroom for improvement in audio generation, highlighting
  an important direction for future research.
paper_authors._id: 68361038450a48f228ff555e
paper_authors.name: Jihan Yao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T19:14:25.358Z
paper_authors._id: 65750954a90ae2daaec7ab93
paper_authors.avatarUrl: /avatars/43ea342d9a10caeadc0183afc3848f5a.svg
paper_authors.fullname: Jihan Yao
paper_authors.type: user
paper_authors.user: yaojh18
paper_authors._id: 68361038450a48f228ff555f
paper_authors.name: Yushi Hu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-05-27T19:21:19.936Z
paper_authors._id: 62b1474bdcbad6848a91a54e
paper_authors.avatarUrl: /avatars/d7308899b46232cad4a48a0e876449a8.svg
paper_authors.fullname: Yushi Hu
paper_authors.type: user
paper_authors.user: yushihu
paper_authors._id: 68361038450a48f228ff5560
paper_authors.name: Yujie Yi
paper_authors._id: 68361038450a48f228ff5561
paper_authors.name: Bin Han
paper_authors._id: 68361038450a48f228ff5562
paper_authors.name: Shangbin Feng
paper_authors._id: 68361038450a48f228ff5563
paper_authors.name: Guang Yang
paper_authors._id: 68361038450a48f228ff5564
paper_authors.name: Bingbing Wen
paper_authors._id: 68361038450a48f228ff5565
paper_authors.name: Ranjay Krishna
paper_authors._id: 68361038450a48f228ff5566
paper_authors.name: Lucy Lu Wang
paper_authors._id: 68361038450a48f228ff5567
paper_authors.name: Yulia Tsvetkov
paper_authors._id: 68361038450a48f228ff5568
paper_authors.name: Noah A. Smith
paper_authors._id: 68361038450a48f228ff5569
paper_authors.name: Banghua Zhu
paper_title: MMMG: a Comprehensive and Reliable Evaluation Suite for Multitask
    Multimodal Generation
paper_summary: Automatically evaluating multimodal generation presents a significant
  challenge, as automated metrics often struggle to align reliably with human
  evaluation, especially for complex tasks that involve multiple modalities. To
  address this, we present MMMG, a comprehensive and human-aligned benchmark for
  multimodal generation across 4 modality combinations (image, audio, interleaved
  text and image, interleaved text and audio), with a focus on tasks that present
  significant challenges for generation models, while still enabling reliable
  automatic evaluation through a combination of models and programs. MMMG
  encompasses 49 tasks (including 29 newly developed ones), each with a carefully
  designed evaluation pipeline, and 937 instructions to systematically assess
  reasoning, controllability, and other key capabilities of multimodal generation
  models. Extensive validation demonstrates that MMMG is highly aligned with
  human evaluation, achieving an average agreement of 94.3%. Benchmarking results
  on 24 multimodal generation models reveal that even though the state-of-the-art
  model, GPT Image, achieves 78.3% accuracy for image generation, it falls short
  on multimodal reasoning and interleaved generation. Furthermore, results
  suggest considerable headroom for improvement in audio generation, highlighting
  an important direction for future research.
paper_ai_summary: MMMG is a comprehensive benchmark for multimodal generation, offering 49 tasks and 937 instructions to align automatic evaluation with human judgment, revealing areas for improvement in reasoning and audio generation.
paper_ai_keywords: multimodal generation evaluation pipeline human-aligned benchmark multimodal reasoning interleaved generation audio generation GPT Image