title: MME-Unify: A Comprehensive Benchmark for Unified Multimodal
    Understanding and Generation Models
summary: Existing MLLM benchmarks face significant challenges in evaluating Unified
  MLLMs (U-MLLMs) due to: 1) lack of standardized benchmarks for traditional
  tasks, leading to inconsistent comparisons; 2) absence of benchmarks for
  mixed-modality generation, which fails to assess multimodal reasoning
  capabilities. We present a comprehensive evaluation framework designed to
  systematically assess U-MLLMs. Our benchmark includes: Standardized Traditional
  Task Evaluation. We sample from 12 datasets, covering 10 tasks with 30
  subtasks, ensuring consistent and fair comparisons across studies." 2. Unified
  Task Assessment. We introduce five novel tasks testing multimodal reasoning,
  including image editing, commonsense QA with image generation, and geometric
  reasoning. 3. Comprehensive Model Benchmarking. We evaluate 12 leading U-MLLMs,
  such as Janus-Pro, EMU3, VILA-U, and Gemini2-flash, alongside specialized
  understanding (e.g., Claude-3.5-Sonnet) and generation models (e.g., DALL-E-3).
  Our findings reveal substantial performance gaps in existing U-MLLMs,
  highlighting the need for more robust models capable of handling mixed-modality
  tasks effectively. The code and evaluation data can be found in
  https://mme-unify.github.io/.
paper_authors._id: 67f34f5dfb6d8a613926ac2b
paper_authors.name: Wulin Xie
paper_authors._id: 67f34f5dfb6d8a613926ac2c
paper_authors.name: Yi-Fan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-09T07:38:04.393Z
paper_authors._id: 623d8ca4c29adf5ef6175615
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/623d8ca4c29adf5ef6175615/q7lHao7UPwU1u7YLSP56m.jpeg
paper_authors.fullname: Yi-Fan Zhang
paper_authors.type: user
paper_authors.user: yifanzhang114
paper_authors._id: 67f34f5dfb6d8a613926ac2d
paper_authors.name: Chaoyou Fu
paper_authors._id: 67f34f5dfb6d8a613926ac2e
paper_authors.name: Yang Shi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-08T06:53:09.884Z
paper_authors._id: 673c7319d11b1c2e246ead9c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/673c7319d11b1c2e246ead9c/IjFIO--N7Hm_BOEafhEQv.jpeg
paper_authors.fullname: Yang Shi
paper_authors.type: user
paper_authors.user: DogNeverSleep
paper_authors._id: 67f34f5dfb6d8a613926ac2f
paper_authors.name: Bingyan Nie
paper_authors._id: 67f34f5dfb6d8a613926ac30
paper_authors.name: Hongkai Chen
paper_authors._id: 67f34f5dfb6d8a613926ac31
paper_authors.name: Zhang Zhang
paper_authors._id: 67f34f5dfb6d8a613926ac32
paper_authors.name: Liang Wang
paper_authors._id: 67f34f5dfb6d8a613926ac33
paper_authors.name: Tieniu Tan
paper_title: MME-Unify: A Comprehensive Benchmark for Unified Multimodal
    Understanding and Generation Models
paper_summary: Existing MLLM benchmarks face significant challenges in evaluating Unified
  MLLMs (U-MLLMs) due to: 1) lack of standardized benchmarks for traditional
  tasks, leading to inconsistent comparisons; 2) absence of benchmarks for
  mixed-modality generation, which fails to assess multimodal reasoning
  capabilities. We present a comprehensive evaluation framework designed to
  systematically assess U-MLLMs. Our benchmark includes: Standardized Traditional
  Task Evaluation. We sample from 12 datasets, covering 10 tasks with 30
  subtasks, ensuring consistent and fair comparisons across studies." 2. Unified
  Task Assessment. We introduce five novel tasks testing multimodal reasoning,
  including image editing, commonsense QA with image generation, and geometric
  reasoning. 3. Comprehensive Model Benchmarking. We evaluate 12 leading U-MLLMs,
  such as Janus-Pro, EMU3, VILA-U, and Gemini2-flash, alongside specialized
  understanding (e.g., Claude-3.5-Sonnet) and generation models (e.g., DALL-E-3).
  Our findings reveal substantial performance gaps in existing U-MLLMs,
  highlighting the need for more robust models capable of handling mixed-modality
  tasks effectively. The code and evaluation data can be found in
  https://mme-unify.github.io/.
paper_ai_summary: A new evaluation framework assesses Unified MLLMs across traditional and mixed-modality tasks, revealing performance gaps and the need for more robust models.
paper_ai_keywords: Unified MLLMs U-MLLMs standardized benchmarks mixed-modality generation multimodal reasoning image editing commonsense QA geometric reasoning Janus-Pro EMU3 VILA-U Gemini2-flash Claude-3.5-Sonnet DALL-E-3