title: Creation-MMBench: Assessing Context-Aware Creative Intelligence in MLLM
summary: Creativity is a fundamental aspect of intelligence, involving the ability to
  generate novel and appropriate solutions across diverse contexts. While Large
  Language Models (LLMs) have been extensively evaluated for their creative
  capabilities, the assessment of Multimodal Large Language Models (MLLMs) in
  this domain remains largely unexplored. To address this gap, we introduce
  Creation-MMBench, a multimodal benchmark specifically designed to evaluate the
  creative capabilities of MLLMs in real-world, image-based tasks. The benchmark
  comprises 765 test cases spanning 51 fine-grained tasks. To ensure rigorous
  evaluation, we define instance-specific evaluation criteria for each test case,
  guiding the assessment of both general response quality and factual consistency
  with visual inputs. Experimental results reveal that current open-source MLLMs
  significantly underperform compared to proprietary models in creative tasks.
  Furthermore, our analysis demonstrates that visual fine-tuning can negatively
  impact the base LLM's creative abilities. Creation-MMBench provides valuable
  insights for advancing MLLM creativity and establishes a foundation for future
  improvements in multimodal generative intelligence. Full data and evaluation
  code is released on https://github.com/open-compass/Creation-MMBench.
paper_authors._id: 67da34a648348387ebac36ff
paper_authors.name: Xinyu Fang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:17:40.816Z
paper_authors._id: 64f5f8dd9b17cd59c453c57f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64f5f8dd9b17cd59c453c57f/MulhwLcePFUWUQel8LQZ8.jpeg
paper_authors.fullname: Xinyu Fang
paper_authors.type: user
paper_authors.user: nebulae09
paper_authors._id: 67da34a648348387ebac3700
paper_authors.name: Zhijian Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-20T10:46:07.437Z
paper_authors._id: 6726407e98e7ba4c5a2e515f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/iVXMudnRASUhynOAce-1F.png
paper_authors.fullname: Zhijian Chen
paper_authors.type: user
paper_authors.user: J-Roger
paper_authors._id: 67da34a648348387ebac3701
paper_authors.name: Kai Lan
paper_authors._id: 67da34a648348387ebac3702
paper_authors.name: Shengyuan Ding
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:18:02.611Z
paper_authors._id: 646cd947da8e99940b6e55cf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646cd947da8e99940b6e55cf/9c0P0WppFqNW9pdo8LgOS.jpeg
paper_authors.fullname: Shengyuan Ding
paper_authors.type: user
paper_authors.user: ChrisDing1105
paper_authors._id: 67da34a648348387ebac3703
paper_authors.name: Yingji Liang
paper_authors._id: 67da34a648348387ebac3704
paper_authors.name: Xiangyu Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-19T13:14:48.768Z
paper_authors._id: 6530e62f536dbca918e71c3e
paper_authors.avatarUrl: /avatars/efc93bc767e561c6c6d429f65c23382d.svg
paper_authors.fullname: Xiangyu Z
paper_authors.type: user
paper_authors.user: PhoenixZ
paper_authors._id: 67da34a648348387ebac3705
paper_authors.name: Farong Wen
paper_authors._id: 67da34a648348387ebac3706
paper_authors.name: Zicheng Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-08T06:57:41.676Z
paper_authors._id: 6526cc6bab4f5d98382f5603
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6526cc6bab4f5d98382f5603/NekYYb61I5nt4au6gXsVK.jpeg
paper_authors.fullname: Zicheng Zhang
paper_authors.type: user
paper_authors.user: zhangzicheng
paper_authors._id: 67da34a648348387ebac3707
paper_authors.name: Guofeng Zhang
paper_authors._id: 67da34a648348387ebac3708
paper_authors.name: Haodong Duan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:18:37.741Z
paper_authors._id: 63ee1379190ddd6214efd73a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
paper_authors.fullname: HAODONG DUAN
paper_authors.type: user
paper_authors.user: KennyUTC
paper_authors._id: 67da34a648348387ebac3709
paper_authors.name: Kai Chen
paper_authors._id: 67da34a648348387ebac370a
paper_authors.name: Dahua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:18:43.705Z
paper_authors._id: 636317ed80c1a705a6eff396
paper_authors.avatarUrl: /avatars/3db090e101b916d9256d0d3e043db71d.svg
paper_authors.fullname: Dahua Lin
paper_authors.type: user
paper_authors.user: lindahua
paper_title: Creation-MMBench: Assessing Context-Aware Creative Intelligence in MLLM
paper_summary: Creativity is a fundamental aspect of intelligence, involving the ability to
  generate novel and appropriate solutions across diverse contexts. While Large
  Language Models (LLMs) have been extensively evaluated for their creative
  capabilities, the assessment of Multimodal Large Language Models (MLLMs) in
  this domain remains largely unexplored. To address this gap, we introduce
  Creation-MMBench, a multimodal benchmark specifically designed to evaluate the
  creative capabilities of MLLMs in real-world, image-based tasks. The benchmark
  comprises 765 test cases spanning 51 fine-grained tasks. To ensure rigorous
  evaluation, we define instance-specific evaluation criteria for each test case,
  guiding the assessment of both general response quality and factual consistency
  with visual inputs. Experimental results reveal that current open-source MLLMs
  significantly underperform compared to proprietary models in creative tasks.
  Furthermore, our analysis demonstrates that visual fine-tuning can negatively
  impact the base LLM's creative abilities. Creation-MMBench provides valuable
  insights for advancing MLLM creativity and establishes a foundation for future
  improvements in multimodal generative intelligence. Full data and evaluation
  code is released on https://github.com/open-compass/Creation-MMBench.
paper_ai_summary: Creation-MMBench evaluates the creative capabilities of Multimodal Large Language Models (MLLMs) in image-based real-world tasks, revealing performance gaps and the negative impacts of visual fine-tuning.
paper_ai_keywords: Multimodal Large Language Models MLLMs Creation-MMBench instance-specific evaluation criteria visual fine-tuning multimodal generative intelligence