title: ProBench: Judging Multimodal Foundation Models on Open-ended
    Multi-domain Expert Tasks
summary: Solving expert-level multimodal tasks is a key milestone towards general
  intelligence. As the capabilities of multimodal large language models (MLLMs)
  continue to improve, evaluation of such advanced multimodal intelligence
  becomes necessary yet challenging. In this work, we introduce ProBench, a
  benchmark of open-ended user queries that require professional expertise and
  advanced reasoning. ProBench consists of 4,000 high-quality samples
  independently submitted by professionals based on their daily productivity
  demands. It spans across 10 fields and 56 sub-fields, including science, arts,
  humanities, coding, mathematics, and creative writing. Experimentally, we
  evaluate and compare 24 latest models using MLLM-as-a-Judge. Our results reveal
  that although the best open-source models rival the proprietary ones, ProBench
  presents significant challenges in visual perception, textual understanding,
  domain knowledge and advanced reasoning, thus providing valuable directions for
  future multimodal AI research efforts.
paper_authors._id: 67cfc1c5182d970d40896a5e
paper_authors.name: Yan Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-11T08:21:28.574Z
paper_authors._id: 655b813476e4fad5529f3256
paper_authors.avatarUrl: /avatars/73d83e45d921531f9830a0ea80f76491.svg
paper_authors.fullname: Yan Yang
paper_authors.type: user
paper_authors.user: HelloKKMe
paper_authors._id: 67cfc1c5182d970d40896a5f
paper_authors.name: Dongxu Li
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-03-11T04:53:27.837Z
paper_authors._id: 6357362f811ee2fa05070f64
paper_authors.avatarUrl: /avatars/2cf37efb80f5cfb3e4e9d08674de6dd1.svg
paper_authors.fullname: Dongxu Li
paper_authors.type: user
paper_authors.user: dxli1
paper_authors._id: 67cfc1c5182d970d40896a60
paper_authors.name: Haoning Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T14:32:00.625Z
paper_authors._id: 632c7a0d1d303f5f9acf01b8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/632c7a0d1d303f5f9acf01b8/T010IFuCp6UaOeIyWhbCk.jpeg
paper_authors.fullname: Haoning Wu
paper_authors.type: user
paper_authors.user: haoningwu
paper_authors._id: 67cfc1c5182d970d40896a61
paper_authors.name: Bei Chen
paper_authors._id: 67cfc1c5182d970d40896a62
paper_authors.name: Liu Liu
paper_authors._id: 67cfc1c5182d970d40896a63
paper_authors.name: Liyuan Pan
paper_authors._id: 67cfc1c5182d970d40896a64
paper_authors.name: Junnan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T14:31:33.001Z
paper_authors._id: 61f9d3b54ac99e8a1bae85f4
paper_authors.avatarUrl: /avatars/ac47d13204dd22452e4bc46e280842d5.svg
paper_authors.fullname: JunnanLi
paper_authors.type: user
paper_authors.user: JunnanLi
paper_title: ProBench: Judging Multimodal Foundation Models on Open-ended
    Multi-domain Expert Tasks
paper_summary: Solving expert-level multimodal tasks is a key milestone towards general
  intelligence. As the capabilities of multimodal large language models (MLLMs)
  continue to improve, evaluation of such advanced multimodal intelligence
  becomes necessary yet challenging. In this work, we introduce ProBench, a
  benchmark of open-ended user queries that require professional expertise and
  advanced reasoning. ProBench consists of 4,000 high-quality samples
  independently submitted by professionals based on their daily productivity
  demands. It spans across 10 fields and 56 sub-fields, including science, arts,
  humanities, coding, mathematics, and creative writing. Experimentally, we
  evaluate and compare 24 latest models using MLLM-as-a-Judge. Our results reveal
  that although the best open-source models rival the proprietary ones, ProBench
  presents significant challenges in visual perception, textual understanding,
  domain knowledge and advanced reasoning, thus providing valuable directions for
  future multimodal AI research efforts.
paper_ai_keywords: multimodal large language models (MLLMs) benchmark user queries professional expertise advanced reasoning high-quality samples daily productivity demands fields sub-fields visual perception textual understanding domain knowledge