ProBench: Judging Multimodal Foundation Models on Open-ended
  Multi-domain Expert Tasks
Solving expert-level multimodal tasks is a key milestone towards general
intelligence. As the capabilities of multimodal large language models (MLLMs)
continue to improve, evaluation of such advanced multimodal intelligence
becomes necessary yet challenging. In this work, we introduce ProBench, a
benchmark of open-ended user queries that require professional expertise and
advanced reasoning. ProBench consists of 4,000 high-quality samples
independently submitted by professionals based on their daily productivity
demands. It spans across 10 fields and 56 sub-fields, including science, arts,
humanities, coding, mathematics, and creative writing. Experimentally, we
evaluate and compare 24 latest models using MLLM-as-a-Judge. Our results reveal
that although the best open-source models rival the proprietary ones, ProBench
presents significant challenges in visual perception, textual understanding,
domain knowledge and advanced reasoning, thus providing valuable directions for
future multimodal AI research efforts.
67cfc1c5182d970d40896a5e
Yan Yang
claimed_verified
2025-03-11T08:21:28.574Z
655b813476e4fad5529f3256
/avatars/73d83e45d921531f9830a0ea80f76491.svg
Yan Yang
user
HelloKKMe
67cfc1c5182d970d40896a5f
Dongxu Li
extracted_pending
2025-03-11T04:53:27.837Z
6357362f811ee2fa05070f64
/avatars/2cf37efb80f5cfb3e4e9d08674de6dd1.svg
Dongxu Li
user
dxli1
67cfc1c5182d970d40896a60
Haoning Wu
admin_assigned
2025-03-11T14:32:00.625Z
632c7a0d1d303f5f9acf01b8
https://cdn-avatars.huggingface.co/v1/production/uploads/632c7a0d1d303f5f9acf01b8/T010IFuCp6UaOeIyWhbCk.jpeg
Haoning Wu
user
haoningwu
67cfc1c5182d970d40896a61
Bei Chen
67cfc1c5182d970d40896a62
Liu Liu
67cfc1c5182d970d40896a63
Liyuan Pan
67cfc1c5182d970d40896a64
Junnan Li
admin_assigned
2025-03-11T14:31:33.001Z
61f9d3b54ac99e8a1bae85f4
/avatars/ac47d13204dd22452e4bc46e280842d5.svg
JunnanLi
user
JunnanLi
ProBench: Judging Multimodal Foundation Models on Open-ended
  Multi-domain Expert Tasks
Solving expert-level multimodal tasks is a key milestone towards general
intelligence. As the capabilities of multimodal large language models (MLLMs)
continue to improve, evaluation of such advanced multimodal intelligence
becomes necessary yet challenging. In this work, we introduce ProBench, a
benchmark of open-ended user queries that require professional expertise and
advanced reasoning. ProBench consists of 4,000 high-quality samples
independently submitted by professionals based on their daily productivity
demands. It spans across 10 fields and 56 sub-fields, including science, arts,
humanities, coding, mathematics, and creative writing. Experimentally, we
evaluate and compare 24 latest models using MLLM-as-a-Judge. Our results reveal
that although the best open-source models rival the proprietary ones, ProBench
presents significant challenges in visual perception, textual understanding,
domain knowledge and advanced reasoning, thus providing valuable directions for
future multimodal AI research efforts.
ProBench, a benchmark for professional-level multimodal tasks, highlights challenges in visual perception, textual understanding, domain knowledge, and reasoning for large language models.
multimodal large language models MLLM-as-a-Judge benchmark visual perception textual understanding domain knowledge advanced reasoning