Scientists' First Exam: Probing Cognitive Abilities of MLLM via
  Perception, Understanding, and Reasoning
Scientific discoveries increasingly rely on complex multimodal reasoning
based on information-intensive scientific data and domain-specific expertise.
Empowered by expert-level scientific benchmarks, scientific Multimodal Large
Language Models (MLLMs) hold the potential to significantly enhance this
discovery process in realistic workflows. However, current scientific
benchmarks mostly focus on evaluating the knowledge understanding capabilities
of MLLMs, leading to an inadequate assessment of their perception and reasoning
abilities. To address this gap, we present the Scientists' First Exam (SFE)
benchmark, designed to evaluate the scientific cognitive capacities of MLLMs
through three interconnected levels: scientific signal perception, scientific
attribute understanding, scientific comparative reasoning. Specifically, SFE
comprises 830 expert-verified VQA pairs across three question types, spanning
66 multimodal tasks across five high-value disciplines. Extensive experiments
reveal that current state-of-the-art GPT-o3 and InternVL-3 achieve only 34.08%
and 26.52% on SFE, highlighting significant room for MLLMs to improve in
scientific realms. We hope the insights obtained in SFE will facilitate further
developments in AI-enhanced scientific discoveries.
684b8c603b733ba333686ffe
Yuhao Zhou
684b8c603b733ba333686fff
Yiheng Wang
684b8c603b733ba333687000
Xuming He
684b8c603b733ba333687001
Ruoyao Xiao
684b8c603b733ba333687002
Zhiwei Li
684b8c603b733ba333687003
Qiantai Feng
684b8c603b733ba333687004
Zijie Guo
684b8c603b733ba333687005
Yuejin Yang
684b8c603b733ba333687006
Hao Wu
684b8c603b733ba333687007
Wenxuan Huang
claimed_verified
2025-06-17T07:22:33.270Z
675118b088a927f8898f81b4
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/qVVgc2kuR37QqO5-Lu8Xq.png
Wilson Huang
user
WilsonHwang
684b8c603b733ba333687008
Jiaqi Wei
684b8c603b733ba333687009
Dan Si
684b8c603b733ba33368700a
Xiuqi Yao
684b8c603b733ba33368700b
Jia Bu
684b8c603b733ba33368700c
Haiwen Huang
684b8c603b733ba33368700d
Tianfan Fu
684b8c603b733ba33368700e
Shixiang Tang
684b8c603b733ba33368700f
Ben Fei
684b8c603b733ba333687010
Dongzhan Zhou
684b8c603b733ba333687011
Fenghua Ling
684b8c603b733ba333687012
Yan Lu
684b8c603b733ba333687013
Siqi Sun
claimed_verified
2025-06-17T13:06:51.338Z
67d3a1a5943a965360fcae51
/avatars/165ed684b0750e7f57b9f2babfb47a8c.svg
Siqi Sun
user
siqisun
684b8c603b733ba333687014
Chenhui Li
684b8c603b733ba333687015
Guanjie Zheng
684b8c603b733ba333687016
Jiancheng Lv
684b8c603b733ba333687017
Wenlong Zhang
684b8c603b733ba333687018
Lei Bai
Scientists' First Exam: Probing Cognitive Abilities of MLLM via
  Perception, Understanding, and Reasoning
Scientific discoveries increasingly rely on complex multimodal reasoning
based on information-intensive scientific data and domain-specific expertise.
Empowered by expert-level scientific benchmarks, scientific Multimodal Large
Language Models (MLLMs) hold the potential to significantly enhance this
discovery process in realistic workflows. However, current scientific
benchmarks mostly focus on evaluating the knowledge understanding capabilities
of MLLMs, leading to an inadequate assessment of their perception and reasoning
abilities. To address this gap, we present the Scientists' First Exam (SFE)
benchmark, designed to evaluate the scientific cognitive capacities of MLLMs
through three interconnected levels: scientific signal perception, scientific
attribute understanding, scientific comparative reasoning. Specifically, SFE
comprises 830 expert-verified VQA pairs across three question types, spanning
66 multimodal tasks across five high-value disciplines. Extensive experiments
reveal that current state-of-the-art GPT-o3 and InternVL-3 achieve only 34.08%
and 26.52% on SFE, highlighting significant room for MLLMs to improve in
scientific realms. We hope the insights obtained in SFE will facilitate further
developments in AI-enhanced scientific discoveries.
Scientists' First Exam (SFE) benchmark assesses scientific cognitive capacities of Multimodal Large Language Models through perception, understanding, and comparative reasoning.
Multimodal Large Language Models SFE benchmark scientific signal perception scientific attribute understanding scientific comparative reasoning expert-verified VQA GPT-o3 InternVL-3