ProJudge: A Multi-Modal Multi-Discipline Benchmark and
  Instruction-Tuning Dataset for MLLM-based Process Judges
As multi-modal large language models (MLLMs) frequently exhibit errors when
solving scientific problems, evaluating the validity of their reasoning
processes is critical for ensuring reliability and uncovering fine-grained
model weaknesses. Since human evaluation is laborious and costly, prompting
MLLMs as automated process judges has become a common practice. However, the
reliability of these model-based judges remains uncertain. To address this, we
introduce ProJudgeBench, the first comprehensive benchmark specifically
designed for evaluating abilities of MLLM-based process judges. ProJudgeBench
comprises 2,400 test cases and 50,118 step-level labels, spanning four
scientific disciplines with diverse difficulty levels and multi-modal content.
In ProJudgeBench, each step is meticulously annotated by human experts for
correctness, error type, and explanation, enabling a systematic evaluation of
judges' capabilities to detect, classify and diagnose errors. Evaluation on
ProJudgeBench reveals a significant performance gap between open-source and
proprietary models. To bridge this gap, we further propose ProJudge-173k, a
large-scale instruction-tuning dataset, and a Dynamic Dual-Phase fine-tuning
strategy that encourages models to explicitly reason through problem-solving
before assessing solutions. Both contributions significantly enhance the
process evaluation capabilities of open-source models. All the resources will
be released to foster future research of reliable multi-modal process
evaluation.
67cfcf664dac6ed12db8b10a
Jiaxin Ai
67cfcf664dac6ed12db8b10b
Pengfei Zhou
admin_assigned
2025-03-17T14:54:16.357Z
65df481e530333731ea24617
/avatars/3ed41f5d7d0489193807b5e6260f16c9.svg
ZHOU PENGFEI
user
lyuukuu
67cfcf664dac6ed12db8b10c
Zhaopan Xu
admin_assigned
2025-03-17T14:52:13.027Z
646dee28174cc96d50951991
/avatars/17d88a24c905e9819268b27037015a35.svg
xu
user
xuzhaopan
67cfcf664dac6ed12db8b10d
Ming Li
67cfcf664dac6ed12db8b10e
Fanrui Zhang
admin_assigned
2025-03-17T14:52:27.748Z
665305eff0c8c891cae7fe01
/avatars/1f372e3bc6a4eb19ef702ec96a391c96.svg
Fanrui Zhang
user
fanrui00
67cfcf664dac6ed12db8b10f
Zizhen Li
67cfcf664dac6ed12db8b110
Jianwen Sun
67cfcf664dac6ed12db8b111
Yukang Feng
admin_assigned
2025-03-17T14:51:23.539Z
66d94f2a36aa5055694dfe04
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/grAN83brH0E4_S0__yLdv.jpeg
fengyukang
user
finyorko
67cfcf664dac6ed12db8b112
Baojin Huang
67cfcf664dac6ed12db8b113
Zhongyuan Wang
67cfcf664dac6ed12db8b114
Kaipeng Zhang
admin_assigned
2025-03-17T14:50:51.940Z
65f1713552c38a91e0a445e8
/avatars/47ab3ada51c9b9976ac1cd0c4301c373.svg
kaipeng
user
kpzhang996
ProJudge: A Multi-Modal Multi-Discipline Benchmark and
  Instruction-Tuning Dataset for MLLM-based Process Judges
As multi-modal large language models (MLLMs) frequently exhibit errors when
solving scientific problems, evaluating the validity of their reasoning
processes is critical for ensuring reliability and uncovering fine-grained
model weaknesses. Since human evaluation is laborious and costly, prompting
MLLMs as automated process judges has become a common practice. However, the
reliability of these model-based judges remains uncertain. To address this, we
introduce ProJudgeBench, the first comprehensive benchmark specifically
designed for evaluating abilities of MLLM-based process judges. ProJudgeBench
comprises 2,400 test cases and 50,118 step-level labels, spanning four
scientific disciplines with diverse difficulty levels and multi-modal content.
In ProJudgeBench, each step is meticulously annotated by human experts for
correctness, error type, and explanation, enabling a systematic evaluation of
judges' capabilities to detect, classify and diagnose errors. Evaluation on
ProJudgeBench reveals a significant performance gap between open-source and
proprietary models. To bridge this gap, we further propose ProJudge-173k, a
large-scale instruction-tuning dataset, and a Dynamic Dual-Phase fine-tuning
strategy that encourages models to explicitly reason through problem-solving
before assessing solutions. Both contributions significantly enhance the
process evaluation capabilities of open-source models. All the resources will
be released to foster future research of reliable multi-modal process
evaluation.
ProJudgeBench evaluates the error detection and classification abilities of multi-modal large language models using human-annotated test cases, revealing performance gaps and proposing ProJudge-173k and Dynamic Dual-Phase fine-tuning strategies to enhance open-source model capabilities.
multi-modal large language models ProJudgeBench process judges step-level labels error detection error classification Dynamic Dual-Phase fine-tuning