title: ProJudge: A Multi-Modal Multi-Discipline Benchmark and
    Instruction-Tuning Dataset for MLLM-based Process Judges
summary: As multi-modal large language models (MLLMs) frequently exhibit errors when
  solving scientific problems, evaluating the validity of their reasoning
  processes is critical for ensuring reliability and uncovering fine-grained
  model weaknesses. Since human evaluation is laborious and costly, prompting
  MLLMs as automated process judges has become a common practice. However, the
  reliability of these model-based judges remains uncertain. To address this, we
  introduce ProJudgeBench, the first comprehensive benchmark specifically
  designed for evaluating abilities of MLLM-based process judges. ProJudgeBench
  comprises 2,400 test cases and 50,118 step-level labels, spanning four
  scientific disciplines with diverse difficulty levels and multi-modal content.
  In ProJudgeBench, each step is meticulously annotated by human experts for
  correctness, error type, and explanation, enabling a systematic evaluation of
  judges' capabilities to detect, classify and diagnose errors. Evaluation on
  ProJudgeBench reveals a significant performance gap between open-source and
  proprietary models. To bridge this gap, we further propose ProJudge-173k, a
  large-scale instruction-tuning dataset, and a Dynamic Dual-Phase fine-tuning
  strategy that encourages models to explicitly reason through problem-solving
  before assessing solutions. Both contributions significantly enhance the
  process evaluation capabilities of open-source models. All the resources will
  be released to foster future research of reliable multi-modal process
  evaluation.
paper_authors._id: 67cfcf664dac6ed12db8b10a
paper_authors.name: Jiaxin Ai
paper_authors._id: 67cfcf664dac6ed12db8b10b
paper_authors.name: Pengfei Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-17T14:54:16.357Z
paper_authors._id: 65df481e530333731ea24617
paper_authors.avatarUrl: /avatars/3ed41f5d7d0489193807b5e6260f16c9.svg
paper_authors.fullname: ZHOU PENGFEI
paper_authors.type: user
paper_authors.user: lyuukuu
paper_authors._id: 67cfcf664dac6ed12db8b10c
paper_authors.name: Zhaopan Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-17T14:52:13.027Z
paper_authors._id: 646dee28174cc96d50951991
paper_authors.avatarUrl: /avatars/17d88a24c905e9819268b27037015a35.svg
paper_authors.fullname: xu
paper_authors.type: user
paper_authors.user: xuzhaopan
paper_authors._id: 67cfcf664dac6ed12db8b10d
paper_authors.name: Ming Li
paper_authors._id: 67cfcf664dac6ed12db8b10e
paper_authors.name: Fanrui Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-17T14:52:27.748Z
paper_authors._id: 665305eff0c8c891cae7fe01
paper_authors.avatarUrl: /avatars/1f372e3bc6a4eb19ef702ec96a391c96.svg
paper_authors.fullname: Fanrui Zhang
paper_authors.type: user
paper_authors.user: fanrui00
paper_authors._id: 67cfcf664dac6ed12db8b10f
paper_authors.name: Zizhen Li
paper_authors._id: 67cfcf664dac6ed12db8b110
paper_authors.name: Jianwen Sun
paper_authors._id: 67cfcf664dac6ed12db8b111
paper_authors.name: Yukang Feng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-17T14:51:23.539Z
paper_authors._id: 66d94f2a36aa5055694dfe04
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/grAN83brH0E4_S0__yLdv.jpeg
paper_authors.fullname: fengyukang
paper_authors.type: user
paper_authors.user: finyorko
paper_authors._id: 67cfcf664dac6ed12db8b112
paper_authors.name: Baojin Huang
paper_authors._id: 67cfcf664dac6ed12db8b113
paper_authors.name: Zhongyuan Wang
paper_authors._id: 67cfcf664dac6ed12db8b114
paper_authors.name: Kaipeng Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-17T14:50:51.940Z
paper_authors._id: 65f1713552c38a91e0a445e8
paper_authors.avatarUrl: /avatars/47ab3ada51c9b9976ac1cd0c4301c373.svg
paper_authors.fullname: kaipeng
paper_authors.type: user
paper_authors.user: kpzhang996
paper_title: ProJudge: A Multi-Modal Multi-Discipline Benchmark and
    Instruction-Tuning Dataset for MLLM-based Process Judges
paper_summary: As multi-modal large language models (MLLMs) frequently exhibit errors when
  solving scientific problems, evaluating the validity of their reasoning
  processes is critical for ensuring reliability and uncovering fine-grained
  model weaknesses. Since human evaluation is laborious and costly, prompting
  MLLMs as automated process judges has become a common practice. However, the
  reliability of these model-based judges remains uncertain. To address this, we
  introduce ProJudgeBench, the first comprehensive benchmark specifically
  designed for evaluating abilities of MLLM-based process judges. ProJudgeBench
  comprises 2,400 test cases and 50,118 step-level labels, spanning four
  scientific disciplines with diverse difficulty levels and multi-modal content.
  In ProJudgeBench, each step is meticulously annotated by human experts for
  correctness, error type, and explanation, enabling a systematic evaluation of
  judges' capabilities to detect, classify and diagnose errors. Evaluation on
  ProJudgeBench reveals a significant performance gap between open-source and
  proprietary models. To bridge this gap, we further propose ProJudge-173k, a
  large-scale instruction-tuning dataset, and a Dynamic Dual-Phase fine-tuning
  strategy that encourages models to explicitly reason through problem-solving
  before assessing solutions. Both contributions significantly enhance the
  process evaluation capabilities of open-source models. All the resources will
  be released to foster future research of reliable multi-modal process
  evaluation.
paper_ai_keywords: ProJudgeBench multi-modal large language models (MLLMs) Reasoning processes Automated process judges Step-level labels Scientific disciplines Multi-modal content Error classification Dynamic Dual-Phase fine-tuning Instruction-tuning dataset Problem-solving reasoning