title: UPME: An Unsupervised Peer Review Framework for Multimodal Large
    Language Model Evaluation
summary: Multimodal Large Language Models (MLLMs) have emerged to tackle the
  challenges of Visual Question Answering (VQA), sparking a new research focus on
  conducting objective evaluations of these models. Existing evaluation methods
  face limitations due to the significant human workload required to design Q&A
  pairs for visual images, which inherently restricts the scale and scope of
  evaluations. Although automated MLLM-as-judge approaches attempt to reduce the
  human workload through automatic evaluations, they often introduce biases. To
  address these problems, we propose an Unsupervised Peer review MLLM Evaluation
  framework. It utilizes only image data, allowing models to automatically
  generate questions and conduct peer review assessments of answers from other
  models, effectively alleviating the reliance on human workload. Additionally,
  we introduce the vision-language scoring system to mitigate the bias issues,
  which focuses on three aspects: (i) response correctness; (ii) visual
  understanding and reasoning; and (iii) image-text correlation. Experimental
  results demonstrate that UPME achieves a Pearson correlation of 0.944 with
  human evaluations on the MMstar dataset and 0.814 on the ScienceQA dataset,
  indicating that our framework closely aligns with human-designed benchmarks and
  inherent human preferences.
paper_authors._id: 67eb932522a341478ae86cb6
paper_authors.name: Qihui Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-01T07:46:00.373Z
paper_authors._id: 67a99d1fef1439e285c4cbec
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/VrwUmrY2wsg4sVSIMc--K.png
paper_authors.fullname: Qihui Zhang
paper_authors.type: user
paper_authors.user: 77Hui
paper_authors._id: 67eb932522a341478ae86cb7
paper_authors.name: Munan Ning
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-01T08:18:39.336Z
paper_authors._id: 65e14c28b1a6de8a71e70172
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65e14c28b1a6de8a71e70172/D097SILGsqoufpp3sG8tV.jpeg
paper_authors.fullname: Munan Ning
paper_authors.type: user
paper_authors.user: MunanNing
paper_authors._id: 67eb932522a341478ae86cb8
paper_authors.name: Zheyuan Liu
paper_authors._id: 67eb932522a341478ae86cb9
paper_authors.name: Yanbo Wang
paper_authors._id: 67eb932522a341478ae86cba
paper_authors.name: Jiayi Ye
paper_authors._id: 67eb932522a341478ae86cbb
paper_authors.name: Yue Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-01T08:19:26.627Z
paper_authors._id: 6637443ecd9097ac3c996d3c
paper_authors.avatarUrl: /avatars/d1c38bf03c2517ba0a7004b2f9f9bc96.svg
paper_authors.fullname: yue
paper_authors.type: user
paper_authors.user: yuehuang
paper_authors._id: 67eb932522a341478ae86cbc
paper_authors.name: Shuo Yang
paper_authors._id: 67eb932522a341478ae86cbd
paper_authors.name: Xiao Chen
paper_authors._id: 67eb932522a341478ae86cbe
paper_authors.name: Yibing Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-01T08:19:07.616Z
paper_authors._id: 62c51800cb7033fd49b8efb7
paper_authors.avatarUrl: /avatars/06c2be0015f8022f9912f2279f2b3597.svg
paper_authors.fullname: Song
paper_authors.type: user
paper_authors.user: Yibing
paper_authors._id: 67eb932522a341478ae86cbf
paper_authors.name: Li Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-04T07:09:54.399Z
paper_authors._id: 66135a5e50350afe76beebce
paper_authors.avatarUrl: /avatars/370a4b83949355feb050c2cb0425c264.svg
paper_authors.fullname: yl2488
paper_authors.type: user
paper_authors.user: yl2488
paper_title: UPME: An Unsupervised Peer Review Framework for Multimodal Large
    Language Model Evaluation
paper_summary: Multimodal Large Language Models (MLLMs) have emerged to tackle the
  challenges of Visual Question Answering (VQA), sparking a new research focus on
  conducting objective evaluations of these models. Existing evaluation methods
  face limitations due to the significant human workload required to design Q&A
  pairs for visual images, which inherently restricts the scale and scope of
  evaluations. Although automated MLLM-as-judge approaches attempt to reduce the
  human workload through automatic evaluations, they often introduce biases. To
  address these problems, we propose an Unsupervised Peer review MLLM Evaluation
  framework. It utilizes only image data, allowing models to automatically
  generate questions and conduct peer review assessments of answers from other
  models, effectively alleviating the reliance on human workload. Additionally,
  we introduce the vision-language scoring system to mitigate the bias issues,
  which focuses on three aspects: (i) response correctness; (ii) visual
  understanding and reasoning; and (iii) image-text correlation. Experimental
  results demonstrate that UPME achieves a Pearson correlation of 0.944 with
  human evaluations on the MMstar dataset and 0.814 on the ScienceQA dataset,
  indicating that our framework closely aligns with human-designed benchmarks and
  inherent human preferences.
paper_ai_summary: Unsupervised Peer Review MLLM Evaluation framework uses image data to automatically generate questions and assess answers, reducing human workload and mitigating bias.
paper_ai_keywords: Multimodal Large Language Models Visual Question Answering MLLM-as-judge Unsupervised Peer review vision-language scoring system response correctness visual understanding reasoning image-text correlation Pearson correlation