UPME: An Unsupervised Peer Review Framework for Multimodal Large
  Language Model Evaluation
Multimodal Large Language Models (MLLMs) have emerged to tackle the
challenges of Visual Question Answering (VQA), sparking a new research focus on
conducting objective evaluations of these models. Existing evaluation methods
face limitations due to the significant human workload required to design Q&A
pairs for visual images, which inherently restricts the scale and scope of
evaluations. Although automated MLLM-as-judge approaches attempt to reduce the
human workload through automatic evaluations, they often introduce biases. To
address these problems, we propose an Unsupervised Peer review MLLM Evaluation
framework. It utilizes only image data, allowing models to automatically
generate questions and conduct peer review assessments of answers from other
models, effectively alleviating the reliance on human workload. Additionally,
we introduce the vision-language scoring system to mitigate the bias issues,
which focuses on three aspects: (i) response correctness; (ii) visual
understanding and reasoning; and (iii) image-text correlation. Experimental
results demonstrate that UPME achieves a Pearson correlation of 0.944 with
human evaluations on the MMstar dataset and 0.814 on the ScienceQA dataset,
indicating that our framework closely aligns with human-designed benchmarks and
inherent human preferences.
67eb932522a341478ae86cb6
Qihui Zhang
claimed_verified
2025-04-01T07:46:00.373Z
67a99d1fef1439e285c4cbec
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/VrwUmrY2wsg4sVSIMc--K.png
Qihui Zhang
user
77Hui
67eb932522a341478ae86cb7
Munan Ning
admin_assigned
2025-04-01T08:18:39.336Z
65e14c28b1a6de8a71e70172
https://cdn-avatars.huggingface.co/v1/production/uploads/65e14c28b1a6de8a71e70172/D097SILGsqoufpp3sG8tV.jpeg
Munan Ning
user
MunanNing
67eb932522a341478ae86cb8
Zheyuan Liu
67eb932522a341478ae86cb9
Yanbo Wang
67eb932522a341478ae86cba
Jiayi Ye
67eb932522a341478ae86cbb
Yue Huang
admin_assigned
2025-04-01T08:19:26.627Z
6637443ecd9097ac3c996d3c
/avatars/d1c38bf03c2517ba0a7004b2f9f9bc96.svg
yue
user
yuehuang
67eb932522a341478ae86cbc
Shuo Yang
67eb932522a341478ae86cbd
Xiao Chen
67eb932522a341478ae86cbe
Yibing Song
admin_assigned
2025-04-01T08:19:07.616Z
62c51800cb7033fd49b8efb7
/avatars/06c2be0015f8022f9912f2279f2b3597.svg
Song
user
Yibing
67eb932522a341478ae86cbf
Li Yuan
claimed_verified
2025-04-04T07:09:54.399Z
66135a5e50350afe76beebce
/avatars/370a4b83949355feb050c2cb0425c264.svg
yl2488
user
yl2488
UPME: An Unsupervised Peer Review Framework for Multimodal Large
  Language Model Evaluation
Multimodal Large Language Models (MLLMs) have emerged to tackle the
challenges of Visual Question Answering (VQA), sparking a new research focus on
conducting objective evaluations of these models. Existing evaluation methods
face limitations due to the significant human workload required to design Q&A
pairs for visual images, which inherently restricts the scale and scope of
evaluations. Although automated MLLM-as-judge approaches attempt to reduce the
human workload through automatic evaluations, they often introduce biases. To
address these problems, we propose an Unsupervised Peer review MLLM Evaluation
framework. It utilizes only image data, allowing models to automatically
generate questions and conduct peer review assessments of answers from other
models, effectively alleviating the reliance on human workload. Additionally,
we introduce the vision-language scoring system to mitigate the bias issues,
which focuses on three aspects: (i) response correctness; (ii) visual
understanding and reasoning; and (iii) image-text correlation. Experimental
results demonstrate that UPME achieves a Pearson correlation of 0.944 with
human evaluations on the MMstar dataset and 0.814 on the ScienceQA dataset,
indicating that our framework closely aligns with human-designed benchmarks and
inherent human preferences.
Unsupervised Peer Review MLLM Evaluation framework uses image data to automatically generate questions and assess answers, reducing human workload and mitigating bias.
Multimodal Large Language Models Visual Question Answering MLLM-as-judge Unsupervised Peer review vision-language scoring system response correctness visual understanding reasoning image-text correlation Pearson correlation